I am analyzing a pcap of command-and-control traffic caught during execution of a malicious file. First, a DNS request for a malicious domain was sent and after response is received and TCP connection to the remote destination (purple colour) is attempted, server responds with [RST, ACK] which means that the server did instantly close the session. enter image description here

I know that one of the possible reasons of such response is destination port not listening, is this the most common reason to the point where [RST, ACK] can be effectively treated as "destination port not listening" or are there any other possibilities? Specifically, I wonder if it is possible that destination server closed the session because data sent by source matched some condition (e.g. indicated that the source host is a virtual machine) of undesired source hosts. I know that "custom TCP protocol" is used for this malware's command-and-control.

I understand that concrete answer to why the session was closed would require visibility on the server side, hence I am not looking for something like that, but I am trying to make an educated guess on what did likely happen.

up vote 6 down vote accepted

A RST received after a SYN request means the TCP handshake has failed. No user data was transmitted.

Whether this happens due to the destination port not listening, a firewall response, or other filtering is up to the the destination host/network. There's no way to be sure from viewing from the outside.

The RST can be caused by any decision on the destination side, including source address, source port, missing prior activities and effectively anything else you can think of.

With an established connection (that you don't have with RST directly following SYN) the destination can also close the socket based on content, of course.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.