FIN tells the receiver of the FIN that the sender of the FIN is done sending data, but it will still listen. The receiver of a FIN is under no obligation to stop, unless it is also done, and it can keep sending data to the sender of the FIN until it is also done, whereupon it sends a FIN. It is all very courteous and orderly with ACKs to let the FIN sender know that the FIN receiver has seen the FIN, else the FIN sender will need to resend the FIN in case it got lost.
On the other hand, a RST is sent when a TCP receives something unexpected or for which it has no connection. TCP cannot send an ACK if there is no connection because there is no data to ACK if there is no connection. Courtesy and orderliness are non-existent for a RST because the sender of the RST is not interested in hearing from the sender of the segment that caused the RST, so an ACK from the receiver of a RST would again generate a RST.
As a general rule, reset (RST) must be sent whenever a segment arrives
which apparently is not intended for the current connection. A reset
must not be sent if it is not clear that this is the case.
When a RST is generated, the segment causing the RST is dropped because there is no destination process for reception of the segment data.
There are three groups of states:
If the connection does not exist (CLOSED) then a reset is sent in
response to any incoming segment except another reset. In
particular, SYNs addressed to a non-existent connection are rejected
by this means.
If the incoming segment has an ACK field, the reset takes its
sequence number from the ACK field of the segment, otherwise the
reset has sequence number zero and the ACK field is set to the sum
of the sequence number and segment length of the incoming segment.
The connection remains in the CLOSED state.
If the connection is in any non-synchronized state (LISTEN,
SYN-SENT, SYN-RECEIVED), and the incoming segment acknowledges
something not yet sent (the segment carries an unacceptable ACK), or
if an incoming segment has a security level or compartment which
does not exactly match the level and compartment requested for the
connection, a reset is sent.
If our SYN has not been acknowledged and the precedence level of the
incoming segment is higher than the precedence level requested then
either raise the local precedence level (if allowed by the user and
the system) or send a reset; or if the precedence level of the
incoming segment is lower than the precedence level requested then
continue as if the precedence matched exactly (if the remote TCP
cannot raise the precedence level to match ours this will be
detected in the next segment it sends, and the connection will be
terminated then). If our SYN has been acknowledged (perhaps in this
incoming segment) the precedence level of the incoming segment must
match the local precedence level exactly, if it does not a reset
must be sent.
If the incoming segment has an ACK field, the reset takes its
sequence number from the ACK field of the segment, otherwise the
reset has sequence number zero and the ACK field is set to the sum
of the sequence number and segment length of the incoming segment.
The connection remains in the same state.
If the connection is in a synchronized state (ESTABLISHED,
FIN-WAIT-1, FIN-WAIT-2, CLOSE-WAIT, CLOSING, LAST-ACK, TIME-WAIT),
any unacceptable segment (out of window sequence number or
unacceptible acknowledgment number) must elicit only an empty
acknowledgment segment containing the current send-sequence number
and an acknowledgment indicating the next sequence number expected
to be received, and the connection remains in the same state.
If an incoming segment has a security level, or compartment, or
precedence which does not exactly match the level, and compartment,
and precedence requested for the connection, a reset is sent and
connection goes to the CLOSED state. The reset takes its sequence
number from the ACK field of the incoming segment.
