Is there anything inside the application layer in the three way handshake phase?

Question

So is the application layer of TCP handshake packets empty no matter what the protocol is?

and when the first packet of TCP handshake(phase 1) arrives at the server, which process does the server hand the packet to if there is no application layer in that packet and basically no message? i mean what that process gonna do with an empty message? can you guys explain in a detailed manner? (for example if the O.S handles the 3 way handshake packets, then how does it happen exactly and which process gets the packet?)

Answer 1

There doesn’t have to be any payload data in a TCP segment. The segment could contain only control information signalled in the TCP header. If a segment arrives with no payload data, the receiving TCP doesn’t have to send anything to the application, unless the control information requires it to (e.g. the flags signal a FIN or RST and TCP needs to signal that to the application).

Interestingly there is nothing stopping TCP sending application data within the TCP payload during the three-way handshake. If TCP receives payload data before completion of the three-way handshake it must buffer the data and not pass it onto the application until the handshake is completed and the connection is in an ESTABLISHED state.

From RFC 793:

Although these examples do not show connection synchronization using data-carrying segments, this is perfectly legitimate, so long as the receiving TCP doesn't deliver the data to the user until it is clear the data is valid (i.e., the data must be buffered at the receiver until the connection reaches the ESTABLISHED state). The three-way handshake reduces the possibility of false connections. It is the implementation of a trade-off between memory and messages to provide information for this checking.

There is also an extension called TCP Fast Open (RFC 7413) that is useful for short lived connections and allows the data, under certain conditions, to be sent to the application before the three-way handshake completes.

So to answer your question it depends on the implementation and whether the application has any data to send during the three-way handshake. Some connections have no application data in the payload of the handshake segments, others do.

There is always an application to hand data to for each TCP connection (unless it is an attempt to connect to an unopen port). An application on one side initiated the connection (through TCP) and another application on the other side opened a port and is listening on that port (through TCP). If the segment has no payload data, TCP doesn’t need to send anything to the application. If it does, it can sent it once the connection is ESTABLISHED or earlier with Fast Open