Does TCP open a new connection for every packet that is sent?

Question

This might be a silly question but a me and few buddies have been discussing the potential limitations of TCP. We have an application that is going to listen for clients (think of a gateway) and route all connected clients data through a single connected kafka publisher to one topic.

One of my buddies is saying that TCP will be a problem for this gateway because it is going to establish a new connection for every message it sends (not kafka but the underlying transportation protocol itself is the issue), requiring a new port each time. At the rate we'll be sending these clients messages (gigabytes), kafka will run out of ports to read from??

I've done development for several years and have never heard of this before and would like to get a lower level understanding (which I thought I had) of how TCP works. My understanding is that when you establish a TCP connection, that connection remains open until it is timed out by the application or forcibly closed by either the server or client. Data that is sent over this connection is a stream and won't open / close new connections regardless of the 3 V's (volume, velocity, variety).

As far as the ports go, one port is used for broadcasting and the internal file descriptor port is something the application manages for read / write of individual clients. I've never understood TCP to establish new connections for every packet that it writes.

I apologize in advance if this question is not direct and or too vague. I really am baffled and am hoping someone could provide some more context to what my colleagues are saying?

Answer 1

One of my buddies is saying that TCP will be a problem for this gateway because it is going to establish a new connection for every message it sends (not kafka but the underlying transportation protocol itself is the issue), requiring a new port each time. At the rate we'll be sending these clients messages (gigabytes), kafka will run out of ports to read from??

Your friend is badly confused. TCP is a stream-oriented protocol. It has no notion of messages. Of course, it does use packets at the IP layer, but to the application this is an implementation detail. TCP inserts packet boundaries where it makes sense to do so, and not necessarily once per write() or send(). Similarly, it combines successive packets together if you receive more than one between calls to read() or recv().

Needless to say, this stream-oriented design would be completely unworkable if every send established a new connection. So, the only way to establish a new connection is to close and reopen the connection manually.

(In practice, most protocols built on top of TCP have something which resembles messages, such as HTTP requests and responses. But TCP does not know or care about the structures of such things.)

It is possible that your friend was thinking of UDP, which does have messages, but is also connectionless. Most socket implementations allow you to "connect" a UDP socket to a remote host, but this is just a convenient way to avoid having to repeatedly specify the IP address and port. It does not actually do anything at the networking level. Nevertheless, you can manually keep track of which peers you are talking to under UDP. But if you do that, then deciding what counts as a "connection" is your problem, not the OS's. If you want to re-establish a "connection" on every message, you could do that. It probably isn't a very good idea, however.

Answer 2

My understanding is that when you establish a TCP connection, that connection remains open until it is timed out by the application or forcibly closed by either the server or client.

From the perspective of TCP, there is no client or server (client/server is an application concept that is off-topic here). TCP establishes a connection between peers, and both peers can send and receive on the connection until either peer closes it, or it times out from inactivity.

Data that is sent over this connection is a stream and won't open / close new connections regardless of the 3 V's (volume, velocity, variety).

What may be confusing the situation is that some applications, e.g. browsers, will open multiple connections in order to simultaneously load things like elements of a web page.

TCP does not open a new connection for every segment it sends, but an application may open multiple TCP connections. Also, when a TCP connection is closed, the TCP port used in the connection is freed, and it is available to be used again. This answer gives some information, and it points you to the RFC for TCP.

Answer 3

No, TCP does not need to open a new connection for every packet that is sent.

You can send multiple packets by way of HTTP persistent connections, where:

... a single TCP connection to send and receive multiple HTTP requests/responses [is used], as opposed to opening a new connection for every single request/response pair.

Attached is a figure showing the difference between multiple connections (many connections established to send one object per connection) and a persistent connection (one connection established and multiple objects sent therein):

Source: https://www.vcloudnine.de/how-to-dramatically-improve-website-load-times/

Answer 4

As others have pointed out, TCP absolutely allows a connection to stay open for any amount of time, exchanging any number of "messages" in either direction during that time. That said, it's ultimately up to the applications (both client and server) to determine if that capability is utilized.

In order to reuse the existing TCP connection (socket), the client application must keep that socket open and use it when it needs to write more data. If the client does not do this, but instead discards the old socket and opens a new socket every time it needs one, then it will indeed force a new connection which could cause resource issues on either the client or server if done frequently enough to exhaust either TCP stack's connection pool.

Likewise, the server must be smart enough to keep the socket open on its side and wait for more data. Like the client, it has the option of closing the socket at which point a fault-tolerant client wishing to send more data will have no choice but to open a new socket, leading to the same problem.

Finally, as others have mentioned, TCP is stream-oriented. There is no framing whatsoever. Just because one peer wrote the data a particular way (e.g. 1 1024 byte write call following by 2 256 byte write calls), that does not guarantee that the other peer will read it in the same size chunks (e.g. it might get all 1536 bytes in one read call). Thus if you are sending multiple "messages" over raw TCP sockets, you have to provide your own framing protocol to delineate the different messages. While there certainly are simple ways to do this, it's generally ill-advised as there are many protocols built on top of TCP to solve this problem. For further discussion, consult this: https://blog.stephencleary.com/2009/04/message-framing.html

Answer 5

Your interpretation of how TCP works is correct.

As for what your friend said, I see two possibilities here:

1. You misunderstood your friend, who was referring to some application-layer limitation that results in each message being sent over a new connection (and this is not necessarily unusual; it may or may not be possible to decide on this behaviour, depending on what software stack you're using);

2. Your friend is wrong.

Answer 6

I think your friend was talking about HTTP, not TCP.

HTTP was originally a stateless protocol: each HTTP request would use a separate TCP connection. This is why we need cookies (or something similar) to implement sessions.

Answer 7

You have mentioned "single connection and requiring a new port each time", and I would interpret as you have many client using the PAT technique in the same network environment to connect to server outside your organization. The PAT would have the limit of 65535 (TCP session limit on IPv4 Address). If it is true, you have the limit.

Does TCP open a new connection for every packet that is sent? NO, it does not as long as the TCP session is valid. and ...

Answer 8

I like the excellent wikipedia page on TCP. It clearly shows what happens with the port number. It, by chance, also contains a helpful chapter on ressource usage:

Resource usage

Most implementations allocate an entry in a table that maps a session to a running operating system process. Because TCP packets do not include a session identifier, both endpoints identify the session using the client's address and port. Whenever a packet is received, the TCP implementation must perform a lookup on this table to find the destination process. Each entry in the table is known as a Transmission Control Block or TCB. It contains information about the endpoints (IP and port), status of the connection, running data about the packets that are being exchanged and buffers for sending and receiving data.

The number of sessions in the server side is limited only by memory and can grow as new connections arrive, but the client must allocate a random port before sending the first SYN to the server. This port remains allocated during the whole conversation, and effectively limits the number of outgoing connections from each of the client's IP addresses. If an application fails to properly close unrequired connections, a client can run out of resources and become unable to establish new TCP connections, even from other applications.

In short, TCP uses up one very finite ressource, which is the number of ports on the client (which is limited by the size of the port field in the TCP header, 16 bits).

So, TCP is able to run out of ports, if a client opens up a lot of TCP connections in parallel without closing them. The problem only occurs client-side, and it does not matter if the connections are with the same or different server IP addresses or server ports.

In your setting, you seem to have one application which takes in many client requests (these could be individual TCP requests, as maybe your clients use this to log some events to your application and don't hold the TCP channel open inbetween), and create a new internal request to your Kafka broker (which very easily could be individual TCP connections if you chose to implement them like this). In this case, the bottleneck (in terms of ressources, not performance) would be if you manage to get huge numbers of requests at the same time from your clients (no problem for you, as on the server side you only need one port for all of them), and you open up a huge number of forward requests to your Kafka, and Kafka is not able to process them fast enough, ending up with you having more than 16bits worth of connections open concurrently.

You are the own judge here; check your application and try to find out if you are connecting to Kafka with a separate request each time (maybe through some REST API proxy). If you do so, and you have huge numbers of clients, then you are certainly in danger.

If you only have a handful of clients, less than 65k-ish, and/or you keep a single connection to your Kafka browser, then you'll be fine.