This might be a silly question but a me and few buddies have been discussing the potential limitations of TCP. We have an application that is going to listen for clients (think of a gateway) and route all connected clients data through a single connected kafka publisher to one topic.
One of my buddies is saying that TCP will be a problem for this gateway because it is going to establish a new connection for every message it sends (not kafka but the underlying transportation protocol itself is the issue), requiring a new port each time. At the rate we'll be sending these clients messages (gigabytes), kafka will run out of ports to read from??
I've done development for several years and have never heard of this before and would like to get a lower level understanding (which I thought I had) of how TCP works. My understanding is that when you establish a TCP connection, that connection remains open until it is timed out by the application or forcibly closed by either the server or client. Data that is sent over this connection is a stream and won't open / close new connections regardless of the 3 V's (volume, velocity, variety).
As far as the ports go, one port is used for broadcasting and the internal file descriptor port is something the application manages for read / write of individual clients. I've never understood TCP to establish new connections for every packet that it writes.
I apologize in advance if this question is not direct and or too vague. I really am baffled and am hoping someone could provide some more context to what my colleagues are saying?
SO_REUSEADDRto close sockets faster, increasing range of ephemeral ports etc. In addition
TCP_FASTOPENand several OS-level toggles can be used to work around other well-known limitations of TCP. Either way, there is no point in discussing limitations of TCP when you don't even have a workload to test on.