Recently I had a discussion about differences between TCP and UDP, and the other person insisted that there's a difference in packet forwarding: The packets in a TCP connection follow an established path, so that in a diamond configuration where a host has two paths to another host a connection will use only one path. Moreover, if a path is broken the TCP connections on it will not use the other path, they will time out instead. UDP traffic is not affected in any way.

This goes against what I've learned about packet forwarding in general, but I haven't been able to confirm or deny it. Is it true? Why would switches give TCP connections this special treatment, doesn't it make the network less reliable in general?

  • 2
    The person you were talking to said TCP fails to do exactly what it was designed to do: reliably transfer data and intelligently deal with packet retransmission. – Ryan Foley Nov 23 '14 at 20:20

Actually there is some truth in what the other person was saying, though it is largely false.

Is it true (that packets in a TCP connection follow an established path)?

Yes, in general, all packets in a TCP stream will follow the same path through the network - even in the presence of a "diamond" network, all packets in the same stream will be routed down the same side of the diamond. However, it's not true that if that side of the diamond goes down, the TCP stream will time out - in that case it will be rerouted and its path will change.

Why would switches give TCP connections this special treatment?

Here I'll answer why switches and routers will always try to send packets in a stream along the same side of a "diamond", hopefully giving some insight into where your interlocutor got their (flawed) understanding of the situation. The brief answer is "performance".

Background: While it's true that TCP can handle reordered packets successfully, in practice reordering tends to cause significant performance issues. For instance, if the other end sees packets in the order "1 2 4 3", it will notice when it sees packet 4 that it has not yet seen packet 3, and request for the other end to resend it - and it will then receive packet 3 twice (the one originally sent plus the resend). This results in bandwith being wasted (by sending packets twice unnecessarily), and also in reduced connection performance, since the sender will assume the "packet loss" it thinks it has seen is due to congestion, so it will slow down its sending.

Therefore most quality switches and routers will go to quite some lengths to avoid packet reordering, and as part of this will try to send all packets from the same TCP stream along the same path (in case one path has higher latency than the other).

Contrary to what others have said, this happens even for core routers. Where there are multiple links that are being load-balanced over, the router will try to send packets from the same TCP stream over the same link. Although this might seem to require the tracking of a huge amount of state, in fact it can be done without tracking all of the streams: the router takes the identifiers of the stream (source/destination IP address, and sometimes source/destination port), and combines them (hashes them) into a single number. This number is used to select which link to send the packet on. (To give an example of one vendor's implementation of this feature, search for ip cef load-sharing algorithm.)

Doesn't it make the network less reliable in general?

Yes, of course it would, if networks did exactly what your interlocutor described - so they don't do that. The behaviour the other person talked about where a path is dropped and TCP connections using that path time out is not typical: though it may be possible to configure some products to behave like this, in general both TCP and UDP will start to use the alternative route straight away.

When a link goes down, the hashes will be redistributed across the remaining links. This will result in TCP streams changing their paths - which may sometimes result in some reordering - but this is generally acceptable as it happens occasionally, rather than on every packet.

Other relevant information

The discussion was originally about the difference between handling of TCP and UDP packets.

In fact, all of the above handing is done for UDP packets too: that is, UDP packets from the same "connection" will also typically always follow the same path. This is desirable since UDP packets are often used for realtime media streams such as phone calls - if many of your voice packets arrive out of order, this can result in terrible audio quality (since for a phone call large buffering is highly undesirable, so the received will typically drop packets that arrive out of sequence).

  • I find the mention of reordering avoidance helpful, does this feature have a common name so I could investigate further? – Joni Nov 23 '14 at 22:57
  • 1
    Reordering has nothing to do with your question. – Mike Pennington Nov 23 '14 at 23:09
  • 1
    @joni Flow-based forwarding. It's not 'stateful' like a traditional firewall would be stateful, but it does (generally) depend on hashing the source & dest address, and source and dest port--though this varies by platform and configuration. In any case, in case of a broken path will not cause the flow to break--it'll get hashed to another link under normal circumstances. – dk1 Nov 23 '14 at 23:20
  • continued in chat – Mike Pennington Nov 24 '14 at 12:30
  • I'm accepting this answer because it explains why the packets of a connection are forwarded over the same path and how it can be implemented. This gives a more compete picture of the situation and some insight to why my interviewer may have thought that a failure on a path would cause TCP connections to time out. – Joni Nov 24 '14 at 18:51

Transport of both UDP and TCP packets from one router to the next is done at the IP layer and solely based on the information at this layer. This means, that there is no distinction between UDP and TCP and in both cases change of path, congestion or router hickups can cause loss, duplication or reordering of packets.

But, contrary to UDP, TCP can deal with duplication and reordering due to the sequence number in each packet and with packet loss by acknowledging packets and retransmitting unacknowledged packets. But this kind of repair is only done at the endpoints of the communication.

  • 1
    That's what I though, but the other person knew more about networks than I and was very insistent. Is it possible to configure switches/routers to have this behavior, i.e. to keep track of the next hop separately for each TCP connection? – Joni Nov 23 '14 at 20:44
  • 2
    At least core routers on the internet don't do such stateful routing because it would be much too expensive (memory and performance) to keep all the state information and track each connection. Routers at less busy places might have such stateful tracking implemented (but most will not), but these don't usually switch routes to the same target. Load balancers need this kind of stateful handling, but these are not routers. In summary: in a few cases like load balancing you will have such stateful handling, but mostly not because there is no need for it. – Steffen Ullrich Nov 23 '14 at 20:51
  • 2
    @joni, if the person was referring to services through an F5 load-balancer, in fact he might be correct. By default, F5 has a feature enabled called auto_lasthop; this feature remembers the mac-address which transmitted the last packet, and always tries to deliver return traffic to the same mac-addr. By default, this causes problems with Cisco HSRP and other high-availability topologies, unless you use something like F5 Last-hop pools. The auto_lasthop behavior tends to show up more in TCP traffic than UDP traffic – Mike Pennington Nov 23 '14 at 20:57

You're correct, not the person you were speaking to.

TCP establishes a connection between a "client" and "server", anything in between is just the the paths. It doesn't care how the traffic gets there, just that it does get there (using the built in sequence numbers, acknowledgments, and congestion avoidance capabilities). In your diamond example, you load share over both of the middle routers, you could just have one primary and one backup path, etc. It doesn't matter, as long as IP reachability is present, TCP can handle the loss.

  • "TCP establishes a connection between a 'client' and 'server...'" No. TCP establishes connections between peers. TCP knows nothing about clients and servers because client/server is an application-layer concept. – Ron Maupin Dec 19 '16 at 5:17
  • @RonMaupin but anyway the handshake is asymmetric -- one peer initiates -- and I think some people use "client" as a name for the peer that initiates. – Marius Jan 24 '17 at 22:27
  • 1
    @Marius, that would not actually be accurate. Even with a peer-to-peer application, one peer must initiate the connection, and I have seen applications where a server initiates the connection to poll clients. The client/server model is an application concept. – Ron Maupin Jan 24 '17 at 22:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.