2

At work we are moving from a TCP to a UDP protocol for a kind of 3D engine because we have 1s hang when a packet is lost. Since our engine is not designed to handle packet loss nor out of order arrival, we use a library (Lidgren) to do reliable UDP. So far, this is working as expected but it is slightly slower than TCP (20 to 30 %) in term of latency. A coworker of mine argues that this is because the reliability and ordering of packet involves CPU overhead for UDP while it is handled directly by the network adapter for TCP. So, is the network adapter doing some special work when using TCP and not UDP to spare CPU work?

2

The network adapter is a layer-1/2 device that really doesn't know or care about the layer-3 (IP, etc.) or layer-4 (TCP, UDP, etc.) protocol that is carried as the payload of layer-2 (ethernet, etc) protocol. The NIC is happy to carry something like IPX that uses SPX as its layer-4 protocol.

| improve this answer | |
  • Not in general, as many modern network cards are using TCP offload engine where all TCP/IP stack would be handled by ASIC. Especially on 1Gbps and faster connections, which works pretty good to reduce overhead on CPU. – Datagram.Network Feb 3 '17 at 15:01
  • Cards that do TCP offload also do UDP offload, and this would not explain the difference in TCP/UDP speed the question. – Ron Maupin Feb 3 '17 at 17:30
  • There is many NIC's designed especially for low latency with full UDP stack offload. So we cannot generalise that NIC's only do layer 1/2 which was true few years back. Right now most SOHO NIC's doing partial offloading of checksum, ARP etc. Of course is not the answer for this particular question. But I believe I've explained why would you expect bigger latency in my answer. – Datagram.Network Feb 3 '17 at 17:59
  • @Datagram.Network - Although some cards support TCP offload engine, in practice it isn't used in common operating systems. The reason is that different manufacturers have different view of how a TCP offload engine works, and the entire network stack would require a rewrite per each manufacturer. Checksum offload? Yes. Large send offload? Yes. Large receive offload? Yes. But full TCP offload? No, not usually supported by the operating system. – juhist Feb 5 '17 at 10:08
4

This is a good place to list what kinds of TCP offloading features a NIC hardware can have.

The most basic form of offload is checksum offload. It improves performance somewhat, but not much. Checksum is something that is very easy to do in hardware. Checksum offload, however, can show strange checksums when a .pcap is captured, so it is not fully transparent to the user. The solution to the .pcap problem is to turn checksum offload off.

Large send offload works because often an application write()'s a large buffer such as 64 kilobytes. Let's assume that of this, the TCP window and congestion control allow sending 24 kilobytes immediately. The operating system constructs a virtual TCP segment of 24 kilobytes and the 24 kilobyte virtual TCP segment passes through the operating system stack only once. Then the NIC breaks the large virtual 24 kilobyte segment into smaller physical segments. The idea here is to reduce software overheads by handling large virtual packets instead of small physical packets. The per-packet processing overhead applies then to a larger chunk of data. So, the effect is similar to running a large MTU.

Large receive offload is large send offload in the oppoisite direction, and works with the interrupt mitigation features of a NIC. In this case, the idea is that the NIC won't interrupt the operating system per each arriving packet but actually waits for a while for more packets to arrive. If the further packets seem to be consecutive packets belonging to the same connection, they are combined into one larger virtual segment (let's say a 32 kilobyte segment). Then the operating system's stack will process the large 32 kilobyte virtual segment once, and the packet processing overhead is amortized over 32 kilobytes instead of the usual 1500 bytes. This is the receive-side equivalent of large send offload. However, this feature will break routers, so it cannot be used in machines doing IP forwarding.

All of these previously listed TCP offload features are stateless. They are nice because they do not require practically any modifications to the operating system's TCP stack. There is a more full way of performing TCP offloading, called TCP offload engine. In it, part of the state of the TCP connection is offloaded to the NIC, and the NIC performs part of the TCP packet processing. However, the problem is that different manufacturers have different views of what the NIC should do. Thus, in practice if there are 10 NIC vendors, you need 10 different copies of the operating system TCP stack, each tailor-modified to work with a specific NIC vendor. Not going to happen! Also, once the rules are hardcoded to an ASIC, you cannot fix bugs in the rules.

In modern Linux systems using good network interface cards, checksum offloading, large send offloading and large receive offloading are used. However, TCP offload engine is not supported by any standard Linux kernel. Same probably applies to Windows as well.

| improve this answer | |
2

EDIT:

High speed adapters (1 Gb and above) do have special hardware to process TCP on the NIC itself, thereby saving CPU cycles. This is usually found on servers since they process more traffic than user workstations. "TCP offload" can be disabled through the adapter settings if it is causing application problems.

| improve this answer | |
  • Not in general, as many modern network cards are using TCP offload engine where all TCP/IP stack would be handled by ASIC. Especially on 1Gbps and faster connections, which works pretty good to reduce overhead. – Datagram.Network Feb 3 '17 at 15:07
  • 1
    @Datagram.Network You are right -- I haven't kept up with NIC developments. I will revise my answer. – Ron Trunk Feb 3 '17 at 15:10
  • I am running Windows 10 and have a Intel Ethernet Connection I218-LM card. Is there a way to know if this adapter has such a device? – codec-abc Feb 3 '17 at 15:21
  • Based on the data sheet (Google is your friend), it doesn't appear to have it. – Ron Trunk Feb 3 '17 at 15:27
  • Doesn't support full stack offload only partial for ARP, Checksum etc. – Datagram.Network Feb 3 '17 at 15:28
1

IMHO I assume that lidgren is working on Layer 7 (application layer). Which would increase CPU overhead and latency. Packets are still handled as normal by lower layers supported by operating system, as Lidgren don't change how UDP is handled by operating system. So to process additional checks on higher layer it would take time and use additional resources. Not really sure but maybe lidgren is adding additional headers in data which are handled by application and to process this you need CPU and is where latency could increase.

| improve this answer | |
  • Thanks. So you confirm that packet reliability and reordering for TCP are handled by the operating system (thus consuming CPU time) and not some specific hardware component on the network adapter? And if someone implement reliability and reordering for UDP in the application layer performance should be equal to the one using TCP (assuming the implementation is equally good) – codec-abc Feb 3 '17 at 14:48
  • I'm not saying that would be better as TCP, it could have a lower re transmission time than TCP but doesn't mean that would be faster in general. As reliability is handled on Layer 7 which takes more time than on lower layer. – Datagram.Network Feb 3 '17 at 14:51
  • Could you explain why if reliability is implemented on Layer 7 it would take more time than on lower levels? – codec-abc Feb 3 '17 at 14:54
  • Please see below video how encapsulation and decapsulation works. youtube.com/watch?v=EGnqIILfE9Q If you adding another information on layer 7 in data it means that this data need to encapsulated/decapsulated by application and this would take additional time. – Datagram.Network Feb 3 '17 at 14:55
  • 2
    Probably because it is written in a higher level language and runs in user space. The IP stack is optimized for speed, runs at the kernel level, and is a higher priority task. – Ron Trunk Feb 3 '17 at 14:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.