At work we are moving from a TCP to a UDP protocol for a kind of 3D engine because we have 1s hang when a packet is lost. Since our engine is not designed to handle packet loss nor out of order arrival, we use a library (Lidgren) to do reliable UDP. So far, this is working as expected but it is slightly slower than TCP (20 to 30 %) in term of latency. A coworker of mine argues that this is because the reliability and ordering of packet involves CPU overhead for UDP while it is handled directly by the network adapter for TCP. So, is the network adapter doing some special work when using TCP and not UDP to spare CPU work?
This is a good place to list what kinds of TCP offloading features a NIC hardware can have.
The most basic form of offload is checksum offload. It improves performance somewhat, but not much. Checksum is something that is very easy to do in hardware. Checksum offload, however, can show strange checksums when a .pcap is captured, so it is not fully transparent to the user. The solution to the .pcap problem is to turn checksum offload off.
Large send offload works because often an application write()'s a large buffer such as 64 kilobytes. Let's assume that of this, the TCP window and congestion control allow sending 24 kilobytes immediately. The operating system constructs a virtual TCP segment of 24 kilobytes and the 24 kilobyte virtual TCP segment passes through the operating system stack only once. Then the NIC breaks the large virtual 24 kilobyte segment into smaller physical segments. The idea here is to reduce software overheads by handling large virtual packets instead of small physical packets. The per-packet processing overhead applies then to a larger chunk of data. So, the effect is similar to running a large MTU.
Large receive offload is large send offload in the oppoisite direction, and works with the interrupt mitigation features of a NIC. In this case, the idea is that the NIC won't interrupt the operating system per each arriving packet but actually waits for a while for more packets to arrive. If the further packets seem to be consecutive packets belonging to the same connection, they are combined into one larger virtual segment (let's say a 32 kilobyte segment). Then the operating system's stack will process the large 32 kilobyte virtual segment once, and the packet processing overhead is amortized over 32 kilobytes instead of the usual 1500 bytes. This is the receive-side equivalent of large send offload. However, this feature will break routers, so it cannot be used in machines doing IP forwarding.
All of these previously listed TCP offload features are stateless. They are nice because they do not require practically any modifications to the operating system's TCP stack. There is a more full way of performing TCP offloading, called TCP offload engine. In it, part of the state of the TCP connection is offloaded to the NIC, and the NIC performs part of the TCP packet processing. However, the problem is that different manufacturers have different views of what the NIC should do. Thus, in practice if there are 10 NIC vendors, you need 10 different copies of the operating system TCP stack, each tailor-modified to work with a specific NIC vendor. Not going to happen! Also, once the rules are hardcoded to an ASIC, you cannot fix bugs in the rules.
In modern Linux systems using good network interface cards, checksum offloading, large send offloading and large receive offloading are used. However, TCP offload engine is not supported by any standard Linux kernel. Same probably applies to Windows as well.
High speed adapters (1 Gb and above) do have special hardware to process TCP on the NIC itself, thereby saving CPU cycles. This is usually found on servers since they process more traffic than user workstations. "TCP offload" can be disabled through the adapter settings if it is causing application problems.
IMHO I assume that lidgren is working on Layer 7 (application layer). Which would increase CPU overhead and latency. Packets are still handled as normal by lower layers supported by operating system, as Lidgren don't change how UDP is handled by operating system. So to process additional checks on higher layer it would take time and use additional resources. Not really sure but maybe lidgren is adding additional headers in data which are handled by application and to process this you need CPU and is where latency could increase.