Trying to find out exact TCP overhead cost

Question

Corresponding this topic:

https://stackoverflow.com/questions/3613989/what-of-traffic-is-network-overhead-on-top-of-http-s-requests

The maximum segment size (which does not include the TCP or IP headers) is typically negotiated between the layers to the size of the MTU minus the headers size. For Ethernet MTU is usually configured at 1500 bytes. The TCP header is 160 bits, or 20 bytes. The fixed part of the IPv4 header is 160 bits, or 20 bytes as well. ... . Thus:

• for HTTP over TCP/IPv4

overhead = TCP + IP = 40 bytes

payload = 1500 - 40 = 1460 bytes

overhead % = 2% (40 * 100 / 1460)

Here's 100 Mbit and 1Gbit iperf results in TCP mode with default Debian distros:

[  5] local 10.0.51.1 port 5001 connected with 10.0.51.20 port 45009
[  5]  0.0-10.0 sec   112 MBytes  94.1 Mbits/sec
[  4] local 10.0.51.1 port 5001 connected with 10.0.51.94 port 35065
[  4]  0.0-10.0 sec  1.10 GBytes   941 Mbits/sec

I can lower it to almost 2% overhead by raising MTU to 9000 :

[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.14 GBytes   982 Mbits/sec

But shouldn't it be even less?

overhead = TCP + IP = 40 bytes
payload = 9000 - 40 = 8960 bytes
overhead % = 0.4% (40 * 100 / 8960)

Why actual "bandwidth loss" is notably greater than the theoretical? If formula missing something valuable?

Answer 1

Ethernet packets 1.5k

1500 - 20 B (IPv4) - 20 B (TCP+checksum) = 1460 B DATA (and 40 B Overhead)

Add 40 B + 14 B (Ethernet) + 4 B (FCS) + 12 B (Interframe gap) + 8 B (preamble) = 78 B Overhead

78 / 1460 * 100 = 5.34% overhead

1460 / ( 1460 + 78) * 100 = 94.93% Throughput/Goodput

1,000,000,000(1Gbit) * 94.93% = 949Mbit/s(0.949Gbit/s)

you measured 941Mbit/s that gives (949 - 941) / 949 * 100 = 0.84 % error between theoretical and actual.

---

Jumbo packets 9k - Theoretical max

(9000-40) / ( 9000 - 40 + 78 ) *100 = 99.14%  (Overhead 0.86%)  

1,000,000,000(1Gbit) * 99.14% = 991Mbit/s(0.99Gbit/s)

Answer 2

Overhead is usually calculated based on the total data size. That way, the figure matches the efficiency value.

For TCP over IPv4 over Ethernet you have (without header options):

• L1 overhead - preamble, IPG: 8+12 = 20
• L2 overhead - Ethernet header, FCS = 18
• L3 overhead - IPv4 header = 20
• L4 overhead - TCP header = 20

The L3 maximum packet size of 1500 results in a total L1 data size of 1500+18+20 = 1538 bytes and a maximum L4 payload size of 1500-20-20 = 1460 bytes.

• Overhead: 78/1538 *100% = 5.07%
• Efficiency: 1460/1538 *100% = 94.93%

With 9k jumbo frames (non-802.3), you'd get

• Overhead: 78/9038 *100% = .86%
• Efficiency: 8960/9038 *100% = 99.14%

These are theoretical, best-case values. In real life, you'd also have some hardware and OS overhead slightly eating into the efficiency value. Features like offloading and multi-core interrupt steering can reduce the processing overhead and get you closer to the theoretical figures (more relevant with higher speed NICs). The ones you've measured look pretty realistic as Pieter's already pointed out.