I was looking at the counters on a machine in production and found that the read throughput was 3.5 GBps and the write throughput was 4.5 GBps ( the counter measures the read bytes and write bytes per second for the network interface). The machine has one NIC with 41Gbps i.e. ~5 GBps. How is this possible ? My understanding is that network bandwidth is the total ( read + write) bits that can be transferred per second.
Can total throughput (read throughput + write throughput) exceed the bandwidth of a network?
4"My understanding is that network bandwidth is the total ( read + write) bits that can be transferred per second." No, modern wired networks are full duplex. Each direction is independent of the other.– Ron Maupin ♦Nov 3, 2021 at 2:04
When you say independent of the other you mean the total throughput (read + write) can be greater than the n/w bandwidth ? i.e. the sum of both "read bytes per second" and "write bytes per second" <= (2*bandwidth)– Volatile_volvoNov 3, 2021 at 6:40
3Basically yes - a 40G Ethernet link can carry 40Gbit/s up and 40Gbit/s down at the same time.– Zac67 ♦Nov 3, 2021 at 6:59
A modern Ethernet link is generally full duplex: both directions work independently of each other. Full duplex has become the standard with fully switched Ethernet nearly 20 years ago.
For Ethernet, the nominal speed is generally the one used at the top of the physical layer (L1). The line code (8b/10b, 64b/66b, ...) is irrelevant, but the highest-level L1 overhead - preamble, SFD, IPG, 20 bytes total - already eats into that nominal speed. Also, depending on the actual protocol stack, all the higher-layer overheads also eat into the usable bandwidth.
Most commonly, a standard Ethernet frame has an minimum overhead of 18 bytes (DestMAC, SourceMAC, EtherType, FCS), with 1500 bytes maximum payload. If IPv4 is used that's another minimum overhead of 20 bytes. For TCP, it's another 20 bytes. All in all, 1460 usable bytes with a total of 1538 bytes on the wire result in a maximum efficiency of 94.9%.
So, for TCP over IPv4 over 40GBASE-X without any bottleneck, you could expect a peak throughput of 4.746 GB/s per direction.
1corrected the question, it was 4.5 GB/s not 7.5 GB/s , thanks. Nov 3, 2021 at 8:05
@thor_hayek Excellent, thanks for clarifying.– Zac67 ♦Nov 3, 2021 at 8:51