Can total throughput (read throughput + write throughput) exceed the bandwidth of a network?

Question

I was looking at the counters on a machine in production and found that the read throughput was 3.5 GBps and the write throughput was 4.5 GBps ( the counter measures the read bytes and write bytes per second for the network interface). The machine has one NIC with 41Gbps i.e. ~5 GBps. How is this possible ? My understanding is that network bandwidth is the total ( read + write) bits that can be transferred per second.

Answer 1

A modern Ethernet link is generally full duplex: both directions work independently of each other. Full duplex has become the standard with fully switched Ethernet nearly 20 years ago.

For Ethernet, the nominal speed is generally the one used at the top of the physical layer (L1). The line code (8b/10b, 64b/66b, ...) is irrelevant, but the highest-level L1 overhead - preamble, SFD, IPG, 20 bytes total - already eats into that nominal speed. Also, depending on the actual protocol stack, all the higher-layer overheads also eat into the usable bandwidth.

Most commonly, a standard Ethernet frame has an minimum overhead of 18 bytes (DestMAC, SourceMAC, EtherType, FCS), with 1500 bytes maximum payload. If IPv4 is used that's another minimum overhead of 20 bytes. For TCP, it's another 20 bytes. All in all, 1460 usable bytes with a total of 1538 bytes on the wire result in a maximum efficiency of 94.9%.

So, for TCP over IPv4 over 40GBASE-X without any bottleneck, you could expect a peak throughput of 4.746 GB/s per direction.