"Expected" packet loss under heavy network load in TCP cubic

Question

context

We have two machines that receive a lot of data, and mirror each other's content with rsync. When monitoring these boxes via ICMP echo requests (using the mtr tool), we see up to a couple % of packet loss.

I investigated statistics on the dual 10GigE NICs on this host without spotting anything that could lead me to why this is happening. I do not see loss at any other hop in the network path either.

question

TCP-Cubic will slowly ramp its window size until a congestion event is received; in practice such an event is the loss of a packet. Loss, then, becomes an important part of network feedback and so some loss is "expected". What I cannot figure out is:

a) Will loss in TCP (from congestion control) explain loss seen in other protocols, such as ICMP?

b) How can I calculate what the "expected" amount of loss should be? From TCP parameters? Or via network probing, for example with iperf?

Hope everyone is having a good day :)

Answer 1

Will loss in TCP (from congestion control) explain loss seen in other protocols, such as ICMP?

Possibly. When there's congestion, any excess packets are dropped, whatever they carry (ignoring QoS scenarios). However, ICMP is generally low priority, so a drop may be more likely across a router or on a host - depending on the implementation.

Note that "packet loss" can be due to congestion along the network path, but also on the source or the destination host (interface or processing limits). Dual 10G loads are not handled lightly, even on modern machines, depending on their hardware, software and configuration.

How can I calculate what the "expected" amount of loss should be? From TCP parameters? Or via network probing, for example with iperf?

I don't think calculating the loss is easily possible. You'd need to create a very exact model of your environment, including the device implementations and configurations, and with all the expected workload scenarios. In reality, there are too many parameters that you'd need to watch.

I suppose a practical way is to actually measure the amount of packet loss (=duplicate transmissions on the source) in some model cases (or simply over a longer time period) and then estimate a threshold for a "good" state. You can also correlate endpoint variables (CPU, memory, storage loads, service threads, ...). Note that hosts and their configurations are off topic here, however.

In case of rapidly changing workloads, the packet loss can well amount to a few percent.