0

context

We have two machines that receive a lot of data, and mirror each other's content with rsync. When monitoring these boxes via ICMP echo requests (using the mtr tool), we see up to a couple % of packet loss.

I investigated statistics on the dual 10GigE NICs on this host without spotting anything that could lead me to why this is happening. I do not see loss at any other hop in the network path either.

question

TCP-Cubic will slowly ramp its window size until a congestion event is received; in practice such an event is the loss of a packet. Loss, then, becomes an important part of network feedback and so some loss is "expected". What I cannot figure out is:

a) Will loss in TCP (from congestion control) explain loss seen in other protocols, such as ICMP?

b) How can I calculate what the "expected" amount of loss should be? From TCP parameters? Or via network probing, for example with iperf?

Hope everyone is having a good day :)

1
  • 1
    ICMP is an integral part of IP, but it is a low-priority protocol, and devices will wait on ICMP when performing other tasks. It is possible that the ICMP replies do arrive, but after the timeout of your ping. You cannot rely on ICMP to tell you anything about other protocols; it is only to determine IP connectivity on devices that are not blocking ICMP echo. To determine TCP loss, you can look at the TCP traffic with a monitoring too like Wireshark to see the retransmissions, but you cannot use a different protocol to determine TCP loss.
    – Ron Maupin
    Sep 29, 2022 at 12:53

1 Answer 1

1

Will loss in TCP (from congestion control) explain loss seen in other protocols, such as ICMP?

Possibly. When there's congestion, any excess packets are dropped, whatever they carry (ignoring QoS scenarios). However, ICMP is generally low priority, so a drop may be more likely across a router or on a host - depending on the implementation.

Note that "packet loss" can be due to congestion along the network path, but also on the source or the destination host (interface or processing limits). Dual 10G loads are not handled lightly, even on modern machines, depending on their hardware, software and configuration.

How can I calculate what the "expected" amount of loss should be? From TCP parameters? Or via network probing, for example with iperf?

I don't think calculating the loss is easily possible. You'd need to create a very exact model of your environment, including the device implementations and configurations, and with all the expected workload scenarios. In reality, there are too many parameters that you'd need to watch.

I suppose a practical way is to actually measure the amount of packet loss (=duplicate transmissions on the source) in some model cases (or simply over a longer time period) and then estimate a threshold for a "good" state. You can also correlate endpoint variables (CPU, memory, storage loads, service threads, ...). Note that hosts and their configurations are off topic here, however.

In case of rapidly changing workloads, the packet loss can well amount to a few percent.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.