mtr output with high packet loss on one hop

Question

I'm investigating a complaint of poor performance from an end user accessing a site I help maintain.

I have these two mtr outputs to the end user, the first from the site:

                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 198.199.92.253                    0.0%   200    7.3   4.2   0.9  89.6   8.0
 2. 69.22.130.37                      0.0%   200    0.4   2.5   0.3  51.4   8.0
 3. 69.22.143.170                    42.5%   200    1.3   2.0   1.1  47.9   4.9
 4. 69.22.143.165                     0.0%   200    2.3   6.4   1.6  56.9   9.7
 5. 206.223.116.86                    0.0%   200    2.5   3.0   1.8  14.1   1.5
 6. 64.125.24.1                       0.0%   200    3.0   6.9   2.2  65.7   9.9
 7. 64.125.26.230                     0.0%   200   56.3  61.4  55.6 119.0  10.0
 8. 64.125.24.33                      0.5%   200   76.4  78.9  76.0 116.1   7.1
 9. 64.125.29.38                      0.0%   200   73.9  77.5  73.4 238.8  13.4
10. 64.125.31.181                     0.5%   200  160.0 159.4 156.2 181.4   4.3
11. 64.125.32.93                      0.0%   200  156.9 157.8 155.9 217.0   6.8
12. 62.253.174.190                    0.0%   200  166.0 166.5 165.6 172.8   1.2
13. 62.253.175.217                    0.0%   200  162.1 163.1 161.7 200.4   4.2
14. 213.105.159.194                   0.0%   200  163.8 165.0 163.4 241.2   8.3
15. 81.97.112.218                     0.0%   200  164.7 166.5 164.5 220.0   7.4
16. ???

and the second from my network:

                                       Packets               Pings
 Host                                Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. 216.16.235.1                      0.0%   115    0.4   0.4   0.3   0.6   0.1
 2. 216.16.232.37                     0.0%   115    0.9   0.9   0.6  18.8   1.7
 3. 216.16.255.141                    0.0%   115    0.8   0.7   0.6   1.1   0.1
 4. 216.16.255.130                    0.0%   114    1.0   1.0   0.8   1.4   0.1
 5. 24.153.3.141                      0.0%   114    1.9   3.6   1.5   6.5   1.2
 6. 64.71.241.97                      0.0%   114    3.4   5.3   3.3   7.4   1.1
 7. ???
 8. 64.124.128.193                   34.5%   114   18.9  19.2  18.7  39.0   2.3
 9. 64.125.21.74                      0.0%   114   19.2  20.4  18.9  49.9   5.0
10. 64.125.29.38                      0.0%   114   19.3  23.1  19.2  48.3   7.6
11. 64.125.27.186                     0.9%   114  105.2 106.1 105.1 137.8   3.2
12. 64.125.32.93                      0.0%   114  106.3 107.1 106.1 163.3   5.8
13. 62.253.174.190                    0.0%   114  181.8 123.5 115.8 206.3  20.9
14. 62.253.175.217                    0.0%   114  113.8 114.8 113.6 144.3   4.2
15. 213.105.159.194                   0.0%   114  115.3 115.9 115.2 163.5   4.6
16. 81.97.112.218                     0.0%   114  113.3 114.4 113.2 140.7   4.5
17. ???

How should I interpret those hops with MASSIVE packet loss? I'm inclined to think those hops have some sort of asymmetric routing going on and one of the paths is congested.

Could this cause any problem for the end user?

Answer 1

MTR uses ICMP, which is often rate limited on the internet due to how it can be misused to created problems (high CPU, wasted bandwidth, DoS, etc).

When you see output like this, generally this is a sign that rate limiting for ICMP is in place. With a quick web search I found this documentation regarding using MTR. While it is not official it looks decent (at least with a quick scan) and provides examples of some problems you may find using MTR.

Answer 2

As @YLearn already wrote ICMP ratelimiting on the router with packetloss might well be the cause, because replying to ICMP requests is done in CPU while forwarding of packets is usually done in the ASICs. So this does not need to be a problem at all.

A very good guide on interpreting traceroute (and MTR) output was written by Richard Steenbergen a few years ago. He did a nice presentation on it at NANOG.

Answer 3

I'll preface this by saying that yes, the routing you mention could be part of it. It would be the route BACK to your host from that hop that is taking a congested path that the others aren't.

My guy thinking: if this was a problem in the data plane on that particular router, I would expect to see packet loss for all hops after this hop - but you don't see that.

When the TTL of a packet reaches 0, it's up to control plane on the router to generate the ICMP response back to the sending host. My guess is that the control plane CPU (at the point in time you were performing the trace) on that particular router is highly utilised, and its sending some responses back to you outside the timeout value of MTR.