4

We are currently having issues with our WebService being intermittently inaccessible to clients. During diagnosis with the NOC and with some internal testing we believe we have a router dropping packets.

Our system has 4 Windows Server 2003 machines connected via a switch on one port (internal communication) and a router (Cisco RV082) on the second port (web traffic).

From our WebServer, if we WinMTR to the Router, managed switch or over the internal network (5.x) we get a 0% packet loss.

However, when we run the same WinMTR tool to our gateway, we receive a 0% packet loss on the gateway, but as of last test a 9% packet loss on the router.

|------------------------------------------------------------------------------------------

|                                      WinMTR statistics   

|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------

|
|                             192.168.1.1 -    9 | 1986 | 1818 |    0 |    0 |    1 |    0

|      static-xxx-xx-x-xxx.xxx.xxxxxx.xxx -    0 | 3667 | 3667 |    0 |    0 |   76 |    0 

|________________________________________________|______|______|______|______|______|______

   WinMTR v0.92 GPL V2 by Appnor MSP

I'm not entirely sure how to process this information. Does this imply that there is a break between the router and the gateway as the router seems fine on it's own? Or is it a problem with the router not responding properly when we go to the WAN port.

Edit

The existing problem server goes from 

Server -> Router -> dumb switch -> gateway

We have another server in the same cabinet that goes 

Server -> different router -> dumb switch -> gateway

Thank you in advance.

Improve this question
5
  • Statistics directly from the router would help you a lot. What's the traffic load on the router's interface (idle, average, saturated?). does the router show any errors in the inbound interface (CRC/buffer drops, etc) or the outbound interface (queue drops, etc) NB: If your team isn't the one managing that router, just ask for those statistic (say it's to prove to your boss it's not the router). If someone forgot to check those, the issue might simply go away (c'mon, I know that US, network guys aren't perfect either). Ideally, this should be done at each hops
    – Remi Letourneau
    Jun 6, 2014 at 19:17
  • The router is a fairly low level router (Cisco RV082) and doesn't show much in terms of diagnostic information. "Idling" there are about 200 devices connecting to it every 30 seconds, fully saturated would be the same 200 devices (every 5 seconds) + client access (20-30 every second or so) . It reports no packet errors in the last nearly 5,000,000 packets, but beyond that there is unfortunately, no information accessible.
    – Trevor Watson
    Jun 6, 2014 at 20:49
  • What about if you test through your router but not ending up on it? If you can access the internet try your ISP's device (your router's next hop) As this is a low end router, it might not have the horsepower to handle all the traffic you're sending to it's control plane (it's own IP), but might not have a problem routing packets destined for somewhere else than itself, giving you a false positive (NB: that can happen even on middle/high end gear - control plane doesn't always have the horsepower to process high traffic tests)
    – Remi Letourneau
    Jun 6, 2014 at 21:03
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer.
    – Ron Maupin ♦
    Aug 8, 2017 at 21:08
  • I see the same problem. I have 20-40% package loss on my gateway (router). I tried to switch between another router, but I observe the same problem. I can't replicate the problem with ping (which is also ICMP).
    – Melroy van den Berg
    Dec 2 at 23:39

3 Answers 3

Reset to default
7

However, when we run the same WinMTR tool to our gateway, we receive a 0% packet loss on the gateway, but as of last test a 9% packet loss on the router...
[snip]
... I'm not entirely sure how to process this information...

You've misunderstood how WinMTR should be used; this is a common problem. The first line you should look at in any mtr output is the last line; you got back all 3667 packets that you sent.

Summary

You are seeing some transit packet loss.

However, there is no packet loss to your gateway endpoint.  

Why would some transit traffic be dropped with no endpoint loss?

It's not uncommon for transit devices to randomly drop some fraction of traceroute traffic. There are several reasons why it may happen... among them:

  • ICMP error messages have been rate-limited by the OS on the Cisco RV082
  • The RV082 is busy doing other things and isn't bothering to respond to some of the mtr queries
  • There is a bug
  • Some combination of the above

Rhetorical question

  • Q: What should you do if you have no packet loss to your server?
  • A: Other priorities; there is nothing to see here, move on.
Improve this answer
2
  • Thanks Mike! I knew it was a non-issue in practice, but it's nice to see a few reasons why.
    – Smithers
    Apr 20, 2015 at 20:33
  • @Smithers is it a non-issue? I do see the same thing happening. I even switched between routes. Connect directly to the router. At this point I'm suspecting mtr itself.
    – Melroy van den Berg
    Dec 2 at 23:44
0

The only way to determine what is happening exactly is to get a packet capture of the traffic so you can see what the router is actually doing. Most modern routers will de prioritise icmp traffic in favour of forwarding actual traffic if they are under load. As others have pointed out this particular router is a fairly low level device so this could be the case. I have seen this exact issue with a (fairly powerful) router reporting packet loss when testing with MTR when in fact no loss existed.

Improve this answer
1
  • 1
    I remember thar Cisco's Nexus 5K were using Celerons for the control plane... CPU were spiking to 100% each time we logged-on to configure/check something and was non-responsive for a few seconds. Of course, it was a bug, but it gives an idead of how even high end gear can have problem of that type
    – Remi Letourneau
    Jun 7, 2014 at 0:34
0

I found a core backbone router in London doing precisely this rate limiting of ICMP just now, as I ran a trace to ubuntu.com from New Zealand, this on macOS 10.13 mtr. If I ping ubuntu.com I don't see any loss and the site is performant when visited. But the boxen known as level3-ic-333036-las-b21.c.telia.net is biffing 78% of my mtr pings which are 60/minute by the default. Truth be told is probably a well setup router, although it seems heavily loaded with the longest reply taking 2100 ms and pretty large deviation of 321 ms! It is 10:26 am GMT on a Tuesday in this nearly 4 minute trace. Auckland --> Los Angeles --> London. I can see that mcclure.canonical.com and bundle-200.cor02.lax01.ca.vocus.net are probably the fastest machines with only 0.4 ms of deviation in their replies!

                                                                          My traceroute  

 My traceroute  [vUNKNOWN]
MODERNA-51.local (10.0.0.15)                                            2019-07-16T22:26:26+1200
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                        Packets               Pings
 Host                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. orcon                                              0.0%   233    0.7   0.7   0.5   1.3   0.2
 2. default-rdns.vocus.co.nz                          52.2%   233  9232. 9136. 8356. 9949. 283.1
 3. ae8-20.cpcak-mdr-r1.vocus.co.nz                    0.0%   233   13.8  15.3  13.4  56.7   5.6
 4. ae9-20.cpcak-mdr-r1.vocus.co.nz                    0.0%   233   13.7  15.0  13.5  49.9   4.0
 5. be-101.bdr02.akl05.akl.vocus.net.nz                0.0%   233   15.2  17.6  14.1  55.2   8.6
 6. bundle-10.cor01.akl05.akl.vocus.net.nz             0.0%   233  138.5 138.7 138.2 146.6   0.6
 7. bundle-200.cor02.lax01.ca.vocus.net                0.0%   232  138.6 138.5 138.1 142.2   0.4
 8. bundle-102.bdr02.lax01.ca.vocus.net                0.0%   232  139.1 139.2 138.1 158.4   1.6
 9. las-b24-link.telia.net                             0.0%   232  139.6 140.0 137.5 167.2   5.1
10. las-b21-link.telia.net                             0.0%   232  139.1 139.0 138.4 160.1   1.5
11. level3-ic-333036-las-b21.c.telia.net              78.0%   232  138.3 202.3 137.9 2100. 321.5
12. ???
13. source-mana.ear2.london1.level3.net                0.0%   232  269.0 271.7 268.7 373.2  13.0
14. mcclure.canonical.com                              0.0%   232  269.1 268.9 268.4 270.8   0.4
15. www-ubuntu-com.nuno.canonical.com                  0.0%   232  269.3 269.1 268.5 272.7   0.5
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.