How to find the source of increased latency?

Question

I've got monitoring setup on several devices in our office. The ping response time to small access switches is commonly 1-4ms... As of 3AM this morning, this has sky-rocketed to 300ms on average.

Where does one start looking in a situation like this? What things can I observe in the switch to find the source of latency?

NOTE: It's not load related.. all the links bandwidth usage is normal and unaffected, most links are very under utilized. Also - monitoring is local to the devices reporting the latency so there is no WAN factor here.

Answer 1

First, latency isn't directly tied to bandwidth. There are many reasons why a device would delay a packet other than a congested link.

Have you attempted a traceroute? This will show you the latency between hops, if you're looking for a L3 boundary as a suspect.

You might also check to see if any of the devices in the path have a significant usage of CPU/RAM.

Answer 2

if this is just based on the LAN, there's a few things you can do to start off to try and find out what is causing this:

• Show process cpu history command: if the CPU usage is very high, then you need to see which process is causing this and perhaps hit google with the offending process.

• show debug command: a common cause I've found is people leaving debug commands running on the switch. A common favourite was IP accounting being left on devices that were already over-utilized. Use "undebug all" to get rid of the debugs.

• Give it a reboot: probably not possibly during the day, but use the "reload in" command to time it at night or over the weekend. You'd be surprised how many issues a quick reboot can fix.

• shut trunk ports - If its a L3 switch, another common issue I've seen is too much traffic using this device for routing between VLANs. If possible, temporarily shut some of the trunk ports to see if this reduces the latency.

Its good to be aware that your pings are low priority, in regards to latency and also when being processed by the CPU. Might also be a good idea to double check your QoS settings and make sure there's no silly mistakes causing this, as much as that is unlikely.

Answer 3

I use cacti to monitor bandwidth, and openNMS to monitor latency. If you are monitoring all the devices linked to this switch you may see a corollary between usage and the latency. (i know you said it is not a bandwidth issue, but you never now) I have seen lower-end switches sag under heavy usage, which causes a lot of latency. Do you have any "dumb" devices feeding this switch that may be the source of the sag even though this switch is not passing much traffic. Also with cacti you may be able to poll the CPU usage, and you may see a spike at the time of the latency.

As mentioned above, MTR or neotrace are also useful to keep an eye on the situation and you may see where the latency starts, which may not be this switch itself.

Answer 4

If this is not happening on LAN, you could limit the "wan port" throghtput, this will force a better TDM. Try something arround 80% of your maximun throughput and see if it helps. You may need to tweek depending the amount of terminals.