14

I've got monitoring setup on several devices in our office. The ping response time to small access switches is commonly 1-4ms... As of 3AM this morning, this has sky-rocketed to 300ms on average.

Where does one start looking in a situation like this? What things can I observe in the switch to find the source of latency?

NOTE: It's not load related.. all the links bandwidth usage is normal and unaffected, most links are very under utilized. Also - monitoring is local to the devices reporting the latency so there is no WAN factor here.

  • 3
    Assuming this is a Cisco IOS switch... Please post show proc cpu history for the switch with the high ping-times. If that CPU is consistently high, or spiking high on a regular basis, run show proc cpu sort – Mike Pennington Jun 27 '13 at 21:36
  • Is the latency only towards the switch control-plane or do you get same latency when you ping something behind the switch? – ytti Jun 28 '13 at 7:08
  • @MikePennington - imgur.com/a/gfX9q#0 - this is very cool! Looks like it spikes pretty high up consistently though on average it's low .. – A L Jun 28 '13 at 15:52
  • @Ytti - didn't mean to post this on a separate line .. anyway - So I dug deeper into this. cp <-> cp response is actually low from distribution to access, or at least was at the time I tested. From an access level port to the devices on the access layer switches is where we're seeing the extreme latency. – A L Jun 28 '13 at 15:54
  • @user1353, thank you... that imgur that you posted is not consistently high enough to cause consistently increased ping times from CPU on that switch – Mike Pennington Jun 28 '13 at 16:09
6

First, latency isn't directly tied to bandwidth. There are many reasons why a device would delay a packet other than a congested link.

Have you attempted a traceroute? This will show you the latency between hops, if you're looking for a L3 boundary as a suspect.

You might also check to see if any of the devices in the path have a significant usage of CPU/RAM.

| improve this answer | |
  • I would agree with Mierdin and also recommend MTR for continuously running a traceroute in this sort of situation. Wikipedia Link: en.m.wikipedia.org/wiki/MTR_(software) – Brett Lykins Jun 27 '13 at 22:26
  • @Mierdin - Thanks for your feedback, so there is no L3 factor here, traceroute shows an initially high response of about 500ms, then 260ms, then 76ms arriving at the device - these are for each try on the same single hop, not for multiple hops. See my comment to MikePennington for the CPU related info. – A L Jun 28 '13 at 15:57
3

if this is just based on the LAN, there's a few things you can do to start off to try and find out what is causing this:

  • Show process cpu history command: if the CPU usage is very high, then you need to see which process is causing this and perhaps hit google with the offending process.

  • show debug command: a common cause I've found is people leaving debug commands running on the switch. A common favourite was IP accounting being left on devices that were already over-utilized. Use "undebug all" to get rid of the debugs.

  • Give it a reboot: probably not possibly during the day, but use the "reload in" command to time it at night or over the weekend. You'd be surprised how many issues a quick reboot can fix.

  • shut trunk ports - If its a L3 switch, another common issue I've seen is too much traffic using this device for routing between VLANs. If possible, temporarily shut some of the trunk ports to see if this reduces the latency.

Its good to be aware that your pings are low priority, in regards to latency and also when being processed by the CPU. Might also be a good idea to double check your QoS settings and make sure there's no silly mistakes causing this, as much as that is unlikely.

| improve this answer | |
  • Great feedback, I had already checked the show debug, and a reboot is not possible at this time. – A L Jun 28 '13 at 16:05
2

I use cacti to monitor bandwidth, and openNMS to monitor latency. If you are monitoring all the devices linked to this switch you may see a corollary between usage and the latency. (i know you said it is not a bandwidth issue, but you never now) I have seen lower-end switches sag under heavy usage, which causes a lot of latency. Do you have any "dumb" devices feeding this switch that may be the source of the sag even though this switch is not passing much traffic. Also with cacti you may be able to poll the CPU usage, and you may see a spike at the time of the latency.

As mentioned above, MTR or neotrace are also useful to keep an eye on the situation and you may see where the latency starts, which may not be this switch itself.

| improve this answer | |
0

If this is not happening on LAN, you could limit the "wan port" throghtput, this will force a better TDM. Try something arround 80% of your maximun throughput and see if it helps. You may need to tweek depending the amount of terminals.

| improve this answer | |
  • As i understand OP has clearly stated in the note, that this is not load related. – user36472 Dec 2 '17 at 13:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.