0

According to How to prioritize ICMP time exceeded packets my switch is dropping ICMP error replies because the CPU is apparently too busy.

It is a PowerConnect 7048 and it dropping anywhere between 5% to 30% of ICMP error replies. If you wish to see my topology and additional details, they are available in the above link.

This is what my CPU usage looks like

switch#show process cpu

Memory Utilization Report

status      bytes
------ ----------
  free  559607072
 alloc  435529120



CPU Utilization:


  PID      Name                    5 Secs     60 Secs    300 Secs
-----------------------------------------------------------------
 53923d0 tNet0                      0.00%       0.09%       0.09%
 53b99f0 BusM A                     0.16%       0.12%       0.10%
 55c1ab0 ipnetd                     0.00%       0.01%       0.00%
 5637b80 envMonTask                 8.71%       3.22%       1.83%
 5641370 osapiTimer                 0.00%       0.10%       0.08%
 568bba0 bcmDPC                     0.00%       0.03%       0.04%
 5926150 bcmL2X.0                   3.01%       3.18%       3.15%
 5b34440 bcmCNTR.0                  1.84%       1.49%       1.55%
 5bb3620 bcmTX                      0.00%       0.02%       0.04%
 63468d0 bcmRX                      1.84%       1.66%       1.72%
 6366ac0 bcmNHOP                    0.16%       0.04%       0.02%
 6379390 bcmATP-TX                  0.00%       0.02%       0.03%
 6382890 bcmATP-RX                  0.00%       0.07%       0.05%
 695e640 MAC Send Task              0.00%       0.03%       0.03%
 6967b40 MAC Age Task               0.16%       0.02%       0.01%
 8280660 bcmLINK.0                  0.00%       0.09%       0.14%
 850b8b0 LOG                        0.00%       0.01%       0.01%
 b822e50 tL7Timer0                  0.00%       0.01%       0.02%
 b848740 osapiMonTask               0.00%       0.00%       0.02%
 c9a7f90 servPortMonTask            0.00%       0.01%       0.00%
 cb1a310 portMonTask                0.00%       0.00%       0.01%
 cb44a80 simPts_task                0.00%       0.01%       0.04%
 d25df40 dtlTask                    0.50%       0.31%       0.28%
 d2b0f50 hapiRxTask                 0.00%       0.12%       0.14%
 d2da220 emWeb                      0.00%       0.02%       0.00%
 dd953c0 hapiL3AsyncTask            0.00%       0.49%       0.46%
 e6cdc50 trafficStormControl        0.00%       0.01%       0.02%
 e9f14c0 DHCP snoop                 0.16%       0.03%       0.03%
 f0b8930 SNMPTask                   0.00%       0.04%       0.03%
109f7730 dot1s_timer_task           0.50%       0.35%       0.34%
10a08a00 dot1s_task                 0.00%       0.00%       0.01%
11dd3920 tacacs_rx_task             0.16%       0.02%       0.00%
11de64b0 unitMgrTask                0.00%       0.02%       0.01%
11f865a0 snoopTask                  0.33%       0.16%       0.14%
12f18930 dhcpsPingTask              0.00%       0.00%       0.01%
13055e50 spmTask                    0.16%       0.03%       0.02%
130dcbe0 ipMapForwardingTask        2.51%       2.70%       2.97%
13327130 tArpCallback               0.00%       0.13%       0.15%
136bfd30 OSPF Proto                 0.50%       0.12%       0.08%
137788d0 ARP Timer                  1.50%       1.45%       1.54%
1bee9f50 lldpTask                   0.00%       0.31%       0.34%
1c90dce0 tCptvPrtl                  0.00%       0.01%       0.02%
1da0a130 RMONTask                   0.16%       0.10%       0.10%
1da26e50 boxs Req                   0.00%       0.00%       0.01%
1e9d81f0 OSPF Receive               0.00%       0.03%       0.06%
1ea71060 sshd[0]                    0.00%       0.03%       0.01%
-----------------------------------------------------------------
 Total CPU Utilization             22.44%      16.96%      15.98%
  • I have tried running show process cpu in a loop while looking at MTR and to try to correlate the times when it's dropping to the times that it's fine, but I could not see any correlation. Sometimes it's dropping when CPU is higher and sometimes it's dropping when CPU is lower (CPU always in the range 15 to 30% busy though).
  • My MTR looks something like this ...??..?..???....?..?...???.....??????...?....? in other words it can get busy for anywhere between 1 to 10 seconds and I've never seen it be busy for more than 10 seconds at a time
  • I also drilled down to envMonTask which has up to 15% spikes lasting a few seconds, but again no correlation with drops observed on MTR. Sometimes envMonTask is busy and yet it's not dropping.

So drilling down by watching the pattern didn't work. What is the cause of this problem? And how do I fix it?

6
  • Have you tried using a shell command, i.e., traceroute? Do you get the same results? – Ron Trunk Jan 21 at 16:27
  • Yes, same about 5 to 30% chance of a drop – user3280964 Jan 21 at 16:33
  • Often, a vendor will rate limit those types of ICMP packets to prevent DoS attacks based on them. For example, Cisco rate limits certain ICMP packets that a router would generate to one each 500 ms (half a second). It is possible that someone could tie up your router CPU with ICMP generation if there were not a limit. Imagine with no such limit that a person with a a dozen machines is sending packets designed to create ICMP errors. Your switch CPU could easily get overloaded. – Ron Maupin Jan 21 at 16:33
  • That doesn't add up on this particular switch because I can also do a ping -f -t1 with the intention of flooding the switch to see what happens. What happens is the same pattern, it answers all those hundreds of pings with an ICMP error one second and then nothing next second (random). If it was a hard limit, it would be more like ......??????......??????.......??????...... and yes I've seen that (what you are describing) on other devices. This isn't it. – user3280964 Jan 21 at 16:36
  • Configuration? What else is talking to it? It sounds like something is hammering SNMP. (RMON?) – Ricky Jan 22 at 2:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.