According to How to prioritize ICMP time exceeded packets my switch is dropping ICMP error replies because the CPU is apparently too busy.
It is a PowerConnect 7048 and it dropping anywhere between 5% to 30% of ICMP error replies. If you wish to see my topology and additional details, they are available in the above link.
This is what my CPU usage looks like
switch#show process cpu
Memory Utilization Report
status bytes
------ ----------
free 559607072
alloc 435529120
CPU Utilization:
PID Name 5 Secs 60 Secs 300 Secs
-----------------------------------------------------------------
53923d0 tNet0 0.00% 0.09% 0.09%
53b99f0 BusM A 0.16% 0.12% 0.10%
55c1ab0 ipnetd 0.00% 0.01% 0.00%
5637b80 envMonTask 8.71% 3.22% 1.83%
5641370 osapiTimer 0.00% 0.10% 0.08%
568bba0 bcmDPC 0.00% 0.03% 0.04%
5926150 bcmL2X.0 3.01% 3.18% 3.15%
5b34440 bcmCNTR.0 1.84% 1.49% 1.55%
5bb3620 bcmTX 0.00% 0.02% 0.04%
63468d0 bcmRX 1.84% 1.66% 1.72%
6366ac0 bcmNHOP 0.16% 0.04% 0.02%
6379390 bcmATP-TX 0.00% 0.02% 0.03%
6382890 bcmATP-RX 0.00% 0.07% 0.05%
695e640 MAC Send Task 0.00% 0.03% 0.03%
6967b40 MAC Age Task 0.16% 0.02% 0.01%
8280660 bcmLINK.0 0.00% 0.09% 0.14%
850b8b0 LOG 0.00% 0.01% 0.01%
b822e50 tL7Timer0 0.00% 0.01% 0.02%
b848740 osapiMonTask 0.00% 0.00% 0.02%
c9a7f90 servPortMonTask 0.00% 0.01% 0.00%
cb1a310 portMonTask 0.00% 0.00% 0.01%
cb44a80 simPts_task 0.00% 0.01% 0.04%
d25df40 dtlTask 0.50% 0.31% 0.28%
d2b0f50 hapiRxTask 0.00% 0.12% 0.14%
d2da220 emWeb 0.00% 0.02% 0.00%
dd953c0 hapiL3AsyncTask 0.00% 0.49% 0.46%
e6cdc50 trafficStormControl 0.00% 0.01% 0.02%
e9f14c0 DHCP snoop 0.16% 0.03% 0.03%
f0b8930 SNMPTask 0.00% 0.04% 0.03%
109f7730 dot1s_timer_task 0.50% 0.35% 0.34%
10a08a00 dot1s_task 0.00% 0.00% 0.01%
11dd3920 tacacs_rx_task 0.16% 0.02% 0.00%
11de64b0 unitMgrTask 0.00% 0.02% 0.01%
11f865a0 snoopTask 0.33% 0.16% 0.14%
12f18930 dhcpsPingTask 0.00% 0.00% 0.01%
13055e50 spmTask 0.16% 0.03% 0.02%
130dcbe0 ipMapForwardingTask 2.51% 2.70% 2.97%
13327130 tArpCallback 0.00% 0.13% 0.15%
136bfd30 OSPF Proto 0.50% 0.12% 0.08%
137788d0 ARP Timer 1.50% 1.45% 1.54%
1bee9f50 lldpTask 0.00% 0.31% 0.34%
1c90dce0 tCptvPrtl 0.00% 0.01% 0.02%
1da0a130 RMONTask 0.16% 0.10% 0.10%
1da26e50 boxs Req 0.00% 0.00% 0.01%
1e9d81f0 OSPF Receive 0.00% 0.03% 0.06%
1ea71060 sshd[0] 0.00% 0.03% 0.01%
-----------------------------------------------------------------
Total CPU Utilization 22.44% 16.96% 15.98%
- I have tried running
show process cpuin a loop while looking at MTR and to try to correlate the times when it's dropping to the times that it's fine, but I could not see any correlation. Sometimes it's dropping when CPU is higher and sometimes it's dropping when CPU is lower (CPU always in the range 15 to 30% busy though). - My MTR looks something like this
...??..?..???....?..?...???.....??????...?....?in other words it can get busy for anywhere between 1 to 10 seconds and I've never seen it be busy for more than 10 seconds at a time - I also drilled down to
envMonTaskwhich has up to 15% spikes lasting a few seconds, but again no correlation with drops observed on MTR. SometimesenvMonTaskis busy and yet it's not dropping.
So drilling down by watching the pattern didn't work. What is the cause of this problem? And how do I fix it?
traceroute? Do you get the same results?ping -f -t1with the intention of flooding the switch to see what happens. What happens is the same pattern, it answers all those hundreds of pings with an ICMP error one second and then nothing next second (random). If it was a hard limit, it would be more like......??????......??????.......??????......and yes I've seen that (what you are describing) on other devices. This isn't it.