1

According to why ICMP Time Exceeded packet takes longer ICMP Echo Reply, it is normal for ICMP time exceeded be treated as low priority and slowed down or even discarded. I'm observing this problem on a lightly loaded PowerConnect 7048 and I would like to bump up the priority so that those packets aren't discarded.

My topology is as follows:

   HostA                   PC7048 (routing)               HostB
192.168.0.1/24 ----- 192.168.0.2/24,10.10.0.2/24 ----- 10.10.0.1/24
  • When I ping from 192.168.0.1 to 10.10.0.1 I get 0% packet loss
  • When I ping from 192.168.0.1 to 10.10.0.2 I get 0% packet loss
  • When I use MTR from 192.168.0.1 to 10.10.0.1 I get 5% to 30% packet loss on the first hop. By doing tcpdump I confirmed that the way MTR seems to work is by shortening TTL and listening for ICMP time exceeded response (same approach like plain traceroute). This confirms that I have the same problem as why ICMP Time Exceeded packet takes longer ICMP Echo Reply

So I understand why this is happening. Or at least kind of. It's suppose to drop ICMP time exceeded packets when it's "busy" except I would not call my PC7048 busy at all. It's pushing at most 2Gbps spread across about 15 active ports - so in my opinion almost idle. It also does OSPF routing with approx 120 routes in the routing table, so it does this in L3 but still, that's not too much to ask from this model of a switch. Is it?

This is my processor usage

switch#show process cpu

Memory Utilization Report

status      bytes
------ ----------
  free  559607072
 alloc  435529120



CPU Utilization:


  PID      Name                    5 Secs     60 Secs    300 Secs
-----------------------------------------------------------------
 53923d0 tNet0                      0.00%       0.09%       0.09%
 53b99f0 BusM A                     0.16%       0.12%       0.10%
 55c1ab0 ipnetd                     0.00%       0.01%       0.00%
 5637b80 envMonTask                 8.71%       3.22%       1.83%
 5641370 osapiTimer                 0.00%       0.10%       0.08%
 568bba0 bcmDPC                     0.00%       0.03%       0.04%
 5926150 bcmL2X.0                   3.01%       3.18%       3.15%
 5b34440 bcmCNTR.0                  1.84%       1.49%       1.55%
 5bb3620 bcmTX                      0.00%       0.02%       0.04%
 63468d0 bcmRX                      1.84%       1.66%       1.72%
 6366ac0 bcmNHOP                    0.16%       0.04%       0.02%
 6379390 bcmATP-TX                  0.00%       0.02%       0.03%
 6382890 bcmATP-RX                  0.00%       0.07%       0.05%
 695e640 MAC Send Task              0.00%       0.03%       0.03%
 6967b40 MAC Age Task               0.16%       0.02%       0.01%
 8280660 bcmLINK.0                  0.00%       0.09%       0.14%
 850b8b0 LOG                        0.00%       0.01%       0.01%
 b822e50 tL7Timer0                  0.00%       0.01%       0.02%
 b848740 osapiMonTask               0.00%       0.00%       0.02%
 c9a7f90 servPortMonTask            0.00%       0.01%       0.00%
 cb1a310 portMonTask                0.00%       0.00%       0.01%
 cb44a80 simPts_task                0.00%       0.01%       0.04%
 d25df40 dtlTask                    0.50%       0.31%       0.28%
 d2b0f50 hapiRxTask                 0.00%       0.12%       0.14%
 d2da220 emWeb                      0.00%       0.02%       0.00%
 dd953c0 hapiL3AsyncTask            0.00%       0.49%       0.46%
 e6cdc50 trafficStormControl        0.00%       0.01%       0.02%
 e9f14c0 DHCP snoop                 0.16%       0.03%       0.03%
 f0b8930 SNMPTask                   0.00%       0.04%       0.03%
109f7730 dot1s_timer_task           0.50%       0.35%       0.34%
10a08a00 dot1s_task                 0.00%       0.00%       0.01%
11dd3920 tacacs_rx_task             0.16%       0.02%       0.00%
11de64b0 unitMgrTask                0.00%       0.02%       0.01%
11f865a0 snoopTask                  0.33%       0.16%       0.14%
12f18930 dhcpsPingTask              0.00%       0.00%       0.01%
13055e50 spmTask                    0.16%       0.03%       0.02%
130dcbe0 ipMapForwardingTask        2.51%       2.70%       2.97%
13327130 tArpCallback               0.00%       0.13%       0.15%
136bfd30 OSPF Proto                 0.50%       0.12%       0.08%
137788d0 ARP Timer                  1.50%       1.45%       1.54%
1bee9f50 lldpTask                   0.00%       0.31%       0.34%
1c90dce0 tCptvPrtl                  0.00%       0.01%       0.02%
1da0a130 RMONTask                   0.16%       0.10%       0.10%
1da26e50 boxs Req                   0.00%       0.00%       0.01%
1e9d81f0 OSPF Receive               0.00%       0.03%       0.06%
1ea71060 sshd[0]                    0.00%       0.03%       0.01%
-----------------------------------------------------------------
 Total CPU Utilization             22.44%      16.96%      15.98%

What resource is the switch running out of that it can't send back a few ICMP time exceeded packets? How do I prioritize ICMP time exceeded packets so that the switch knows these are very important for me and treats them accordingly? The reason these packets are very important to me is that when clients see this, they incorrectly conclude that there is a fault in my network. I could send them to this article that explains why, but the questions remains in their head "why am I paying so much money for oversubscribed switch that's so busy it can't spare a millisecond to send back a couple ICMP time exceeded packets?". In other words it looks really bad when the only row in a trace that shows packet loss belongs to you. How come everyone else is able to get 0% packet loss?

Improve this question
1
  • Often, a vendor will rate limit those types of ICMP packets to prevent DoS attacks based on them. For example, Cisco rate limits certain ICMP packets that a router would generate to one each 500 ms (half a second). It is possible that someone could tie up your router CPU with ICMP generation if there were not a limit.
    – Ron Maupin ♦
    Jan 19, 2021 at 21:31

1 Answer 1

Reset to default
2

A switch forwards in hardware but ICMP messages are handled by the CPU - so you'd need to look at the CPU load, not the amount of regular traffic.

OSPF messages are created and parsed by the CPU as well, but once their content is translated to the routing table, hardware again takes over the actual routing. Accordingly, the amount of routing doesn't matter for the CPU, only the number of peers and their message sizes.

Prioritizing ICMP messages would only be necessary if they were dropped due to high load. I don't think that's commonly possible but the cause is something else here.

Very often, outgoing ICMP messages are limited in frequency to reduce CPU utilization. So, MTR might just be sending too many probe packets for all of them to get a TTL expired message.

On the 7000 series, you can configure the ICMP rate limits using

ip icmp error-interval [burst-interval] [burst-size]

Limit the rate at which IPv4 ICMP error messages are sent.

  • burst-interval — How often the token bucket is initialized (Range: 0–2147483647 milliseconds).
  • burst-size — The maximum number of messages that can be sent during a burst interval (Range: 1–200).

https://downloads.dell.com/manuals/common/powerconnect-7024_user%27s%20guide_en-us.pdf

Improve this answer
7
  • This looked promising, but it didn't work. I tried "ip icmp error-interval 0 200" -> no improvement. "1 200" -> no improvement. "10000 1" -> blocks almost all ICMP errors (as it should) ... so at least I know the command works. There is still something slowing it down. I also tried flooding it with icmp errors (ping -f -t1) but to my surprise the loss percentage is about same when flooding it as when I'm not flooding it. To me this means that the switch is too busy to deal with even few ICMP error replies one second and it's able to handle a flood the next second without problem.
    – user3280964
    Jan 20, 2021 at 20:58
  • try ip icmp error-interval 1000 10, allowing 10 ICMP messages each second which should be enough for testing. You don't want the rate to be too high to prevent DoS-type attacks.
    – Zac67 ♦
    Jan 20, 2021 at 21:03
  • ip icmp error-interval 1000 10 approx 60% loss. ip icmp error-interval 0 200 approx 30% loss.
    – user3280964
    Jan 20, 2021 at 21:17
  • There may be a point where the CPU is actually too busy caring for ICMP, with those rates...
    – Zac67 ♦
    Jan 21, 2021 at 5:51
  • ok, I'm accepting your answer because you did answer my posted question. But now the new question is "What is making CPU so busy that it's unable to respond with ICMP errors?" my CPU usage never goes over 30%
    – user3280964
    Jan 21, 2021 at 16:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.