16

I have an HP c7000 blade chassis which contains Cisco 3120X and Cisco 3120G switches running ios 12.2(58)SE1. The blades themselves are very lightly loaded yet many interfaces on different blade switches in the chassis show a fairly high number of output drops. If I check the number of output drops repeatedly I not only see the counter increasing but sometimes it decreases. The numbers don't correlate with the packets/s recorded on the interface. QoS settings are default for the platform.

The following samples were all taken within a 30 second period:

bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 451110
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 451110
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 902220
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1353330
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1804440
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1804440
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1804440
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 451490

bc1019-3120-stack>sh int gi2/0/7 | i output rate
  5 minute output rate 301000 bits/sec, 119 packets/sec

1) Is there anything else that can cause output drops besides the server nic not receiving the frames quickly enough?

2) What is the maximum number of output drops the interface counter can record? Does it rollover when it reaches the max?

3) What would be considered a healthy rate of output drops?

2
  • As Leonardo Abdalla pointed out, the erratic output drops seen on our blade chassis are a result of bug CSCtq86186
    – User123456
    May 23, 2013 at 12:08
  • It's a bug. We hit the same thing, upgraded to c3750e-universalk9-mz.150-2.SE4.bin and all is well. JB
    – user4297
    Feb 13, 2014 at 22:37

5 Answers 5

14

Unless someone is clearing counters you should never see any odometer-type counters (those that increment based on a packet action) decrease, they should always increase. That part sounds like a bug.

As far as what causes output drops in particular, there are so many different causes that it's very difficult to pinpoint it exactly. Sometimes there is congestion inside the switch's backplane and those might show up as output drops on the outgoing interface. In rare circumstances you can also get microbursts that don't show up when polled at 1 minute intervals that quickly overload the interface, but then drop back down very quickly. I would suggest grabbing the SNMP OID for output drops and then graphing that and see how it corresponds to the CLI counter.

Generally speaking, you don't want any output drops as they indicate a packet that did not make it to its destination. But, if you're running your links hot (which you say you're not) they're unavoidable to an extent, mostly due to interior switch buffering, etc.

2
  • I'm wondering if there's so many drop outs in this case, the counters wrap around.
    – nos
    May 10, 2013 at 15:29
  • 1
    They are 32bit counters, so you're getting nowhere near the limits. (and possibly 64bit internally)
    – Ricky
    May 19, 2013 at 20:13
8

My first thought is unicast flooding, especially if the counters increment in unison across a number of ports in the same vlan. I agree with Aaron that the counter decrementing sounds like a bug. The counter will probably roll over at 2^64, but that won't happen within seconds. I would consider a healthy rate of output drops to be zero, but this is not realistic--even in the datacenter. Are you doing 10G uplinks?

2
  • Yes, one 10gig uplink from each of the two 3120X's in the blade chassis (one port blocked due to stp)
    – User123456
    May 9, 2013 at 21:05
  • Just as a 1G uplink will easily overwhelm a 100M downlink, I'm sure the same is true of 10G/1G. This is especially true when unicast flooding occurs. I doubt unicast flooding would be evident in bandwidth/pps stats. May 9, 2013 at 22:49
5

Seems like you are hitting bug CSCtq86186. This bug has been found on 3750s, 2960s, but it may be affecting the blade switches as well.

1
  • This is exactly the bug we are hitting on our 3120's - fixed in 15.0(2)SE. Thanks!
    – User123456
    May 23, 2013 at 12:04
4

If you are experiencing unicast flooding, running wireshark on one of the hosts or spanning one of the ports should show that pretty quickly.

It sounds like you have redundant cores in a square topology? If so, try adding this command to your vlan interface:

arp timeout 300

CAM tables hold entries for 5 minutes while ARP tables are kept for four hours (defaults). Setting the ARP to match the CAM may eliminate the unicast flooding at the expense of a slight increase to the CPU. Catalyst 6500/6000 Switches ARP or CAM Table Issues Troubleshooting

1

Output drops are rather common on smaller switches with small buffers as any burst will deplete the buffer. I'm not really familiar with the 3120 so I can't speak for the size of it's buffer, but at least that's a common reason till why one could get output drops.

The specifics reasons are head of line blocking (HOLB), where multiple source ports are sending to one destination and so we get congestion. Another common reason is when going from a higher port speed to a lower one, ie 10G to 1G or 40G to 10G.

I recommend you run show controllers ethernet-controller X where X is your port. You should get some information regarding output drops, like if something is trying to output to large frames, which could happen if you don't have consistent MTU across your network.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.