What causes total output drops on a cisco switch interface?

Question

I have an HP c7000 blade chassis which contains Cisco 3120X and Cisco 3120G switches running ios 12.2(58)SE1. The blades themselves are very lightly loaded yet many interfaces on different blade switches in the chassis show a fairly high number of output drops. If I check the number of output drops repeatedly I not only see the counter increasing but sometimes it decreases. The numbers don't correlate with the packets/s recorded on the interface. QoS settings are default for the platform.

The following samples were all taken within a 30 second period:

bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2255550
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 451110
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 451110
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 902220
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1353330
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1804440
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1804440
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 1804440
bc1019-3120-stack>sh int gi2/0/7 | i output drops
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 451490

bc1019-3120-stack>sh int gi2/0/7 | i output rate
  5 minute output rate 301000 bits/sec, 119 packets/sec

1) Is there anything else that can cause output drops besides the server nic not receiving the frames quickly enough?

2) What is the maximum number of output drops the interface counter can record? Does it rollover when it reaches the max?

3) What would be considered a healthy rate of output drops?

Answer 1

Unless someone is clearing counters you should never see any odometer-type counters (those that increment based on a packet action) decrease, they should always increase. That part sounds like a bug.

As far as what causes output drops in particular, there are so many different causes that it's very difficult to pinpoint it exactly. Sometimes there is congestion inside the switch's backplane and those might show up as output drops on the outgoing interface. In rare circumstances you can also get microbursts that don't show up when polled at 1 minute intervals that quickly overload the interface, but then drop back down very quickly. I would suggest grabbing the SNMP OID for output drops and then graphing that and see how it corresponds to the CLI counter.

Generally speaking, you don't want any output drops as they indicate a packet that did not make it to its destination. But, if you're running your links hot (which you say you're not) they're unavoidable to an extent, mostly due to interior switch buffering, etc.

Answer 2

My first thought is unicast flooding, especially if the counters increment in unison across a number of ports in the same vlan. I agree with Aaron that the counter decrementing sounds like a bug. The counter will probably roll over at 2^64, but that won't happen within seconds. I would consider a healthy rate of output drops to be zero, but this is not realistic--even in the datacenter. Are you doing 10G uplinks?

Answer 3

Seems like you are hitting bug CSCtq86186. This bug has been found on 3750s, 2960s, but it may be affecting the blade switches as well.

Answer 4

If you are experiencing unicast flooding, running wireshark on one of the hosts or spanning one of the ports should show that pretty quickly.

It sounds like you have redundant cores in a square topology? If so, try adding this command to your vlan interface:

arp timeout 300

CAM tables hold entries for 5 minutes while ARP tables are kept for four hours (defaults). Setting the ARP to match the CAM may eliminate the unicast flooding at the expense of a slight increase to the CPU. Catalyst 6500/6000 Switches ARP or CAM Table Issues Troubleshooting

Answer 5

Output drops are rather common on smaller switches with small buffers as any burst will deplete the buffer. I'm not really familiar with the 3120 so I can't speak for the size of it's buffer, but at least that's a common reason till why one could get output drops.

The specifics reasons are head of line blocking (HOLB), where multiple source ports are sending to one destination and so we get congestion. Another common reason is when going from a higher port speed to a lower one, ie 10G to 1G or 40G to 10G.

I recommend you run show controllers ethernet-controller X where X is your port. You should get some information regarding output drops, like if something is trying to output to large frames, which could happen if you don't have consistent MTU across your network.