Cisco N9K Output Discards

Question

We're experiencing output discards in couple of our N9K-C9372PXs. Seems like egress queues fills and it ends up in a lots of discarded packets.

All interfaces where discards occur are NEXUS interfaces (to FEX)

show interface ethernet 1/21 | inc "output discard" 
0 lost carrier 0  no carrier 0 babble 32043410 output discard

Bandwidth wise they're OK. I think it's due to bursty traffic from multiple uplinks (traffic comming from 6 Uplinks to 2 FEX ports). So, things like DNS queries - which're using UDP packets are really damaged by that. Could you say how to check the size of the buffer and how they're filling up ?

I'm wondering: is there are any workaround for this type of behavior ?

Now we're using:

show hardware qos ns-buffer-profile 
NS Buffer Profile: Burst optimized

i see there are commands like these:

hardware qos ns-buffer-profile ultra-burst

Maybe you know, would ultra-burst mode help in our situation ? And what is requirements on entering that command ? do it requires reboot ?

or maybe it's possible to make some type of queuing where we'd treat UDP packets differently ?

also we have few older N5K-C5672UP, N5K-C5548UP and them are not experiencing it. Maybe they have 'bigger hardware' buffers ?

Thanks for any input.

Answer 1

Nexus as cut-through switches are a different league than the store-and-forward switches many of us have become accustomed to. They must handle speed step-downs with different strategies than classic output queuing.

Downlinks to FEXes are usually LACP bundles of 10G. So if there are links to the given N9K-C9372PX running at 40G, there will be a speed stepdown to the FEX in question - and some packets will have to be discarded.

The following links might be quite steep an entry into the inner workings of the N9K family - probably still worth the time to read.

Cisco Nexus 9300 Platform Buffer and Queuing Architecture

From which:

Buffer Boost is an egress-port configuration property. It can be enabled or disabled on a per-port basis. It is enabled on all NFE 1 and 10 Gigabit Ethernet front-panel ports by default.

Intelligent Buffer Management on Cisco Nexus 9000 Series Switches White Paper

---

[ADD-ON 1 after comment:]

i'm curious if hardware qos ns-buffer-profile ultra-burst changes something when my traffic is passing only on 10GB interfaces.

According to documentation Buffer Boost Feature on Cisco Nexus 9300 Platform, once Buffer Boost is enabled, NFE ports (such as the 10G ports) - if so configured (per port) - can borrow additional buffer space from the ALE-2's 25MByte buffer. Compared to the 12Mbyte of shared buffer space the NFE ports have by default, that makes quite a difference, I should think.

[/ADD-ON 1]

---

[ADD-ON 2 after some more research]

Found in blogspace: Microburst/Burst Traffic: Cisco Nexus 9508 with N9K-9564PX linecard - ALE/Northstar Buffer. That was for a N9K-9564PX linecard, but the symptoms described and the architecture are probably not that fundamentally different from the N9K-9372PX. They observed unusual amounts of drops on under-utilized links, and found a solution in activating buffer boost. However, the blog post does not contain any info about the need for powercycling after activation.

From where:

The Solution

I'm sure by now you have figure out what we did to resolve our issue. We have changed our buffer from "Mesh Optimized" to "Burst Optimized" and that seem to take care of our burst traffic. We still have minor discards (1 or 2 over the course of 2 days), but to me, that is still acceptable. In addition, we still have "Ultra Burst" option available. Below are the very few commands to change buffer on these line cards.

[/ADD-ON 2]