We're experiencing output discards in couple of our N9K-C9372PXs. Seems like egress queues fills and it ends up in a lots of discarded packets.

All interfaces where discards occur are NEXUS interfaces (to FEX)

show interface ethernet 1/21 | inc "output discard" 
0 lost carrier 0  no carrier 0 babble 32043410 output discard

Bandwidth wise they're OK. I think it's due to bursty traffic from multiple uplinks (traffic comming from 6 Uplinks to 2 FEX ports). So, things like DNS queries - which're using UDP packets are really damaged by that. Could you say how to check the size of the buffer and how they're filling up ?

I'm wondering: is there are any workaround for this type of behavior ?

Now we're using:

show hardware qos ns-buffer-profile 
NS Buffer Profile: Burst optimized

i see there are commands like these:

hardware qos ns-buffer-profile ultra-burst

Maybe you know, would ultra-burst mode help in our situation ? And what is requirements on entering that command ? do it requires reboot ?

or maybe it's possible to make some type of queuing where we'd treat UDP packets differently ?

also we have few older N5K-C5672UP, N5K-C5548UP and them are not experiencing it. Maybe they have 'bigger hardware' buffers ?

Thanks for any input.

  • What FEX model? You mention two uplinks. Unless it's an ancient 2224TP you're using a subset of the available NIF ports and this means that you're only getting a fraction of the maximum hardware buffers available on the platform. It's not uncommon to run out of buffer even on fairly lightly utilized uplinks. – rnxrx Aug 6 '19 at 2:07
  • sorry, i made an mistake, the discards are on Nexus links which are towards the FEX'es. As for now i have no idea why discards occur ?. Ofcourse it'd be due to microburst. I'm planning to change the profile from burst to ultra-burst. Hope it will not have an impact. – Edvinas Aug 6 '19 at 8:16
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can provide and accept your own answer. – Ron Maupin Dec 15 '19 at 5:47

Nexus as cut-through switches are a different league than the store-and-forward switches many of us have become accustomed to. They must handle speed step-downs with different strategies than classic output queuing.

Downlinks to FEXes are usually LACP bundles of 10G. So if there are links to the given N9K-C9372PX running at 40G, there will be a speed stepdown to the FEX in question - and some packets will have to be discarded.

The following links might be quite steep an entry into the inner workings of the N9K family - probably still worth the time to read.

Cisco Nexus 9300 Platform Buffer and Queuing Architecture

From which:

Buffer Boost is an egress-port configuration property. It can be enabled or disabled on a per-port basis. It is enabled on all NFE 1 and 10 Gigabit Ethernet front-panel ports by default.

Intelligent Buffer Management on Cisco Nexus 9000 Series Switches White Paper

[ADD-ON 1 after comment:]

i'm curious if hardware qos ns-buffer-profile ultra-burst changes something when my traffic is passing only on 10GB interfaces.

According to documentation Buffer Boost Feature on Cisco Nexus 9300 Platform, once Buffer Boost is enabled, NFE ports (such as the 10G ports) - if so configured (per port) - can borrow additional buffer space from the ALE-2's 25MByte buffer. Compared to the 12Mbyte of shared buffer space the NFE ports have by default, that makes quite a difference, I should think.

[/ADD-ON 1]

[ADD-ON 2 after some more research]

Found in blogspace: Microburst/Burst Traffic: Cisco Nexus 9508 with N9K-9564PX linecard - ALE/Northstar Buffer. That was for a N9K-9564PX linecard, but the symptoms described and the architecture are probably not that fundamentally different from the N9K-9372PX. They observed unusual amounts of drops on under-utilized links, and found a solution in activating buffer boost. However, the blog post does not contain any info about the need for powercycling after activation.

From where:

The Solution

I'm sure by now you have figure out what we did to resolve our issue. We have changed our buffer from "Mesh Optimized" to "Burst Optimized" and that seem to take care of our burst traffic. We still have minor discards (1 or 2 over the course of 2 days), but to me, that is still acceptable. In addition, we still have "Ultra Burst" option available. Below are the very few commands to change buffer on these line cards.

[/ADD-ON 2]

  • thank you, yes, i understand the case when Nexus up link interfaces is a higher class (for eg. 40GB) than to FEX ones (10 GBs). But in my case all interfaces are 10Gbs. Btw, maybe you know - Which interfaces is Northstar ones - only 40Gbs or 10Gb also ? Thank you – Edvinas Aug 7 '19 at 7:51
  • i'm curious if hardware qos ns-buffer-profile ultra-burst changes something when my traffic is passing only on 10GB interfaces. – Edvinas Aug 7 '19 at 8:40
  • According to cisco.com/c/en/us/products/collateral/switches/…, the N9K-C9372PX is based on ALE-2. The 6x40GE ports are connected to the ALE-2 and share a total of 25MB of buffer memory. The 10G ports are connected to the NFE and share a total of 12MB of buffer memory. – Marc 'netztier' Luethi Aug 7 '19 at 8:48
  • thanks, so far the problem persists. Ultra boost is by default enabled. Tried to enter the command: Burst Optimized, no improvement. So far we dont get any complains, but the discards are still occuring. – Edvinas Dec 16 '19 at 8:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.