1

enter image description here

I have dual stack vPC setup between 4 cisco nexus switch as per above design. I have noticed recently that getting CRC error on cisco 3064 all 4 interface connected to N9K, and during peak traffic time getting more CRC error, these error are not very high also per minute i would say 1 or 2 error randomly popping up on those 4 interfaces. I know CRC has direct connection with Layer 1 (cable, port etc..) but in my case how can be possible both switches and all 4 interface throwing CRC?

After clear counter

# show interface e1/51-52 counters errors

--------------------------------------------------------------------------------
Port          Align-Err    FCS-Err   Xmit-Err    Rcv-Err  UnderSize OutDiscards
--------------------------------------------------------------------------------
Eth1/51               0         12          0         12          0           0
Eth1/52               0         20          0         20          0           0

--------------------------------------------------------------------------------
Port         Single-Col  Multi-Col   Late-Col  Exces-Col  Carri-Sen       Runts
--------------------------------------------------------------------------------
Eth1/51               0          0          0          0          0           0
Eth1/52               0          0          0          0          0           0

--------------------------------------------------------------------------------
Port          Giants SQETest-Err Deferred-Tx IntMacTx-Er IntMacRx-Er Symbol-Err
--------------------------------------------------------------------------------
Eth1/51            0          --           0           0           0          0
Eth1/52            0          --           0           0           0          0

Interface config

interface Ethernet1/51
  description *** vPC trunk to N9K ***
  switchport mode trunk
  switchport trunk allowed vlan 10-12,20-22,27-32,39-40,50,100,200,300
  speed 40000
  channel-group 3 mode active

interface Ethernet1/52
  description *** vPC trunk to N9K ***
  switchport mode trunk
  switchport trunk allowed vlan 10-12,20-22,27-32,39-40,50,100,200,300
  speed 40000
  channel-group 3 mode active

what else could be wrong here, my peak traffic usage on these switches around 10 to 20Gbps during peak

Update

here is the screenshot which showing what is going on, if you see e1/51 throwing errors in daytime and e1/52 throwing error in night time so that signwave.

enter image description here

Update-2

enter image description here

Improve this question
6
  • Do you have any scale for that error rate?
    – Zac67 ♦
    Apr 17, 2020 at 13:42
  • I have updated image in original question, 100m is actullay 100 counters, also noticed since COVID-19 started we are see more error because getting more traffic now days.
    – Satish
    Apr 17, 2020 at 13:47
  • Hmm - m should mean milli, and 0.2 error packets/sec (or 12 errors/min) is even more than in your description, definitely on the rather high side. I'd check the counters on the console, to make sure the order of magnitude is correct.
    – Zac67 ♦
    Apr 17, 2020 at 14:02
  • I have checked on physical interface and running constantly refresh and seeing 1 CRC here and there in minute or two minute interval. (its monitoring software which is not representing value correct way so just ignoring m
    – Satish
    Apr 17, 2020 at 17:53
  • The legend says per second and X merrors/s is roughly similar to X errors/min
    – Zac67 ♦
    Apr 17, 2020 at 18:02

2 Answers 2

Reset to default
3

An interface running at 20 Gbit/s processes between 1.6 and 30 million frames per second or 100 to 1800 Mframes per minute.

One FCS error/min corresponds to a bit error rate of .8x10-12 which is slightly below the standard 10-12 that 802.3 usually designs for.

Two errors/min are a bit on the high side but nothing to really worry about if you've maxed out the reach, for instance (that isn't 40GBASE-T, is it?). Especially when you're in a location with relatively high background radiation, something like that should be expected.

Nevertheless, you should monitor the error rates as you're doing. They could also be a sign of deteriorating fiber modules, water damage to fiber or similar.

Improve this answer
3
  • Reason I’m asking but recently we noticed lots of TCP retransmission in tcp pcap, and sometime tcp drop connections. While debugging that’s issue I noticed CRC error so thought maybe my issue hiding here. I will try to replace cable and see but still doesn’t understand behavior. I do have many switch and they have more traffic compare to this one but they are totally fine.
    – Satish
    Apr 17, 2020 at 12:13
  • 2
    Such a low BER doesn't cause TCP problems apart from (very) occasional retransmits. Is that copper or fiber? Copper is susceptible to EMI and should be kept apart from any power cabling. Fiber is susceptible to mechanical stress, esp. overbending - check the run and of possible try another cable.
    – Zac67 ♦
    Apr 17, 2020 at 13:39
  • all fibers with QSFP ports.
    – Satish
    Apr 17, 2020 at 15:34
1

Have you got errors on any other ports? The Nexus switches in your diagram operate in cut-through mode by default, and packets larger than ~768 bytes can be forwarded even if they have errors. This is configurable as documented by Cisco, here.

This means you could be getting bad packets from one connected device/server and they're just showing up on the vPC links because these bad packets have to traverse them.

Improve this answer
8
  • Hmm.. interesting, we have lots of switches in network and all configured with vPC, I’m not seeing any issues on path of switches. Also I check server side ports and not seeing any error there. Is there any method or way to find out origin ?
    – Satish
    Apr 19, 2020 at 15:05
  • No easy way, no. If the error counts on all ports except these four are 0, I would consider swapping one 40G transceiver/cable-set to an optical cable; maybe rule in or out Zac67's suggestion of EMI. It's too bad there aren't FEC stats exposed on this equipment; it would be a powerful troubleshooting tool in situations like this.
    – Jeff Wheeler
    Apr 19, 2020 at 18:43
  • Tell me one thing, i am getting FCS error on N3K switch input interface so does that means Frame is already corrupted from N9K so should i start looking upstream switches which is connected to N9K? Just trying to understand on which direction bad frames coming from?
    – Satish
    Apr 20, 2020 at 14:51
  • That is possible in cut-through forwarding mode. That's why I suggested checking the error counters on all other ports of the illustrated switches -- but you shouldn't have to check switches further away; only check the N3K ports and the ports on the N9Ks connected to the affected N3Ks. ALTERNATELY, you can perform the procedure in the Cisco doc I linked, and temporarily change to STORE-AND-FORWARD MODE on the illustrated switches. That way, corrupt frames wouldn't be forwarded. Cut-through mode = bad frames can be forwarded; store-and-forward mode = they cannot.
    – Jeff Wheeler
    Apr 20, 2020 at 20:06
  • but changing from cut-through to store-and-forward in live production traffic would be really bad right?
    – Satish
    Apr 21, 2020 at 5:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.