1

I have a very strange problem with links in one datacenter and I'm looking for some other solutions I haven't looked for yet. We have a network based on Cisco Nexus 3K and most of the core links between spine/leaf have FCS CRC errors. I've encountered these before in a different network and the fix was always quite easy with a different cable or optic. Now I ruled out all typical L1 problems, but I still have the issue.

So,

  • Changed the fiber
  • Changed the optic (10G LR)
  • Changed the switchport
  • Cleaned the fiber and checked it with a microscope, superclean
  • We have the issue on multiple switches
  • Links that have the issue are 2 meter direct SMF cables without patchpanels, but also on ~1KM paths. Both on 1310nm duplex, as well as bidi 10G u/d optics.

Now, I graphed the error rate per second and I noticed all graphs have the same shape. When the error rate goes up in one rack, it goes up too in another. Strangely enough, there are 2 leaves that do not have the issue. Configs are nearly the same. All links are L3 routed ports. Switches run on the latest release (7.8). It is very unclear what the user experience is, but 100.000 CRC errors in a few weeks is not normal, in my opinion.

Did anyone else encounter this before and what was your fix?

Thanks a lot!

Improve this question
4
  • You can hire a cabling professional, or, if you have the knowledge and experience, rent an expensive, professional cable tester, that will test your cables (both fiber and copper). You really want to get actual reports for the links with errors.
    – Ron Maupin ♦
    May 19, 2020 at 13:40
  • Are you overdriving the optics? You're using LR optics with 2m cables.
    – Ron Trunk
    May 19, 2020 at 13:47
  • Thanks a lot for your replies. We hired cable engineers today, they couldn't find anything damaged or dirty. All OK. I have the same problem on longer paths, unfortunately.
    – Sage
    May 19, 2020 at 17:36
  • Besides the physical inspection, did the vendor use an actual tester to see things like loss and give you a report for each cable tested? The professional level cable tester creates reports when it tests, and the vendor should provide you with the reports.
    – Ron Maupin ♦
    May 19, 2020 at 17:44

3 Answers 3

Reset to default
1

I wanted to post that the problems suddenly disappeared. It came back for a few days after my comment on May 21, but now it is gone since around May 24. It must have been some sort of external source. I think the DC knows what it is, but they won't tell us.

Thanks a lot for your answers and comments. Lesson learned is that FCS CRC errors apparently can be triggered from something external to your equipment when you ruled out typical L1 issues as mentioned in the OP.

Improve this answer
1
  • 1
    You should accept your own answer once that's possible (and possibly upvote other helpful answers). Let's hope the issue stays gone.
    – Zac67 ♦
    Jun 7, 2020 at 10:13
0

Reconfiguring your switches from cut-through to store-and-forward mode might make troubleshooting easier. See this thread and Cisco document for more details.

Improve this answer
2
  • 1
    hi Jeff, I've looked at this, but I wonder what switching modes will do on a switch with just L3 routing ports enabled. I assume these FCS bits are recalculated on L3 hops? Just as a MAC rewrite, TTL -1, etc.
    – Sage
    May 19, 2020 at 12:12
  • It may be ineffective, but as you're running out of routine troubleshooting steps, the not-so-routine ones are what's left.
    – Jeff Wheeler
    May 19, 2020 at 12:37
0

FCS errors indicate a problem with the physical layer.

Links that have the issue are 2 meter direct SMF cables without patchpanels

Depending on the exact PHY modules, the receivers might be overloaded. Check the datasheet and verify that the Tx power is within the Rx power envelope. Also, check the module diagnostics (DOM) for power levels - don't necessarily take them at face value though, some modules report unrealistic values.

SMF PHYs are designed for kilometers of fiber and a very short link might not provide sufficient attenuation. By standard, 10GBASE-LR (for 10 km reach) can be directly connected,[1] but some transmitters may be a bit on the high side. If the situation improves when you bend the fiber to a relatively tight radius (< 20 mm), an attenuator is exactly what you need. Use the DOM values to see the receive levels decrease while bending.

Alternatively, you could just try with attenuators in place (2 or 3 dB should be enough) and see if the FCS error rate changes.

[1] By 802.3 Clause 52, nominal transmit power is 0.5 dBm, receive power is -14.4 to 0.5 dBm, so without any attenuation, tolerance is tight and the receiver may operate at its upper limit.

Improve this answer
5
  • Hi Zac, Thanks for your reply. I wasn't clear in my initial question. I have tested on 2 meter fibers, but I have the same problem on 1 KM fibers. The DOM values are well within limits. Also note that I have the problem on bidi optics with completely different paths. And, strangely enough the error graph for these paths look the same. The error rate does not correspond with the amount of pps through a link.
    – Sage
    May 19, 2020 at 12:11
  • 1
    Hmm. When the error rates go up simultaneously in different locations, there must be a common cause. Power/electricity? EMI? Radiation? Vibration? Mechanical issues with the fiber trunks? Common frame source? Also possible, but only for rather extreme values: temperature or humidity?
    – Zac67 ♦
    May 19, 2020 at 15:33
  • I think it must be an external radiation source of some sort, I don't have any clue what it can be other than that. The site is located at a commercial DC though, next to one of the main sites of one of the biggest internet exchanges (AMS-IX). There is a construction site next to the DC, maybe that has something to do with the issue... But I think a whole lot more problems would arise if that would be the case, especially for AMSIX. Cable engineers check the cabling today, they couldn't find anything damaged or dirty. Sigh :)
    – Sage
    May 19, 2020 at 17:34
  • Do the racks have proper (metal) doors and side panels? Proper grounding? Then EMI is unlikely to get through. Are the problem switches on a common power phase?
    – Zac67 ♦
    May 19, 2020 at 17:58
  • 1
    Hi Zac, an update; after weeks of issues on all our links the problem suddenly disappeared on all links around 36 hours ago. That makes me quite confident the issue was in the datacenter electrical grid, or was some source of EMI or something. I've asked the DC for an investigation, I hope they're going to be honest with us. I'll post an answer once I have the final solution. Thanks for your help!
    – Sage
    May 21, 2020 at 7:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.