I have a very strange problem with links in one datacenter and I'm looking for some other solutions I haven't looked for yet. We have a network based on Cisco Nexus 3K and most of the core links between spine/leaf have FCS CRC errors. I've encountered these before in a different network and the fix was always quite easy with a different cable or optic. Now I ruled out all typical L1 problems, but I still have the issue.
- Changed the fiber
- Changed the optic (10G LR)
- Changed the switchport
- Cleaned the fiber and checked it with a microscope, superclean
- We have the issue on multiple switches
- Links that have the issue are 2 meter direct SMF cables without patchpanels, but also on ~1KM paths. Both on 1310nm duplex, as well as bidi 10G u/d optics.
Now, I graphed the error rate per second and I noticed all graphs have the same shape. When the error rate goes up in one rack, it goes up too in another. Strangely enough, there are 2 leaves that do not have the issue. Configs are nearly the same. All links are L3 routed ports. Switches run on the latest release (7.8). It is very unclear what the user experience is, but 100.000 CRC errors in a few weeks is not normal, in my opinion.
Did anyone else encounter this before and what was your fix?
Thanks a lot!