1

After a bit of advice on an intermittent issue i'm investigating. We have two buildings on the same site connected by 548m fibre run that we own and it was installed around 10 years ago. Over the last 6-8 months we have been getting a number of connection drops and intermittent issues. The edge switch is a Cisco 3650 (WS-C3650-24PS) running sw version 03.03.03se connected by 1000BaseLX SFP on SM fibre - Connected to a Cisco 2960 (WS-C2960XR-24TS-I) running sw version 15.2(6)E1 connected by 1000BaseLX SFP on SM fibre.

The site is experiencing seemingly random connection drops (Packet loss of 60-80%), if we do a shut/no shut on the edge switch port it will reinstate the connection after a short period.

We have had the fibre run tested with an OTDR and no apparent issues on the run, but some loss at the far end the would say it needs cleaning. I've carried out a rough clean on the fibres, replaced the SFP's and patch cables with new ones we are still seeing runts/input errors on the 2960 and unknown protocol drops on the 3650 but no input errors.

Output from the interface on the 3650

 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output never, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 541000 bits/sec, 211 packets/sec
  5 minute output rate 2726000 bits/sec, 322 packets/sec
     1596899766 packets input, 440076865414 bytes, 0 no buffer
     Received 86226449 broadcasts (53907138 multicasts)
     0 runts, 0 giants, 0 throttles
     302323 input errors, 302323 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 53907138 multicast, 0 pause input
     0 input packets with dribble condition detected
     3026685808 packets output, 2453674763063 bytes, 0 underruns
     0 output errors, 0 collisions, 24 interface resets
     616832 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Receiving interface on the 2960:

 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:15, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1d00h
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 1966000 bits/sec, 232 packets/sec
  5 minute output rate 424000 bits/sec, 152 packets/sec
     16712081 packets input, 13739039186 bytes, 0 no buffer
     Received 193868 broadcasts (170085 multicasts)
     6147 runts, 0 giants, 0 throttles
     48719 input errors, 1 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 170085 multicast, 0 pause input
     0 input packets with dribble condition detected
     9738314 packets output, 2762251700 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Does anyone have any possible advice?

Update: Cisco 2960

Cisco 2960

Cisco 3650

Cisco 3650

Improve this question
14
  • I'm 99.99% sure you are looking at a defective dark fiber.
    – user36472
    Feb 7, 2019 at 9:51
  • We've had a 3rd party test the fibre run with a OTDR - There was some jitter/loss on the line but nothing too out of the ordinary. We are going to engage with 3rd party again for further advice.
    – JLS
    Feb 7, 2019 at 9:56
  • 1
    Input errors, CRC and runts on Cisco equipment is always a physical error and you've changed the SFP's and patch cables, ergo only the dark fiber to blame.
    – user36472
    Feb 7, 2019 at 10:01
  • Surely a OTDR would've seen if there was anything serious on the fibre run?
    – JLS
    Feb 7, 2019 at 10:03
  • Well, define "some jitter/loss" none of our dark fibers have any errors at all.
    – user36472
    Feb 7, 2019 at 10:12

1 Answer 1

Reset to default
1

302323 input errors, 302323 CRC, 0 frame, 0 overrun, 0 ignored

I think this is the key information. Apparently, the fiber from the 2960 transmitter to the 3650 receiver is very bad. If you've already ruled out transceivers and patch cables, double check for fiber type mismatches (SMF vs MMF and SPC vs APC) and what's left is the dark fibre and its termination.

If there's no spare fiber strand to move to you might want to consider trying a pair of bidi transceivers on the remaining strand (1000BASE-BX10).

With the OTDR indicating termination problems the end jack needs proper cleaning or replacement.

The optical Tx and Rx levels from ODM have always to be taken with a grain of salt - but if they're somewhat exact, 4-5 dB loss is a lot for just 500 m SMF. Just to make sure: SMF is very sensitive about minimum bend radius - you need to make sure your patch cords never undercut that radius (ca. 5 cm).

Improve this answer
5
  • Thanks for your input. Presently trying to establish the type of fibre that is being used across the link. Since it was installed over 10 years ago we don't seem to have any documentation relating to the install. The transmit level output comes from the switches not the OTDR.
    – JLS
    Feb 7, 2019 at 12:44
  • With that length you don't need the exact type, just make sure it's really SMF (or find out the type of MMF - from ~10 years ago it should be OM2 or OM3). SMF is 9 µm, MMF 50 µm core. Don't mix SMF and MMF anywhere.
    – Zac67 ♦
    Feb 7, 2019 at 12:47
  • Just having this discussion - We are going to re-visit site and see if there is any distinguishing marks on the fibre coil in the tray.
    – JLS
    Feb 7, 2019 at 12:56
  • The markings are probably not on the pigtail/coil in the tray, but on the outer shielding plastic layer of the cable (of which there might be a coil somewhere in the closet or rack, too). One of the simplest indicators to look for are markings like 62.5/125 or 50/125 (which makes it multimode fiber) or 9/125. (singlemode fiber, just as Zac67 said).
    – Marc 'netztier' Luethi
    Feb 7, 2019 at 13:10
  • The color of the outer plastic/rubbery shielding layer is another hint (just a hint, not a guarantee). Orange is classic multimode (62.5/125), aqua/turqoise is typically OM3 10G Laser optimised multimode (50/125), pinky/violetty ("Erika") is more modern OM4 (50/125) multimode, while single mode (9/125) should be yellow. Howewer: While the color coding is commonly respected on FO patch cords, fixed-installation (multi-fiber-)cables do not always follow the color code - but I've seen it as well.
    – Marc 'netztier' Luethi
    Feb 7, 2019 at 13:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.