My questions are on the bottom, feel free to skip my debugging story if you're in a hurry.
I started to check a network spanning tree status. All switches are connected on fiber, through SFP ports with transceivers.
By chance, I've stumbled upon a situation that confused me. Let me picture the situation for you guys:
I found that a switch was indicating that one of its SFP ports was down. It should be up. I checked the other link counterpart switch, and the corresponding SFP fiber link was indicating to be online. For the same fiber, one switch on one end was saying the link was up, and the other switch on the other and was saying the same link was down.
That got me confused. I'm talking about 2 switches, one of them is a 3COM 4500 with almost my age, and the second one a HP 1920-24G, both using RSTP. I wondered whether the 3COM disabled its port entirely on a situation regarding spanning tree discarding ports and so on.
I couldn't find any switches that match this situation to compare. So I went on field to check things out.
Indeed, one switch had its link OK, LEDs flashing, browser manager saying link was up, but the other switch on the same fiber was indicating a down link. So I disconnected both ends (they're not too far) of the LC connectors on each switch and used a visual fault locator to check the fiber integrity.
The link that was down showed good visible result on both cords (Rx/Tx), as my equipment could not measure attenuation, I only had my personal binary judgement based on a red laser blinking. The other end, which was up, showed visible results as well, but one of the cords, during the light probe, showed a strange red light from their laser output, as if some of the connectors had some micro dust on its top (showed a round red dot with a "blurish hole" not centered not showing light). I can't resemble which one of them it was by now (Rx/Tx). After rubbing the conector on my shirt, I tested again and got apparently a good visual response.
After that, I reconnected everything back, and the link status that was down went online back again. I checked another switch, a third one, far away, that I have not even mentioned here, and one of its SFP ports was disabled on the RSTP, as I was expecting, according to the local network topology, that link was the backup one, and it became the backup as soon as I reestablished the offline oficial link.
Unfortunately, about half an hour later, when I was telling all that to a maintenance supervisor, to keep him informed, I was going to show the switch with all the SFP ports OK, but since Karma happens, the link was down again.
We are going to replace the SC/LC patch cord connectors on both ends and see if the problem ends.
So my final questions are:
1) Does a SFP port LED indicator goes on if the transceiver receives packets? Or when the transceiver receives AND transmits packets?
I ask that because since we use two channels, one to transmit and other to receive, maybe one switch had a transmit link OK (receiving link OK on the other end) and the other receiving link was bad (transmitting link on the other end bad). If a SFP link, for a switch, is online when its port is receiving, regardless of transmission (Rx working, Tx don't know), my assumption is correct based on what I saw today.
2) What is probably causing the link to be faulty sometimes?
That link was good for almost 2 years. Maybe dust? Maybe faulty transceiver? Maybe bad optic patch cord? Maybe the SC pigtail fusion? Maybe environment (hot climate, low concentration acid fumes) ? I can guarantee no one has ever touched that connection since the startup of the link.