One of our Linux hosts has started switching its gears recently. While checking the interface speed, I could see that the interface has switched down to 10Mbps. The NIC and Switch are capable of 1G and auto negotiation is enabled at both sides. I would like to debug the root cause of the issue. We are planning to perform the below mentioned actions.

  1. Perform the cable test to check/replace the Cat6 cable.
  2. Double check on the switch configurations. (Link capability and L1 Advertisements coming out of Switch)

Given that tcpdump works from L2 and up there is no point in getting the tcpdump . Do you guys have any other pointers? Perhaps, syslog/dmseg ?

I have already tried to capture the L1 advertisements but the command is not supported. The switch platform we are using is Cisco 5596.

We generally use the command show controllers <interface> | include Autoneg Lnk Ptr abty to decode the fast link pulse messages but that is not supported in the 5596 switch. We have already confirmed that both NIC (via ethtool) and switch will support 10, 100 and 1G.

How do you approach issues and collect stats on issues at L1/Phy layer (especially link speed switching down from 1G to 10M)?

enter image description here

  • A while ago, there used to be issues when one side of the link was forced to a specific speed/duplex and the other wasn't (because it would disable negotiation). Not sure if those kind of issues still exist nowadays, but you may want to check the configuration for the interface on both sides. – jcaron Oct 18 '18 at 17:33

Likely it's the cable.

While there were some issues with the Autonegotiation standard very early on, they've been resolved for decades and anything from 2000+ can't be affected.

Generally, leave Autonegotiation enabled at all times. Most often, when you think you need to configure it manually it lands on your feet some time later.

When checking for cable or port issues it's a good idea to check the logged events and port error counters first. Frequent relinking and sub-speed linking indicate cable damage (can also be caused by poor cable quality or installation, or exceeded reach). FCS errors, runts, giants and such indicate general transmission problems.

I usually move production to another switch port (configure appropriately!) and patch cable(s) to solve the issue quickly. Then I check out the cable and test the port - 99.5% of the time it's the cable.

Generally, it's a good idea to implement some port monitoring so you'd get an early warning in case a port doesn't link as expected or accumulates errors.

  • Thanks for the pointers.I'm taking all of your suggestions and framing an action plan. Given that the command "show controllers <interface> | include Autoneg Lnk Ptr abty" is not working, Do you know any other alternative commands to check the L1 advertisements coming out of the NIC towards switch port? – Maverick Oct 18 '18 at 12:52
  • No, sorry. Not really familiar with Nexus. – Zac67 Oct 18 '18 at 12:58
  • The team is not willing to replace the cable.So, I went ahead and reconfigured it to Auto negotiation and it was reproduced. I could see the following. Ethtool output showed as 10,100,1000 all modes are supported,Dmesg- A NIC down and UP event ,ethtool -S errors ( txerror- 946,collision 9782,tx_window_error 946,tx_abort_late_coll -946,tx_deferred_ok- 8813,single_coll_ok-5845 etc. When I checked with a working device,I could see there was absolute 0 of tx erros,window errors. Does this mean that the issue is at the switch or within the NIC of equipment itself? – Maverick Nov 28 '18 at 14:32
  • The collision/late_coll counters indicate a duplex mismatch with half duplex at that end. Either the far port is manually forced to full duplex (breaking autoneg) or there really is a problem with autoneg, however unlikely. – Zac67 Nov 28 '18 at 14:42
  • //Either the far port is manually forced to full duplex// Do you mean Autonegotiation + explicit Full Duplex is a violation of configuration? If Yes,Then that seems to be the culprit because our current port configuration is as mentioned below. Type (Non SFP): 10/100/1000BaseT Speed: 10,100,1000,auto Duplex: full Trunk encap. type: 802.1Q Channel: yes Broadcast suppression: no Flowcontrol: rx-(off/on),tx-(off/on) And Yes, The Duplex is set as FULL? Is that the smoking gun? – Maverick Nov 28 '18 at 15:02

Some Cisco Platforms (most access switch types of the Catalyst 2900/3600/3800 series and their older siblings too) support these commands:

interface GigabitEthernet M/NN
 ! select which set of speeds are offered during auto-neg
 speed auto 1000 100 10 

This opens a path to debugging an Autoneg issue between switchport and NIC - by just offering one data rate, you can restrict the autoneg scope and check at which speeds the NIC will autoneg.

However, reading the data sheets of the 5596 (which variety? 5596UP, 5596T?), I think there is no support for 10/100M on any of the RJ45 ports.

That brings us back to Zac67's answer: probably cabling.

  • The Switch model is 5596UP. – Maverick Oct 19 '18 at 5:31

1000BaseT uses all 4 pairs in the cable bidirectionally, so a failure in pair 1 or 4 should theoretically affect both switch and host and force a fallback to 100 Mb/s. A failure in pair 2 or 3 would give a 'link down' (you don't actually say the link is up, but I assume so).

So, I would start with the cable (not forgetting to check for bent pins in the connectors).

If this doesn't cure the problem you've probably got a port failure on one side or the other. You could try forcing 100 Mb/s auto-duplex (if possible, if not full-duplex).

  • Thanks, We tried to change to 100Base T/FULL and it is working without any issues now. We are only encountering the issue with 1000 Base T/FULL. – Maverick Oct 19 '18 at 2:00
  • That's another strong pointer to what Zac67 and grahamj42 were hinting at: cabling issue. If it works at 100M, but not at 1000M, very probably not all 4 wire pairs are ok. – Marc 'netztier' Luethi Oct 19 '18 at 7:40
  • 1
    Note that not all devices support a fallback from 1000BASE-T to 100BASE-TX - it's not by IEEE standard but was introduced by Broadcom. For instance, our HP/Aruba switches don't, not even the most recent ones. I'm using a two-pair loopback adapter to quickly test for that. Without the fallback, a two-pair link simply doesn't come up with 1000/100/10 autonegotiation due to PCS pairing failing. – Zac67 Nov 30 '18 at 6:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.