Cisco: err-disabled due to loop - but there are NO redundant physical paths/ wires/ connections

Question

for the third or fourth time in approx two years we have (painfully) noticed some really strange (at least to us) behaviour in our datacenters:

Uplink-interfaces on access-switches which have only one (!) physical uplink (lower end-points of the hierarchy) are going into err-disable state due to Loop-back detected.

There are no physical (wired) loops. How can this be?

RZ2-AS18# show version

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 28    WS-C3560G-24TS     12.2(55)SE1           C3560-IPSERVICESK9-M

RZ2-AS18# show int des

Gi0/24  up  up  *** Uplink to rz2-cs4 Gi0/24 ***

RZ2-AS18# show logging

Jul 26 11:35:55.328: %ETHCNTR-3-LOOP_BACK_DETECTED: Loop-back detected on GigabitEthernet0/24.
Jul 26 11:35:55.328: %PM-4-ERR_DISABLE: loopback error detected on Gi0/24, putting Gi0/24 in err-disable state
Jul 26 11:35:56.611: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to down
Jul 26 11:35:57.651: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
.Jul 26 12:12:04.173: %LINK-5-CHANGED: Interface GigabitEthernet0/24, changed state to administratively down
.Jul 26 12:12:06.295: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to down
.Jul 26 12:12:09.131: %LINK-3-UPDOWN: Interface GigabitEthernet0/24, changed state to up
.Jul 26 12:12:10.137: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/24, changed state to up
.Jul 26 12:13:05.117: %SYS-5-CONFIG_I: Configured from console by console

rz2-as18# show run int gi 0/24

interface GigabitEthernet0/24
 description *** rz2-cs4 Gi0/24 ***
 switchport trunk encapsulation dot1q
 switchport mode trunk
 switchport nonegotiate
end

General network structure:

• ring between core-switches (CS)
• access-switches connected to core-switches in star-topology

Any ideas? Thoughts? Thanks in advance!

Answer 1

I think this might be related to a bug. This issue is documented in Cisco bug ID CSCea46385.

To resolve the issue, you need to add the no keepalive command to the interface.

Hope this helps you on your quest.

SleepyMan

Answer 2

The problem is that Cisco switches send out a packet for loopback detection that if they receive back on a port they will automatically put the port in an err-disabled state. This is a big problem on uplink ports, because a loop anywhere on the network (even caused by one switch or an IP phone plugged in with two cables...) has the potential to reflect those packets back to the switch, causing the far away switch to put ports in an err-disabled state anywhere on the network.

The "no keepalive" command will prevent this from happening on the interface and should be put in place on all uplink interfaces. It does obviously does not solve the root of the problem, but is much better than the network tearing itself apart.

In your network diagram you have a ring made out of core switches. It seems likely that the spanning tree loop prevention is not working between them. Do a "show run | b span" and verify that all switches have the same spanning tree configuration, with the exception of spanning tree priority. I believe the keepalive packets are circling your loop and returning to the original switch, causing the shutdown.