How can I diagnose a bridging (ethernet) loop?

Question

Given that spanning tree has failed (or you don't have any spanning tree) and get an ethernet loop, what's the best way to diagnose where the problem is?

Which switch?, which cable? and so on.

Answer 1

OK, so assume you have a topology like:

          SW1
         /   \
        /     \
       /       \
PC A--SW2-----SW3--PC B

For some reason there is a bridging loop, STP is disabled or someone applied a filter in the wrong place or such.

PC A wants to communicate with PC B. It first ARPs for the MAC of PC B, the destination is a broadcast with MAC ffff.ffff.ffff. So the frame goes to both SW1 and SW3. The SRC MAC is PC A. SW1 then floods the frame towards SW3 and SW3 will flood the frame coming from SW2 to SW1.

SW1 and SW3 learned the MAC of PC A when the first frame came in. When the second one comes in from the opposite direction it has to relearn it. Because these events occur so fast and repeatedly you will see log messages complaining about MAC flapping. Something like "MAC FLAP 0000.0000.0001 is flapping between Gi0/24 and Gi0/23". This is a good sign that you have a loop.

What you could do then is to try to trace this MAC. Try looking in the ARP cache of a device in the same subnet and see what IP this device has. So with the MAC you could try to trace it with sh mac-address-table or with the IP maybe you have a list with all IPs and where they are connected.

If the host gets a IP address from a DHCP server you could also try there to find where the host is coming from. If you have option 82 enabled that would be a great help.

Other signs are that the CLI will be very sluggish. CPU load will be very high. Switches do almost everything in ASICs so if a switch has a CPU load over 50% it's probably not good. You should implement SNMP monitoring and watch for high CPU load. Also look for the MAC flap messages. If the switches have a loop the LEDs will probably be blinking like crazy.

Things you could do to protect against loops:

• Enable STP! (duh)
• SNMP monitoring of CPU load
• Enable SNMP traps for certain events like STP topology changes
• Enable storm control on the ports to limit broadcast
• Don't span your VLANs too much in your L2 topology
• Enable port security and limit number of MAC addresses per port
• Enable Option82 if you run DHCP

Answer 2

One of my users recently borrowed a desktop switch from someone's desk. Upon returning the switch, they plugged in all of the loose ethernet ends that were nearby. One of those cables went to the network and another was two ends of the same cable. The desktop switch was plugged into the network and also plugged into itself. The switch had no STP, so broadcasts that came in from the network would loop on the other cable in both directions. Of course, each time a broadcast was received on the looped ports it gets replicated back into the network. It drove HSRP absolutely mad and--due to poor design--it also resulted in OSPF adjacency failures across the campus.

The first indication of the problem was a macflap forwarded to my email. This immediately led us to the correct wiring closet. From there, it was a process of elimination based on port LEDs, interface pps and logs. Needless to say, I have since rearchitected the entire campus. The best preventative measure is probably bpduguard. I have since deployed the feature and it was quite simple. Getting that errdisable syslog in my email is nothing short of bliss.

Answer 3

With most equipment, the CPU shoots to 100% and the only thing you can do is break the redundant physical connections. Once the CPU calms down you can plug the links back in one by one and see which one re-causes the loop.

For big chassis (like a 6500) I've had to pull out all of the blades and plug them back in one at a time. Once I figured out which blade, then I had to pull all of the individual links (16 GBICs) and put them back in one at a time as well. Never fun.

Some more modern equipment has a protected CPU which should make this easier to deal with - you can still interact with the box. At that point looking at traffic counters and such to determine the malfunctioning link becomes possible.

Answer 4

I recently started at a company where they use broadcast limits on each port. If a port passes >5% of it's capacity as broadcasts the switch puts it into ERRDISABLE.

 storm-control broadcast level 5.00  
 storm-control action shutdown

This has been a life saver when one group tends to plug in devices that bridge the wireless networks onto the LAN.

Though for your actual question, I've always found it to be manual.

Answer 5

for IOS:

You will probably have MAC addresses flapping between ports.. look for MAC_MOVE_NOTIFICATION (or similar) errors in:

sh logg

Now to find the port:

sh int g0/1 controller

look for out of ordinary Multicast and Broadcast numbers. Any collisions are a bad sign.

Last but not least, you can't log in, because the CPU is pwned :)

sh proc cpu

How is the switch doing here? If it's an L2 switch only, you don't want anything above ~10%

Answer 6

In the case you have unmanaged, or the equivalence of unmanaged (lacking login details, or knowledge of the switch operating system, etc), switches and a bridge loop, I describe how I'd go about finding the loop manually. This also addresses the fundamental bottom of the original question, "you don't have STP".

The basic algorithm for fault-locating this loop is similar to STP except you don't readily have access to sending out BPDU's with port ID's in them.

• First, connect a packet dumping/sniffing-capable device to a port in one of the switches. This device has now become your tree's root device.
  • If you have to fault-locate in multiple locations, e.g. over a "campus" or similar, you stand to gain by being able to remotely login with a portable ssh client to the packet dumping machine.
    • I'd personally use my Linux laptop with an Internet connection with tcpdump in a screen and ssh into it from for example ipad or phone.
  • If you are unable to login yourself remotely, use a friend to visually monitor the tcpdump, which is probably flooding at link speed making it easy to notice a difference whenever the path towards the loop source device is disconnected.
• Next, you're going to have to essentially recreate a tree, starting from your root switch.
  1. And because you can have the scenario where you have multiple looping links feeding into your root device, you must start by removing all connected ports simultaneously at once.
  2. Reconnect the ports one by one and if at any time the packet burst re-appears, follow this port to the connected switch in the other end.
  3. Repeat step 1, until you find the looped port(s) and can't iterate further down in your manual tree.
  4. Having resolved the loop situation in this switch, return to the switch above in the tree and resume step 2. This recursion continues all the way back until the final cable has been re-connected in your root switch.

This is a completely exhaustive manual search for looped ports.

Typically there will just be one pair of ports that are looped, meaning the exhaustive and safe search with first removing all connected (link) ports and then reconnecting them one by one is unnecessary. If just one port pair down the 'tree' is looped, you can find it by simply disconnecting one port at a time.

Nevertheless, the general, "foul-proof", method, or algorithm, becomes what I described above.

Answer 7

Ouch. But ok, I can think of two ways I'd go at this...

Eyeball it: If the switches have port indicators, you should be able to eyeball which ports are the most active. Those are the ones to start looking at first. Hopefully the cables are labeled so you can search for the low hanging fruit of finding two busy ports, on two switches with the same cable.

SNMP monitoring: If you've got SNMP (or similar) usage stats, look for the busiest switch and the busiest ports. Then go looking at cables.

...if you have unlabeled cables, start tracing and labeling as part of your checking out the busiest ports.

Answer 8

To be honest, if you remotely connected (or via console cable) onto the device, you'd notice its very sluggish, there will be a delay from when you type to the letters coming up on the CLI.

If its a Cisco switch, 2 easy ones are to look at the interface statistics, it'll be on 100% (or 255/255) usage, constantly. In my years of dealing with switches, I've yet to see a port legitimatly hit 100% usage. Other than that, check the CPU usage (usually "show process cpu history"), looped interfaces will usually hit your CPU pretty hard unless you're running a high end switch.

STP should really be enabled though!

Answer 9

I am going to answer this question based on the understanding that there is a full outage for the layer 2 domain in question, and that you have no management access because the CPU's are all pegged.

The best way to troubleshoot a bridging loop is to start unplugging uplinks until it goes away. Say you have a standard switched access layer with all the access switches connecting into a pair of distribution switches. Go to the first access switch, and unplug the uplinks, if the LED's for the switchports stop going mental, it’s not that switch, plug it back in and go to the next one. Repeat until you get to a switch where you have unplugged the uplinks and the LED's are continuing to blink rapidly, this is your switch with the loop.

Now start the unplugging process on the end user ports until the LED's calm down, when they do, the last on you unplugged was the problem port, trace the cable and chastise the user appropriately.

Answer 10

I had this issue occur on a network on the other end of the US and had to remotely help some level one analysts via the phone and my wan link to their site. The issue was complicated further by the fact that they had several brands of switches which they had slowly added to the network over the years. When they moved the office, they marked where each port went then re-attached everything the exact same way at the new office and started everything up. Needless to say that the handful of switches that did have working spanning tree didn't converge the same way and they had all kinds of loops and issues. By the time I was done fixing everything no less than three unmanaged switches were discovered to have been connected in loops with the rest of the infrastructure.

The way I was able to track down each of the unmanaged switches was by using a tool called nedi (on the switches which were able to be managed I enabled lldp/cdp). I first generated maps with nedi. Then in areas where the map showed connections from one switch to another then back around to the same switch again I had the network technician on site trace the line manually. I either manually shutdown the interfaces involved with the loop or had the onsite person unplug cables. In the end I was able to get the network working as it should, despite all the crazy off brand switches.

Answer 11

One thing that can be done here, is to see what machines are connected to the switch using the commands show cdp neighbor or show lldp neighbor.

If the BPDU guard command is not being used, and someone connects a rogue switch with lower priority (or older mac-address), the new device will negotiate as Spanning Tree root which will surely cause a problem.

Answer 12

In my experience it's always been the cable that I just plugged in, or no shut, or added to the port-channel. Tougher is when someone else did it and doesn't immediately fess up.

Answer 13

Determining a loop really depends on the brand of switch that you have. For example, on an Extreme switch, I can run elrp-client on a VLAN and the switch will basically send out a broadcast frame on all ports for that VLAN and see if it returns by any of them, if so, it tells me which port(s) the frame was received back on, thereby revealing the loop candidates.

On a Cisco, you can enable storm control, which is a bit more of a blunt instrument since it will basically block the port for a period of time until the status clears (or you clear the errdisable state) - generally speaking however, this sort of thing is only relevant when you're using Cisco switches in a mixed topology of devices that do not do spanning tree nor forward BPDUs.

Answer 14

Without a doubt the fastest approach I have found is by monitoring the packet/sec rates of interfaces. A quick show interfaces with appropriate CLI filter will list each interface and the packet/sec rate. To find the source of the loop look for the only interface with a crazy high packet/sec INPUT rate. Within a typical enterprise environment, with typical utilisation profiles, it works everytime without fail. On a 6500 with many interfaces it doesn't take very long to spot the source...

Answer 15

During loop, for a large number of broadcast traffic (eg ARP Request) at the end station may also increase the load on the CPU (for example if you are using cheap 100Mbit/s realtek card that calculate a checksum on CPU). As physically possible to find a loop if the cable is disconnected, the link lost immediately on 2 ports.