5

We ran into a problem the other day when a member of our staff inadvertently doubly connected in a cheap switch (no STP) into another switch that had STP disabled. (The second switch was connected to our main network.) The resulting packet storm brought down our network.

If our simplified network topology looks something like this

.--.
|S1|
'--'
  |
.--. ___ .--.
|S2| ___ |S3|
'--'     '--'

(Sorry for the crude ascii art, I'm trying to illustrate that S2 and S3 are connected twice, creating a bridging loop)

  • S1 is actually a rack of Cisco switches
  • S2 is a Netgear smart switch with STP disabled
  • S3 is an embedded switch with no STP

This area isn't my forte, so I'm not even sure if this is possible, but if our network looks something like this, is there a way to protect S1 from the ensuing broadcast storm between S2/S3?

Update with a little more info:

  • Rack-mounted Cisco switches are a mix of SG200-50's, -26's, and -18's
  • Netgear is a GS108Tv2 running firmware v5.0.5.7
  • The more esoteric piece of equipment is a small embedded switch that goes in an industrial controller, link is here: https://www.phoenixcontact.com/online/portal/us?uri=pxc-oc-itemdetail:pid=2891001&library=usen&pcck=P-08-08-10-06-01&tab=2. The PDF (downloads tab) indicates something about a "multi-address function" but I don't think this is STP/RSTP.

Thanks for all the answers so far. I really appreciate it.

Improve this question
8
  • 3
    We would need to know the model (and possibly code version) of the S2 device before we could answer this. There are steps you could take on the S1 switches (again would need models/versions) to limit the scope of the problem to the S2 and S3 switches.
    – YLearn
    Dec 16 '13 at 15:48
  • If you have a switch that doesn't support STP, then nothing critical should really be run on it. If nothing critical is running on that switch, then unplug the redundant link and you have solved the problem entirely. Is there a particular business need to have redundant links between a netgear switch and then another embedded?
    – John Kennedy
    Dec 16 '13 at 15:48
  • 4
    run broadcast storm control on all your switchports as well as hw control plane rate limiters, if those are available. BPDUguard is a good idea as well.
    – Mike Pennington
    Dec 16 '13 at 15:55
  • 3
    @ron, the best advice is avoiding non STP switches in your network. storm control is not a bandaid, it is your airbag to survive broadcast storms without taking down the rest of the network
    – Mike Pennington
    Dec 16 '13 at 16:16
  • 1
    @John, the second connection between switches wasn't intended to provide redundancy or anything, it was just an inadvertent connection. The point behind my post is to see what we can to do mitigate the problem on the corporate network (S1) if a user accidentally does this again. Thanks.
    – Dan
    Dec 16 '13 at 18:11
4

Agree with both Ron and Mike Pennington here. The entire point of Spanning Tree is to prevent broadcast storms from blowing up your network and you have just recently observed a practical lesson in how that happens.

Knowing the version of your Netgear switch would be helpful. I'm looking at an older version of the ProSafe switch user manual for the GS748TS and the Spanning-Tree settings are quite granular, so you should be able to configure the switch to prevent such things from happening in the future.

On the older NetGear ProSafe switches I have used the default configuration is that STP is enabled in RSTP (rapid spanning tree) with BPDU flooded to all ports on the switch. This is a good configuration for everyday use. In order to help you more, we would need to know how many VLANs are on your network and so forth.

But at the end of the day this is a hard lesson in the need for company policies that govern what can and cannot be connected to a production network. At my employer (a large Fortune 500 manufacturing enterprise) our network policy is that all access ports on switches are configured with BPDUguard enabled such that when any unknown switching device is plugged into it the port becomes error disabled. If you have SNMP monitoring on your switches, then you will get an alert.

There is a small price to be paid in user frustration but this is more than made up for in the fact that you will automatically have visibility and hopefully control over what gets connected and you won't have to deal with the network outages in the future.

Improve this answer
6

another switch that had STP disabled

I don't have to read any further than that. "Doctor, it hurts when I do X..." Obviously, you shouldn't be disabling STP on anything that supports it, lest this very situation occur. However, given your topology, STP should've partitioned off the S1-S2 link. But seeing as those are LINKSYS switches with a Cisco badge on them, I don't expect them to handle this correctly. Nor do I expect a Netgear GS108T -- a cheap, simple switch (I have a few) -- to have robust storm-control.

Really the only thing that can be said is "Don't do that." (where "that" equals (a) "disable STP" and (b) "plug a switch into itself". I cannot make you do either of those things. And you cannot be sure someone else won't do (b).)

I don't want to sound like I'm pushing the more expensive Cisco switches, but I have the same type of situation pop up from time to time in our VMware lab when someone configures a virtual load balancer (alteon, if you care) with two interfaces in the same network. Our Cisco 2960S's immediately kill that port -- errdisable self-looped port.

Improve this answer
2
  • 3
    so many companies ignore feature gaps in the small business switches and treat them like true enterprise class switches. Months ago, someone objected when I called them prosumer switches, but that's really where they fit. Don't plan on building serious networks with them
    – Mike Pennington
    Dec 16 '13 at 18:53
  • 1
    Indeed. Just because there's a Cisco logo on it, doesn't mean it's Cisco Enterprise Quality(tm) -- which isn't what it used to be either. Those are "SOHO" devices; if your network consists of only that (one) device, then you're doing it right.
    – Ricky
    Dec 17 '13 at 1:12
2

I ran into an issue with these taking our whole network down. 6 Extreme Stacks, 3 HP switches various models and several atleast 8 of these netgear gs108's Gs108T with no spanning tree turned on looped to itself and hooked to any switch except extreme switches with ELRP turned on will eventually take the whole network down. It looks like a broadcast storm or multicast storm. I had a laptop hooked up on a spanned/mirrored port and what you will see is that there are a lot of dup-acks and tcp retransmission packets going through the network for the equipment or pcs that are having issues. My suggestion to see what its doing for yourself is to turn on mac-address tracking and watch what happens. The netgear basically will become a network blackhole. When I performed this experiment to figure out why I was seeing a loop and spanning tree was not taking care of it I found to my horror all the mac addresses on the network started moving from switch to switch finally to the netgear. I found that ELRP could detect the loop so I turned that on to help prevent that from happening again. This cost our company a lot of money in downtime and replacing a few switches along with a trip to Alaska for a week to troubleshoot.

These switches were chosen to replace dumb netgear switches. Think twice before deploying them in critical environments. One person plugging one in without configuring it can take down the whole network. BTW they come with spanning tree turned off on all the ports

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.