Network topology with big spanning tree diameter

Question

I have a quite big network with spanning tree diameter 26 (if I count it correctly). It is build on Cisco swithes (mainly Catalyst 2960 and Catalyst 1000, two SG350), running rapid-pvst. It is now running with default spanning tree timers, that are counted for diameter 7. I think that this makes troubles when topology change occurs. What timers should I set to make it working properly? There are five loops.
I know, that it is not good to have such a big diameter, but I cannot change it. It is connected via fiber optics and distance of switches is in kilometers.

Answer 1

I guess you aren't familiar with the infamous "All Systems Down" article from 2003. [ archive ] I can't find the original CIO article, and it was never archived.

In simple terms, spanning-tree doesn't work beyond the designed limits. Your network is a timebomb. There's only one place where STP could detect a loop -- the 4 switch core. This really needs to be redesigned, but not knowing the physical topology, we can't begin to guess how to go about it. The first thing to consider is breaking the flat layer-2 domain into multiple routed layer-3 segments.

If you insist on playing with the timers, the limit is 17-18. [ the math ]

Answer 2

That topology far exceeds the xSTP design maximum. As you've stated, STP with default settings can handle a network with a maximum diameter of 7. As Ricky has shown so aptly, adjusting those settings may allow a diameter of up to 18.

Even if your network wasn't exceeding that already, a link disruption in the wrong place can substantially increase the diameter, likely rendering your network non-functional.

RPVST+ creates an independent spanning tree for each VLAN, so if your network doesn't exceed the maximum diameter within any VLAN, it might even work. However, I suspect that that isn't the case. Since losing links like CPS2-V7 or CPS3-SAT would seriously increase the network diameter, chances are high that your network might not recover from a single link loss ("train wreck waiting to happen"). Check out the link in Ricky's answer for how that would look like.

Actually, you'd be better off removing the closing loops S5-S6 and LIP-RUD and deactivating STP on those branches altogether. The redundancy those loops are supposed to create is just imaginary anyway.

Since merely tweaking STP settings can't solve your problem, there are three basic approaches:

1. Using (proprietary!) RSTVP+, decrease effective diameter by shortening the VLAN reaches (=remove VLANs from switches and trunks wherever possible). Alternatively, switch to MSTP and create independent regions. In both cases, tweaking the STP settings is still required and might not even be enough.
2. Connect all outer-ring switches to the inner-ring switches (CPS2, CPS3, HUS1, HUS2) directly. As you likely can't run extra fiber you could use WDM to leverage the already present strands.
3. Change the inner ring to routed links instead of switched ones. That is the most substantial redo but offers the best in performance and growth potential. Also, it may not even require additional hardware.