Theoretical: Bottleneck in a switch stacking environment

Question

Quick question regarding stacking environments and their limitations regarding traffic bottlenecks.

Assuming the following use case:

• We have a stack of 8 switchs, forming a Half 10Gbps stack ring.
• Switch 1 master
• Each switch has 48 ports at max 100Mbps, so each switch could have a maximum of 4.8Gbps of traffic
• The uplink of 10Gbps is on the master

Assuming the members 2, 3, and 4 are at 4.8Gbps each, so am I true if I say there is a bottleneck between the member 2 and 1 stack cable in this case ? (4.8 x 3 = 14.4gbps)

Same question with a Full ring, I think we have no longer the bottleneck because of the traffic goes also from the other direction to reach the member 1 master ? (traffic of switch 2, 3 and 4 could pass through members 5, 6, 7, 8, and 1 ?)

Thanks.

Answer 1

You have not given us the switch model, but you tagged your question with the Cisco Catalyst tag, so I will use the Cisco Catalyst 3850 as an example.

The Cisco Catalyst 3850 has a pretty large stack switching capacity:

Cisco StackWise-480 technology provides scalability and resiliency with 480 Gbps of stack throughput

That means you could have 48 gigabit ports per switch for 10 switches (I believe that nine is the actual stack limit) and still have wirespeed switching in the stack.

Your bottleneck will be the uplink from the stack. Cisco recommends a 20:1 access to distribution bandwidth ratio. That means for every 20 gigabit access interfaces, you need one gigabit in the uplink.

Answer 2

With a 10 Gbit/s uplink, 10 Gbit/s stack interlinks and only 100 Mbit/s edge ports the uplink bottleneck is rather theoretical.

48 FE ports on each switch can transport an aggregate, peak edge-to-core bandwidth of 4.8 Gbit/s, so up to two switches won't be able to saturate the uplink bandwidth. The most probable bottleneck is the 10G uplink with an oversubscription of just 3.8:1 - with a "standard" network this is far below the recommended subscription ratio of 20:1 and gives plenty of headroom.

However, mileage varies and with a workload of a full 100 Mbit/s on each edge port permanently, a single 10G uplink won't cut it. This is why you usually avoid rings/chains and use a central switch with the others grouped around (a "fat tree") or spine-leaf in a larger network.

Answer 3

The Cisco switch model makes a big difference in whether or not you hit a bottleneck on the stack ring - more than the raw bandwidth numbers will show.

The 3750(G) switches had a much different stack architecture (Stackwise) than the newer StackWise plus on the 3750(X) and the Stackwise-480 found on the Cisco 3850 switches. The 3750 stack will not do local switching - every Ethernet frame going from any interface to any other interface will go across the stack ring. This includes traffic going between interfaces on the same physical switch. Stackwise also does what is called source stripping, where the packet is removed from the ring only after it returns to the source switch.

The 3750X with StackWise Plus and the 3850 switch with Stackwise-480 can do local switching mainly because it has centralized design for all layer 2 and layer 3 protocols. The 3750 uses distributed design for the MAC address table and SPT. There are other inherent reasons in the different designs as well.

https://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3850-series-switches/white-paper-c11-734429.html#_Toc417362558

How would this affect your theoretical bottleneck question? If you have a large amount of locally switched traffic (servers to database etc.. ) you can hit a bottleneck on your stack's ring before even getting close to your maximum upload capacity on your uplink.

The paper you want to read to understand the differences between these different stacking protocols and which also discusses throughput bottlenecks is the,

Cisco StackWise and StackWise Plus Technology white paper, easily found with a google search. If you search for stackwise plus white paper it will be the first Google search result.