Interconnecting Gigabit Switches with Maximum Bandwidth

Question

My office has about 150 systems to be connected to a network. Based on the budget and availability of devices in my city I have the option of a HPE or Cisco 48 port Gigabit switch. But now when I interconnect the switches using the standard ports the inter switch bandwidth will be 1 Gigabit. So if more than 2 of my systems from 2 different switches transfer data between each other, the 1 Gigabit bandwidth is shared, hence the inter-computer communication is slow. Could someone suggest methods by which I can speed up the communication between the switches?

The most recommended model that I have come across is to use the SFP+ port on the switch with a transceiver capable of 10 Gigabit and connect all the switches to a 8 port 10 Gigabit switch. This appears to be expensive and I am unable to get further information on this setup. Is this the best model?

Is there a better model or a different model that is more cost effective?

Answer 1

Normally you want your interconnection ports to be faster than your access ports to avoid one user filling the network. So if you have 1G access ports you most likely want 10 gigabit inter switch interconnects with a 10G core switch.

Unfortunately there are two different common types of ports used for 10G and the two cannot easily be connected to each other. The types are SFP+ and 10GBASE-T.

SFP+ was designed as a slot for fiber transceivers but for short links it is also possible to directly connect two SFP+ slots with a direct attach cable. There are two relevant types of SFP+ transceivers for a single-site greenfield deployment, SR and LR. Both operate over a pair of fibers.

SR is for multimode. On good quality multimode cable it should be good for 400m on OM4 (less on lower grades). The downside of multimode is it tends not to be very future proof. Each increase in speed results in a significant reduction in operating distance for a given fiber type.

LR is for single mode fiber. It's good for 10km and single mode fiber. Single mode fiber tends to be far more future proof than multimode. Downside is that the installation tolerances are less forgiving and the transcievers are more expensive.

10GBASE-T is a physical layer for twisted pair copper (ideally cat6a but for shorter runs cat6 should also work). Twisted pair is a lot cheaper than transcievers and fiber or SFP+ direct attach cables. Downside is it's relatively power hungry, the range is relatively limited, there is less choice of devices, the 10GBase-T versions of devices are often more expensive than the SFP+ versions and it will likely leave no headroom for upgrading to faster speeds on the same cabling later.

AIUI the big names like to make you pay through the nose for 10G. Some of them also try to bully you into buying their overpriced transcievers. AIUI if you want economical 10G gear Netgear are apparently a good choice.

With some switches it is possible to aggregate ports but it's less than ideal, especially if the number of data streams is small. I would push for 10G over aggregated 1G even if it costs a bit more.

Answer 2

Simplest and cheapest approach (other than not worrying about it at all) is to figure that you have 150 devices, so you need 4 48 port switches, but you don't need 42 of those ports for devices, so you exploit Link Aggregation Control Protocol (LACP) to run multiple cables between switches (without creating loops.) You need a switch that supports it, but most do these days.

More moderately expensive is to skip the 10gig switch and just cable between your 4 switches with 10 Gig from their SFP+ ports. Depending on layout and availability of SFP+ ports you can either use one switch as a star for the rest, or just put them in line. That's only one step "worse" than a 10 gig switch and only for traffic between the "end-most" switches. The model where you have a (probably 4 SFP+) switch with at least 3 SFP+ ports to act as a star is "as good" as a central switch for traffic between the leaves, and better for traffic from the hub to any leaf.

In the LACP sketched, I put more links between the two switches in the middle. Each end link uses 10 ports (5 on each switch) and the central link uses 14 (7 on each switch) for a total of 34, leaving you with 8 free ports (as sketched - this is entirely configurable.) Edit, add: Moving this up from the comments - if you take the simple rightmost sketch and add 2 more SFP+ to loop from the end-most switches to each other, with RSTP (or STP, or...) to block the ports when everything is functional, you will have 75% of the network working when any one switch goes out - of course, if it's the one connected to the (fileserver, router, etc) you will still need to repatch or replace to be functional again. If all are in one rack, that's rather quick - if they are in 4 different rooms, it's harder.

Answer 3

Typical use cases seldom consume all available bandwidth at each port and, therefore, aggregated bandwidth on your uplinks will probably use less than 1Gb unless you're pushing lots of packets. This is called oversubscription which is an expected design element.

I have access switches in a data center with a couple of hundred web server VMs connected to each and all their uplink traffic fits over two pairs of 2x1Gb bonded (etherchannel) ports. That's a 200:4 or 50:1 rough oversubscription.

You really need to understand your expected traffic patterns to determine if oversubscription will work (and at what ratio) or not. Etherchannel can combine eight 1Gb links to make an 8Gb pipe, so consider that as a cheaper alternative to 10Gb.