3

My office has about 150 systems to be connected to a network. Based on the budget and availability of devices in my city I have the option of a HPE or Cisco 48 port Gigabit switch. But now when I interconnect the switches using the standard ports the inter switch bandwidth will be 1 Gigabit. So if more than 2 of my systems from 2 different switches transfer data between each other, the 1 Gigabit bandwidth is shared, hence the inter-computer communication is slow. Could someone suggest methods by which I can speed up the communication between the switches?

The most recommended model that I have come across is to use the SFP+ port on the switch with a transceiver capable of 10 Gigabit and connect all the switches to a 8 port 10 Gigabit switch. This appears to be expensive and I am unable to get further information on this setup. Is this the best model?

Is there a better model or a different model that is more cost effective?

Improve this question
2
  • Cisco has some best practices around oversubscription, which is really inevitable. Your total access port bandwidth to the uplink bandwidth ratio should be 20:1 or less. That means for every twenty 1 Gbps access ports, you need 1 Gbps in the uplink from the access switch to the distribution switch. From the distribution to the core, the bandwidth ratio should be no more than 4:1. – Ron Maupin♦ Nov 14 '16 at 1:08
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer. – Ron Maupin♦ Aug 15 '17 at 5:22
2

Normally you want your interconnection ports to be faster than your access ports to avoid one user filling the network. So if you have 1G access ports you most likely want 10 gigabit inter switch interconnects with a 10G core switch.

Unfortunately there are two different common types of ports used for 10G and the two cannot easily be connected to each other. The types are SFP+ and 10GBASE-T.

SFP+ was designed as a slot for fiber transceivers but for short links it is also possible to directly connect two SFP+ slots with a direct attach cable. There are two relevant types of SFP+ transceivers for a single-site greenfield deployment, SR and LR. Both operate over a pair of fibers.

SR is for multimode. On good quality multimode cable it should be good for 400m on OM4 (less on lower grades). The downside of multimode is it tends not to be very future proof. Each increase in speed results in a significant reduction in operating distance for a given fiber type.

LR is for single mode fiber. It's good for 10km and single mode fiber. Single mode fiber tends to be far more future proof than multimode. Downside is that the installation tolerances are less forgiving and the transcievers are more expensive.

10GBASE-T is a physical layer for twisted pair copper (ideally cat6a but for shorter runs cat6 should also work). Twisted pair is a lot cheaper than transcievers and fiber or SFP+ direct attach cables. Downside is it's relatively power hungry, the range is relatively limited, there is less choice of devices, the 10GBase-T versions of devices are often more expensive than the SFP+ versions and it will likely leave no headroom for upgrading to faster speeds on the same cabling later.

AIUI the big names like to make you pay through the nose for 10G. Some of them also try to bully you into buying their overpriced transcievers. AIUI if you want economical 10G gear Netgear are apparently a good choice.

With some switches it is possible to aggregate ports but it's less than ideal, especially if the number of data streams is small. I would push for 10G over aggregated 1G even if it costs a bit more.

Improve this answer
2
  • Several of the low-cost vendors will sell a guaranteed-to-work-with "brand X" SFP+ for a reasonable price, often the same as their generic model. Shopping without reaching the point of buying I saw $17 MM and $35 SM... – Ecnerwal Nov 14 '16 at 1:28
  • @Sisyphus - All three answers and many comments provided are very helpful. There is the mention in your original post of 'the inter-computer communication is slow' that is strange as to this day I see very large high-schools still on 1G multimode that has excellent throughput between clients/servers. To make the expense for upgrading to the recommended 10G SFP+ over SM usually is warranted with a business case and some troubleshooting of why the upgrade is needed. Not too sure if you've tested your infrastructure to find the bottlenecks. Can you post a diagram of your switching infrastructure? – user4565 Nov 15 '16 at 19:38
2

Simplest and cheapest approach (other than not worrying about it at all) is to figure that you have 150 devices, so you need 4 48 port switches, but you don't need 42 of those ports for devices, so you exploit Link Aggregation Control Protocol (LACP) to run multiple cables between switches (without creating loops.) You need a switch that supports it, but most do these days.

More moderately expensive is to skip the 10gig switch and just cable between your 4 switches with 10 Gig from their SFP+ ports. Depending on layout and availability of SFP+ ports you can either use one switch as a star for the rest, or just put them in line. That's only one step "worse" than a 10 gig switch and only for traffic between the "end-most" switches. The model where you have a (probably 4 SFP+) switch with at least 3 SFP+ ports to act as a star is "as good" as a central switch for traffic between the leaves, and better for traffic from the hub to any leaf.

sketch of switch configs

In the LACP sketched, I put more links between the two switches in the middle. Each end link uses 10 ports (5 on each switch) and the central link uses 14 (7 on each switch) for a total of 34, leaving you with 8 free ports (as sketched - this is entirely configurable.) Edit, add: Moving this up from the comments - if you take the simple rightmost sketch and add 2 more SFP+ to loop from the end-most switches to each other, with RSTP (or STP, or...) to block the ports when everything is functional, you will have 75% of the network working when any one switch goes out - of course, if it's the one connected to the (fileserver, router, etc) you will still need to repatch or replace to be functional again. If all are in one rack, that's rather quick - if they are in 4 different rooms, it's harder.

Improve this answer
6
  • I understand your answer, and it is actually correct, but I shudder at the thought of the single-points-of -failure. The middle drawing has half the other two. I just think that organizations just don't think far enough ahead, and some relatively small amount of money now could save losses of millions of dollars later, but they are too cheap to spend the money for a good design to start with. We just don't allow an access switch to connect to another access switch, and that was after some very painful lessons. – Ron Maupin♦ Nov 14 '16 at 1:03
  • "relatively small amounts of money" for one organization can be "prohibitively large amounts of money" for others. It's good to know all the possible approaches, as well as the potential pitfalls, and solutions thereto if the end-of-year budget fairy comes by with a basket of money and a short deadline to spend it by. If you want to get into RSTP you can of course loop from end to end, as well. – Ecnerwal Nov 14 '16 at 1:09
  • Well, some thousands of dollars for a correct network design compared to millions of dollars in lost revenue for a SPoF is relatively small. I do understand that there are many companies for which this can seem too large an upfront cost, but I have seen others that, in hindsight, are really kicking themselves. – Ron Maupin♦ Nov 14 '16 at 1:13
  • You may have thousands to spend and millions to lose; that's not a given for every enterprise. If I tried to build my network on that model I'd still be waiting for funds to start building it. Instead we've been using it for six years, and it couldn't lose a million dollars if it was down for a month (which would be highly unlikely.) When there ARE millions to lose it's a lot easier to shake the tree for more funds and redundancy. – Ecnerwal Nov 14 '16 at 2:44
  • 1
    I have to agree with Ron on this one, if you were to use two of the switches as a core and then have one 10G uplink plus one slower (say 4x 1G bundle) as a secondary for each of the other two switches. The slower one could be costed to be blocking in normal operation. This design would minimise the impact of switch failure with the same number of 10G SFP+ as your star, and a few extra 1G ports and cables. Good design does not have to be perfect design. There's a big difference between none of your users working and 75% of them working slowly, even if it is only for half-a-day. – marctxk Nov 14 '16 at 10:47
1

Typical use cases seldom consume all available bandwidth at each port and, therefore, aggregated bandwidth on your uplinks will probably use less than 1Gb unless you're pushing lots of packets. This is called oversubscription which is an expected design element.

I have access switches in a data center with a couple of hundred web server VMs connected to each and all their uplink traffic fits over two pairs of 2x1Gb bonded (etherchannel) ports. That's a 200:4 or 50:1 rough oversubscription.

You really need to understand your expected traffic patterns to determine if oversubscription will work (and at what ratio) or not. Etherchannel can combine eight 1Gb links to make an 8Gb pipe, so consider that as a cheaper alternative to 10Gb.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.