4

I'm concerned that I'm misunderstanding how aggregators work in the scenario below:

I have a project where I need 40 Gbps links between switches. My plan was to have my three switches use all four of the 10Gb uplink ports to a fourth "aggregator switch" so that I'd have the necessary bandwidth for a total of 12 10Gb links to the aggregator.

Does link aggregation truly work as if you have an aggregated link? Is this how I want to achieve the 40Gb uplink between switches?

According to IEEE 802.1ad specification, it strikes me as being more about failsafe/failover than about using multiple links to achieve a larger pipe.

If this is not the correct approach can someone suggest a better and more common approach?

7

What really happens is that any one flow only uses one of the links. Different flows are assigned to different links based on a hashing algorithm, so, in aggregate, you get the full bandwidth of the combined links, but any one flow will only get the bandwidth of a single link.

You don't want to spread a single flow across multiple links because that will create problems with out-of-order packet delivery. Protocols, like TCP, can deal with that, but it can slow them down. Other protocols like UDP cannot deal with that, and it is complete death to real-time protocols that use UDP, such as voice or video.

It does provide redundancy, too, but you really are getting the full bandwidth of all the links in the channel, but in aggregate over all flows.

If that doesn't work for you, then you need to get devices with 40 or 100 Gbps interfaces.

| improve this answer | |
  • so if I'm connecting switches A, B, and C to the backbone switch D via home runs from each switch, you're saying that I'll have poor performance by spreading the 10Gb uplinks to the back bone? – inbinder Jul 24 '17 at 21:07
  • 1
    No. If you have multiple flows coming into the access switch, then you will use the full bandwidth of the aggregated uplinks to the distribution switch. Each individual flow cannot use more than the bandwidth of a single link, but that is unlikely anyway since you probably have 1 Gbps access interfaces. What the aggregation does is to fool spanning tree, which would normally block all but one of the links to the distribution switch from an access switch. – Ron Maupin Jul 24 '17 at 21:28
  • 1
    @inbinder, just to be clear, you are not trying to aggregate the 12 links on the distribution switch? That will not work because you can only aggregate links between two devices. You can have an aggregation with each access switch on the distribution switch. Look at the oversubscription ratio for access to distribution. Cisco recommends no more than a 20:1 access to distribution bandwidth ratio, and no more than a 4:1 distribution to backbone bandwidth ratio. – Ron Maupin Jul 24 '17 at 21:32
  • --do switches exist with copper/rj45 support along with a SFP/SFP+ 40/100Gbps uplink? – inbinder Jul 25 '17 at 15:52
  • 1
    In some circumstances, some vendors offer some support for 40 and 100 Gbps copper interfaces. For Example, Cisco offers QSFP modules, but these are limited to certain switch models. Other vendors have other solutions. – Ron Maupin Jul 25 '17 at 16:05
4

Adding to Ron's excellent answer, with the vast majority of switches there are three different hashing algorithms: MAC address (L2), IP address (L3) and IP address plus port numbers (L4).

The algorithm combines both source and destination addresses and mixes them together in its own special way (sometimes documented) to choose a somewhat random yet consistent forwarding port that remains the same throughout the MAC / IP / L4 session.

E.g. a switch may use the lower six bits of the source and of the destination IP address, XOR them and use the result as the index of the port within the trunk.

In a nutshell, you can very well use a single 10G link to connect two switches with 1G ports and expect to get ten full-speed flows. However, if you trunk four 10G links between two switches you can't expect them to pass four different 10G flows at full speed at all times - only if you're very lucky or you've hand-picked the addresses to optimize the flows and know exactly which flows happen at the same time.

| improve this answer | |
  • 1
    Like adding salt to the egg, +1 :-) – user36472 Jul 25 '17 at 6:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.