How does link aggregation work in reality?

Question

I'm concerned that I'm misunderstanding how aggregators work in the scenario below:

I have a project where I need 40 Gbps links between switches. My plan was to have my three switches use all four of the 10Gb uplink ports to a fourth "aggregator switch" so that I'd have the necessary bandwidth for a total of 12 10Gb links to the aggregator.

Does link aggregation truly work as if you have an aggregated link? Is this how I want to achieve the 40Gb uplink between switches?

According to IEEE 802.1ad specification, it strikes me as being more about failsafe/failover than about using multiple links to achieve a larger pipe.

If this is not the correct approach can someone suggest a better and more common approach?

Answer 1

What really happens is that any one flow only uses one of the links. Different flows are assigned to different links based on a hashing algorithm, so, in aggregate, you get the full bandwidth of the combined links, but any one flow will only get the bandwidth of a single link.

You don't want to spread a single flow across multiple links because that will create problems with out-of-order packet delivery. Protocols, like TCP, can deal with that, but it can slow them down. Other protocols like UDP cannot deal with that, and it is complete death to real-time protocols that use UDP, such as voice or video.

It does provide redundancy, too, but you really are getting the full bandwidth of all the links in the channel, but in aggregate over all flows.

If that doesn't work for you, then you need to get devices with 40 or 100 Gbps interfaces.

Answer 2

Adding to Ron's excellent answer, with the vast majority of switches there are three different hashing algorithms: MAC address (L2), IP address (L3) and IP address plus port numbers (L4).

The algorithm combines both source and destination addresses and mixes them together in its own special way (sometimes documented) to choose a somewhat random yet consistent forwarding port that remains the same throughout the MAC / IP / L4 session.

E.g. a switch may use the lower six bits of the source and of the destination IP address, XOR them and use the result as the index of the port within the trunk.

In a nutshell, you can very well use a single 10G link to connect two switches with 1G ports and expect to get ten full-speed flows. However, if you trunk four 10G links between two switches you can't expect them to pass four different 10G flows at full speed at all times - only if you're very lucky or you've hand-picked the addresses to optimize the flows and know exactly which flows happen at the same time.