6

I am trying to increase bandwidth for particular hosts by using lacp on a Cisco Catalyst 4500 switch. However, it seems that I will always end up with the same flow (src/dst mac, ip, and port) being completely switched over one interface, no matter what port-channel load-balance option I choose.

Why is there no such thing as a round robin option for this? (It seems that some older models had this option)

migrated from serverfault.com Sep 15 '16 at 14:35

This question came from our site for system and network administrators.

  • Round-robin is actually a very bad way to do this. It causes out of order packet delivery, and that can kill some protocols, especially real-time protocols. – Ron Maupin Sep 16 '16 at 0:34
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer. – Ron Maupin Aug 15 '17 at 2:31
5

From my understanding, LACP always requires that packets in a particular network "conversation" (a TCP session or a UDP stream between the same source and destination ports) traverse a single LACP path, in order to guarantee delivery order. LACP chooses a path based on a hashing algorithm. In Linux, the hashing algorithms are those described in the bonding driver's xmit_hash_policy setting:

  • layer2 - meaning the hash consists of the source and destination layer 2 (MAC) addresses;
  • layer2+3 - meaning the hash consists of the layer2 information, plus the source and destination layer 3 (IP) addresses;
  • layer3+4 - meaning the hash consists of the layer 3 (IP) addresses plus whatever layer 4 addressing information is available (e.g. TCP or UDP source and destination port numbers, if the packets are not fragmented).

Cisco has a number of additional options.

The server and switch do not need to use the same hashing algorithm; all that is required is that the packets for a particular conversation choose the same path as long as that path is available.

  • For "guaranteed delivery order", we are talking about the wrong OSi layer here. To quote Wikipedia (emph mine): "When balancing traffic, network administrators often wish to avoid reordering Ethernet frames." This suggests that it should be configurable, but not the only option – Hagen von Eitzen Feb 21 '18 at 9:11
2

LACP controls the aggregation of links, it doesn't control switching. The selection of a particular member of a bundle is via an algortithm decided upon and implemented by each switch vendor. It has to be something that can be programmed into hardware in order to operate at wire speeds relatively cheaply, and a hash of various fields of an Ethernet frame is suitable for this. To perform round-robin the ingress hardware would need to know all about the bundle members and keep state about previous frames switched. That problem is an order of magnitude more complex. I don't know of any switches that perform round-robin across a LAG.

Linux and ESX etc can perform round-robin in software, that's fairly straightforward in comparison since the entire interface selection process is already in software. Also notice that typically servers have a higher transmit requirement than receive and so for many applications the increased bandwidth that round-robin achieves is only required server-to-switch.

  • Re "only required server-to-switch" - I disagree. With flow-based decision, a server can receive up to 8Gbps, say, but at most 1Gbps per flow. With round-robin, the server then might transmit up to 8Gbps and even in just one flow. But if these 8Gbps cannot get out of the switch (cause it allows only 1Gbps per flow), there will be congestion and ultimately only 1Gbps per flow as well. – Hagen von Eitzen Sep 15 '16 at 16:54
  • 2
    Also, I fail to understand how incrementing a three-bit counter per port can be an order of magnitude more complex than performing a hash computation that turns 24 bits into 3 bits in a fashion complex enough to be called a hash in the first place ... – Hagen von Eitzen Sep 15 '16 at 16:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.