0

I'm testing bandwidth between two switches and I noticed link aggregation behaves inconsistently between test runs. I'm testing with a single instance of iperf with 2 streams. Sometimes the load is split evenly between each half of the port channel, and other times it goes over just one link. Overall throughput is much better when it goes over one link. This appears to be random each time I re-run the same iperf command. Sample command: iperf -c 10.49.54.40 -P2 -t 3600.

The two switches are connected using a 200Gb LACP port channel made of 100Gb links. Each host has a single 100Gb link.

┌─────────┐         ┌─────────────┐           ┌──────────────┐          ┌─────────┐
│         │         │             │      Et17 │              │          │         │
│ Host A  │         │  Mellanox   ├───────────┤  Arista      │          │ Host B  │
│         ├─────────┤   SN2700    ├───────────┤   7050X      ├──────────┤         │
│         │         │             │      Et19 │              │ Et22     │         │
└─────────┘         └─────────────┘           └──────────────┘          └─────────┘

Both hosts are running iperf 2.0.13 on CentOS. Below is the switch bandwidth showing from 2 different test runs.

#show interfaces counters rates | grep -E 'Po5002|Et17|Et19|Et22|Name'
Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et17/1    HOST::LR06.  0:05       1.1   0.0%        0   25602.8  25.7%      356
Et19/1    HOST::LR06.  0:05      22.7   0.0%       38   28257.6  28.3%      393
Et22/1    HOST::LR06.  0:05   53854.3  54.0%      749      22.3   0.0%       38
Po5002    HOST::LR06.  0:05      22.7   0.0%       38   53548.9  26.8%      745

#show interfaces counters rates | grep -E 'Po5002|Et17|Et19|Et22|Name'
Port      Name        Intvl   In Mbps      %  In Kpps  Out Mbps      % Out Kpps
Et17/1    HOST::LR06.  0:05       1.1   0.0%        0      49.0   0.0%        0
Et19/1    HOST::LR06.  0:05      27.1   0.0%       47   67252.8  67.4%      936
Et22/1    HOST::LR06.  0:05   67295.0  67.4%      936      28.0   0.0%       47
Po5002    HOST::LR06.  0:05      27.6   0.0%       46   66798.6  33.5%      930

Here is the port channel hashing config for IPv4 on Arista side:

Hash seed is 0
Hash function is CRC32HI
Source MAC address hashing for IP packets is OFF
Destination MAC address hashing for IP packets is OFF
IP protocol field hashing is ON
Ingress interface for IP hashing is ON
IP source address hashing is ON
IP destination address hashing is ON
IP TCP/UDP/SCTP source port hashing is ON
IP TCP/UDP/SCTP destination port hashing is ON
IP TTL hashing is OFF
IP symmetric hashing is OFF
System MAC address hashing is ON
Improve this question
5
  • 1
    That's the port-channel's status, not config. You'll need to look at the exact configuration in both direction to determine the path selection algorithm. LACP is not a "per packet" thing; a single TCP connection will ALWAYS follow one link. (to avoid the issue of out-of-order delivery.) [I have NDA docs that show broadcom's exact math.]
    – Ricky
    Commented Nov 5, 2022 at 3:55
  • Thanks, I've updated the question to show hashing config. I'm running iperf with 2 streams so it should be able to take 2 paths.
    – Elliott B
    Commented Nov 7, 2022 at 18:11
  • 1
    "I'm running iperf with 2 streams so it should be able to take 2 paths." Not necessarily. If the two streams hash to the same interface, then they share the bandwidth of that interface. You need a lot of different streams, then each interface will have some, but they may not be evenly divided.
    – Ron Maupin ♦
    Commented Nov 7, 2022 at 18:13
  • Each interface will have the bandwidth of that interface. The bandwidth does not change. A 1 Gbps interface has a bandwidth of 1 Gbps, no more, no less. The bandwidth is how fast the interface [de]serializes bits on the wire. I think you are trying to test the throughput, not bandwidth. The throughput will be less than the bandwidth because ethernet has overhead of seven Preamble octets, one SoF octet, the ethernet header 14 octets, four FCS octets, and a 12-octet inter-packet gap for each frame. The payload will be an IP packet with its overhead, and the transport protocol with its overhead.
    – Ron Maupin ♦
    Commented Nov 7, 2022 at 18:21
  • 1
    You cannot simply bond two interfaces in a channel and say that is the bandwidth. Each stream can use, at most, the bandwidth of one interface. Over an aggregate of many streams, you have an aggregate bandwidth of the combined interfaces.
    – Ron Maupin ♦
    Commented Nov 7, 2022 at 18:24

1 Answer 1

Reset to default
4

It depends on the LACP load balancing algorithm. The LACP load balancing algorithm is only locally significant (they don't have to match on Mellanox and Arista switch).

Load balancing algorithms are usually based on a hash from L2/L2+L3/L2+L3+L4 headers.

The reason you get different results, could be that your source TCP port is changing every time you run the iperf3 -P2 command (same src/dst mac, same src/dst ip, same dstport, but different srcport for every parallel stream in -P2).

For instance, if you have one even numbered and one odd numbered source port, they could get different hashes and therefor be sent on different LACP member interfaces.

Here is some info from the producers:

Improve this answer
3
  • I can't find any reference to even/odd in the Arista docs. I believe the hash should be different any time the port numbers are not the same, even if they both have the same "evenness". I updated the question to show my hash settings.
    – Elliott B
    Commented Nov 7, 2022 at 18:09
  • 1
    I said "could" as an example. I don't know the exact detail on how their hashing algorithms work, but it was an example to show there is a tuple of values that sets the transmit interface. This tuple will always be the same every time it has the same value. The only difference of your iperf3 tests is the src-port you use to do tcp-connection, and therefor it is likely that is why youre getting different results, because iperf3 used a random port on your -P2, and sometimes they hashed to the same interface, and sometimes they hashed to different interfaces.
    – OlofL
    Commented Nov 7, 2022 at 19:53
  • I tried some tests with different iperf source ports and I don't think that's the issue. It actually looks like this is related to current bandwidth on the switch. If bandwidth is very low, then it sends both streams through 1 path. If bandwidth is even slightly elevated, then it splits. I'm still trying to figure out the exact logic but it's something like that.
    – Elliott B
    Commented Nov 8, 2022 at 1:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.