0

We just finished debugging a really interesting bandwidth problem on intercontinental WAN links aggregated by Aruba switches. According to tcpdump captures, using a rate limiter of any speed on a port of a ProCurve 3500 switch will cause the switch to randomly reorder TCP packets sent through it!

The switch is on the latest firmware version, and there are no errata that I can find regarding this peculiarity. Reordering the packets causes the TCP fast retransmit to kick in every few dozen segments, so intercontinental TCP connections are only able to reach maybe ~100Kbps on a good day even with the rate limiter set at 100Mbps for that link.

Testing with UDP flows indicates the switch is functioning as intended in terms of actual bandwidth limits for raw data streams. The switch only starts reordering packets above some minimum threshold bandwidth, in our tests this was somewhere around 4Mbps. Clearing the rate limit without changing anything else allows full-speed TCP transfers over the high latency links and a far lower (more typical) number of TCP fast retransmits.

Tests were done with iperf3 and extensive use of the -b flag was made in order to keep the test data flows well below the configured rate limits. This was done to avoid dragging in the additional oddities of rate limiter behavior during actual congestion on the limited port.

Has anyone else seen this? Is this kind of behavior normal for switches in this class? We prefer the port rate-limited architecture for various reasons; these switches are used to interface with consumer systems with tiered data rates and IP filters / ACLs, but obviously if they are reordering packets they will need to be replaced.

Any assistance appreciated!

EDIT:

Once it became clear the switch was causing the bandwidth problems, two separate hosts were connected directly to two ports on the switch, and iperf3 and tcpdump were used on both hosts to check the data flows across the switch. What was consistently seen was a "block" of several packets being delayed / moved out of order, but not enough where the fast retransmit would have kicked in yet. Basically the following:

HOST A (receiver)       HOST B (sender)
SEQ 1 <         < SEQ 1
SEQ 2 <         < SEQ 2
SEQ 3 <         < SEQ 3
ACK 1 >         > ACK 1
ACK 2 >         > ACK 2
SEQ 4 <         < SEQ 4
ACK 3 >         > ACK 3
ACK 4 >         > ACK 4
 ... now the interesting part ...
SEQ 7 <         < SEQ 5
ACK 5 >         < SEQ 6
SEQ 8 <         < SEQ 7
ACK 5 >         < SEQ 8
SEQ 5 <         < SEQ 9
SEQ 6 <         > ACK 5

Notice how a couple of packets are moved earlier in the transmissions sequence from the receiver's perspective; also note how the received sequence numbers jump backward before the fast retransmit should have kicked in. I omitted the rest of the sequence for clarity, since once the sender starts processing the multiple ACK packets fast retransmit kicks in anyway and things look pretty normal in the recovery path.

Configuration is bog standard. Switch reset to factory defaults, rate limit applied to one of the test ports as follows

config
int 28 rate-limit all out kbps 100000
int 28 rate-limit all in kbps 100000
write mem

That's enough to cause the drops. Both hosts are GbE on short known-good cables, and undoing the rate-limit configuration (no int 28 ...) immediately restores performance and stops the retransmits.

EDIT2:

New dumps captured, there may be a slight amount of time skew between hosts still even with NTP active but I'm not sure how to factor that in.

Problem starts with sequence number 10515814 -- notice the 10515814 data packet is received before three ACKs for 10515814 have even left the receiving host. Unless I'm misunderstanding something, this means fast retransmit hasn't even had a chance to kick in, showing reordering versus straight drop?

Switch config on the receiving host's port was rate limited to 100Mbps, the transmitting hosts's port was unlimited. iperf3 was run with a 40Mbps transmission limit (chosen empirically because it's easier to find the reordering in the dumps under these conditions, same thing happens with 10Mbps and 4Mpbs respectively).

RECEIVER:

21:24:07.302170 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10494094:10501334, ack 1, win 229, options [nop,nop,TS val 1242980393 ecr 1549151364], length 7240
21:24:07.302182 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10501334:10502782, ack 1, win 229, options [nop,nop,TS val 1242980393 ecr 1549151364], length 1448
21:24:07.302188 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10501334, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980393], length 0
21:24:07.302343 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10502782:10507126, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 4344
21:24:07.302358 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10507126, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980393], length 0
21:24:07.302375 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10507126:10510022, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 2896
21:24:07.302388 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10510022, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.302553 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10510022:10515814, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 5792
21:24:07.302568 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10517262:10518710, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.302575 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13478, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10518710}], length 0
21:24:07.302727 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10518710:10520158, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.302734 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13504, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10520158}], length 0
21:24:07.302876 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10515814:10517262, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.302883 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10520158, win 13496, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.303094 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10520158:10523054, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 2896
21:24:07.303101 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10523054, win 13504, options [nop,nop,TS val 1549151366 ecr 1242980394], length 0
21:24:07.303256 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10523054:10524502, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.344514 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10524502, win 13522, options [nop,nop,TS val 1549151407 ecr 1242980395], length 0
21:24:07.344674 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10527398:10530294, ack 1, win 229, options [nop,nop,TS val 1242980436 ecr 1549151407], length 2896
21:24:07.344679 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10524502, win 13522, options [nop,nop,TS val 1549151407 ecr 1242980395,nop,nop,sack 1 {10527398:10530294}], length 0
21:24:07.344870 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10524502:10527398, ack 1, win 229, options [nop,nop,TS val 1242980436 ecr 1549151407], length 2896

TRANSMITTER:

21:24:07.313522 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10510022:10511470, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313525 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10511470:10512918, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313527 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10512918:10514366, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313533 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10514366:10515814, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313537 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10510022, win 13522, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.313545 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10515814:10517262, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313548 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10517262:10518710, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313728 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13478, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10518710}], length 0
21:24:07.313741 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10518710:10520158, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313744 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10520158:10521606, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313747 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10521606:10523054, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313753 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10523054:10524502, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313756 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10524502:10525950, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.313885 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10515814, win 13504, options [nop,nop,TS val 1549151365 ecr 1242980394,nop,nop,sack 1 {10517262:10520158}], length 0
21:24:07.313896 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10515814:10517262, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.314085 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10520158, win 13496, options [nop,nop,TS val 1549151365 ecr 1242980394], length 0
21:24:07.314096 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10520158:10521606, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.314099 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10521606:10523054, ack 1, win 229, options [nop,nop,TS val 1242980394 ecr 1549151365], length 1448
21:24:07.314263 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10523054, win 13504, options [nop,nop,TS val 1549151366 ecr 1242980394], length 0
21:24:07.314273 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10523054:10524502, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.314276 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10524502:10525950, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.314281 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10525950:10527398, ack 1, win 229, options [nop,nop,TS val 1242980395 ecr 1549151366], length 1448
21:24:07.355648 IP 192.168.1.2.5201 > 192.168.1.1.59538: Flags [.], ack 10524502, win 13522, options [nop,nop,TS val 1549151407 ecr 1242980395], length 0
21:24:07.355661 IP 192.168.1.1.59538 > 192.168.1.2.5201: Flags [.], seq 10527398:10528846, ack 1, win 229, options [nop,nop,TS val 1242980436 ecr 1549151407], length 1448

EDIT3:

Switch configuration. As stated, quite basic:

Running configuration:

; J8693A Configuration Editor; Created on release #K.15.16.0019m
; Ver #0f:01.80.00.00.34.00.00.00.00.74.fc.7f.ff.3f.ef:62
hostname "TEST"
module 1 type j86yya
module 2 type j86xxa
no timesync
no telnet-server
no web-management
interface 28
   rate-limit all in kbps 100000
   rate-limit all out kbps 100000
   exit
snmp-server community "public" unrestricted
vlan 1
   name "DEFAULT_VLAN"
   untagged 1-48
   ip address dhcp-bootp
   exit
no tftp client
no tftp server
no autorun
no dhcp config-file-update
no dhcp image-file-update
Improve this question
9
  • 1
    You really need to provide us with more information, e.g. the switch configuration. It is unlikely that the switch is reordering, and it is likely that the switch is policing packets so that the source must resend the dropped segments, and that looks out of order. You could capture the traffic both ways with something like Wireshark so that we can see exactly what is happening.
    – Ron Maupin ♦
    Commented Aug 3, 2019 at 13:29
  • That was my first thought, but why would the switch be dropping packets at all when the data flow is set by iperf far below the actual rate limiting value? E.g. as an extreme test I put the rate-limit up to 500Mbps and used iperf3 with -b 10M -- still seeing hundreds of fast retransmits over a 10 second period, and it will not stabilize. Needless to say transfer rates are all over the place when this happens, since we never properly leave slow start.
    – madscientist159
    Commented Aug 3, 2019 at 20:37
  • Switching is done almost completely in hardware using tiny FIFO queues. Unless you have something strange configured on the switch, the switch cannot reorder the frames. Please edit your question to include the full switch configuration. Also explain which interfaces are being used for this test, and if there are any other hosts connected to the switch, and what they are doing.
    – Ron Maupin ♦
    Commented Aug 4, 2019 at 1:17
  • These switches have QoS functionality, so they do have buffers on each port. I wonder if that is the root of the problem here, some interaction between the QoS hardware and the rate limiter. Note that we do not have QoS configured, it is set at the default configuration which I understand to be as off as QoS can be on these switches.
    – madscientist159
    Commented Aug 4, 2019 at 4:27
  • 1
    By the way, we have tried multiple times to introduce HP devices into our network, spending millions of dollars, but we always end up spending millions more to remove them in the end.
    – Ron Maupin ♦
    Commented Aug 4, 2019 at 4:43

1 Answer 1

Reset to default
2

Have you noticed that rate limiting (at least on HPE/Aruba) is done in bursts, not smoothly? That may interfere with TCP congestion control which might not be able to adapt to the bandwidth limitation. That in turn might cause apparent "reordering" as a combination of the switch's dropping and TCP's retransmissions.

Have you verified that the reordering phenomenon is really caused by the switch? You might need to capture packets before and after the switch for that or check the TCP timestamps.

Usually a much better way to limit/shape traffic is on a router. If you must do it on the switch, try down-linking the interface (auto-100) instead of rate limiting which results in a more "natural" limiting.

edit If you absolutely need to limit on the switch you should check the QoS features - these often provide more finely grained bandwidth control (not sure about the 3500 though).

Improve this answer
5
  • Yes, after it became apparent something odd was going on with the switch itself I did exactly that -- connected two separate systems directly to two ports on the switch, fired up tcpdump and iperf3 on both systems, and observed the reordering along with rapidly increasing fast retransmit counters. I've edited the question to clarify.
    – madscientist159
    Commented Aug 3, 2019 at 20:35
  • You need to watch the timestamps carefully - it's quite possible that you're seeing SEQ4-SEQ7-SEQ5-SEQ6 because 5 and 6 were dropped on the first attempt. This is what I meant with the rate limiter working in bursts - it doesn't drop single, well-distributed frames but a whole bunch at once. It's more like pulse-width modulation if you get the idea.
    – Zac67 ♦
    Commented Aug 3, 2019 at 20:51
  • Sure, I definitely understand the concept. Looks like I need to go back and grab the actual dumps again to verify. Regardless I still don't understand why sending a stream that is capped at a lower bandwidth (literally an order of magnitude lower) on the sending side would interact this badly with the limiter, even with it operating in a sort of "bang-bang" PWM mode.
    – madscientist159
    Commented Aug 3, 2019 at 21:00
  • From the TCP perspective, that (rather granular) limiting looks like a link that abruptly changes between vastly different bandwidths at a high frequency. TCP doesn't adapt too well to that, causing an abnormally high packet loss and subsequent retransmissions.
    – Zac67 ♦
    Commented Aug 3, 2019 at 21:05
  • Sure, and all mention of why HP thought this was a good way of providing rate limiting aside, I do understand that. Fundamentally that's why you rate limit at the source or with a shaper queue in-between, but the problem is the switch is even screwing up data flows that have been rate limited at the source to well below the rate limit configured on the switch. I've updated the question with an actual dump I just pulled from the test environment.
    – madscientist159
    Commented Aug 3, 2019 at 21:38

Not the answer you're looking for? Browse other questions tagged or ask your own question.