There's huge gap between two results for iperf

Question

I used iperf -c FOO_NODE -t 120 -i 10 to test the network with the other node FOO_NODE, sometimes I get the result like:

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  3.75 MBytes  3.15 Mbits/sec
[  3] 10.0-20.0 sec  3.75 MBytes  3.15 Mbits/sec
[  3] 20.0-30.0 sec  3.62 MBytes  3.04 Mbits/sec
[  3] 30.0-40.0 sec  3.88 MBytes  3.25 Mbits/sec
[  3] 40.0-50.0 sec  4.25 MBytes  3.57 Mbits/sec
[  3] 50.0-60.0 sec  3.75 MBytes  3.15 Mbits/sec
[  3] 60.0-70.0 sec  4.00 MBytes  3.36 Mbits/sec
[  3] 70.0-80.0 sec  3.62 MBytes  3.04 Mbits/sec
[  3] 80.0-90.0 sec  3.25 MBytes  2.73 Mbits/sec
[  3] 90.0-100.0 sec  3.75 MBytes  3.15 Mbits/sec
[  3] 100.0-110.0 sec  4.38 MBytes  3.67 Mbits/sec
[  3] 110.0-120.0 sec  3.50 MBytes  2.94 Mbits/sec
[  3]  0.0-120.5 sec  45.6 MBytes  3.18 Mbits/sec

All are in range of 3~6

But sometimes I got the result like this:

[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  54.0 MBytes  45.3 Mbits/sec
[  3] 10.0-20.0 sec  56.6 MBytes  47.5 Mbits/sec
[  3] 20.0-30.0 sec  58.2 MBytes  48.9 Mbits/sec
[  3] 30.0-40.0 sec  58.2 MBytes  48.9 Mbits/sec
[  3] 40.0-50.0 sec  59.4 MBytes  49.8 Mbits/sec
[  3] 50.0-60.0 sec  57.5 MBytes  48.2 Mbits/sec
[  3] 60.0-70.0 sec  58.0 MBytes  48.7 Mbits/sec
[  3] 70.0-80.0 sec  58.2 MBytes  48.9 Mbits/sec
[  3] 80.0-90.0 sec  58.8 MBytes  49.3 Mbits/sec
[  3] 90.0-100.0 sec  58.2 MBytes  48.9 Mbits/sec
[  3] 100.0-110.0 sec  57.6 MBytes  48.3 Mbits/sec
[  3] 110.0-120.0 sec  59.2 MBytes  49.7 Mbits/sec
[  3]  0.0-120.2 sec   694 MBytes  48.5 Mbits/sec

All are in range of 40~50.

And 3~6 happened more often.

I tested that from a node in ec2 to a node in other cloud server center. And I don't think there'll be much congestion cause both of them are less used.

This's weird. What's the problem?

Answer 1

Welcome to the downside of cloud computing! Jitter (latency variability) and aggressive oversubscription. Your node is competing for shared resources at many different points that can be greatly oversubscribed in public clouds.

I would tackle this problem by measuring/capturing the following:

• EC2 to EC2 (intra-cloud) iperf to elminate the traversal of the public Internet to the other cloud server. Be sure to choose instances within the same region and availability zone (AZ) -- instances need to be in the same account to ensure they are in the same AZ.
• Traceroutes during slow and fast iperf runs to ensure paths haven't changed significantly that could account for latency differences.
• If possible, shutdown the instance and bring it back up in the hope that it moves to a different compute host. Your issue could be the "noisy neighbor" problem.

Anything that subtracts from the full bandwidth available to you will reduce your iperf results. That includes both competing traffic on-box and off-box.

Answer 2

UDP has the downside of over subscription required for a peak avg throughput and there is a cost to dropping packets. Also, constant oversubscription doesn't allow intermediate buffers to drain - adding to end/end latency.

So one should really test both TCP and UDP and when using UDP have the iperf client offer a load of 99% of end/end link capacity.

Answer 3

generalnetworkerror had a great answer. I just want to add one additional though: iperf by default uses TCP, which is susceptible to changes in RTT and congestion. A more accurate measure of throughput would be to use the -u option (on both client and server) to measure the bandwidth using UDP. Make sure you also set a bandwidth target on the client as well. Excerpt from the MAN Page:

GENERAL OPTIONS
   -u, --udp
          use UDP rather than TCP

CLIENT SPECIFIC OPTIONS
   -b, --bandwidth n[KM]
          set target bandwidth to n bits/sec (default 1 Mbit/sec).  This setting requires UDP (-u).

Typically, if you are measuring maximum throughput with iperf, you almost always want to use UDP. The exception is if you are specifically looking for TCP throughput data.