0

I used iperf3 to measure the effect that the TCP window has on throughput. I am measuring the throughput between two virtual machines which are located on different physical hosts. So I wrote a Python script that called iperf3 with different window sizes (2 KB -1024 KB) and plotted the throughput. It turned out that none of the windows sizes achieved a throughput nearly as high as I measured without defining the window size.

This command measured a throughput of ~250 Mbits/s:

iperf3 -c 10.0.0.3 -t 120

This is the command I used for the following plot:

iperf3 -c 10.0.0.3 -t 120 -w <window size>

iperf3 measurements with different window sizes

Can anyone explain, why the throughput without specified window size is so much higher? I also can't find the default window size that is used if not specified. Does iperf3 use a flexible windows size if not specified?

EDIT I uploaded two output files of tcpdump here. It captured the iperf3 traffic with and without the -w flag. Starting before iperf3 started until after it ended. I had a look at it in WireShark but still cannot figure out what TCP window size was used for the connections.

Improve this question
2
  • The process is called TCP Sliding window.
    – HAL
    Commented May 17, 2016 at 14:40
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer.
    – Ron Maupin ♦
    Commented Dec 21, 2020 at 17:49

4 Answers 4

Reset to default
3

Receive window sizes (rwin) will increase and decrease automatically during a TCP session depending on how lossy the connection ends up being. It is therefore possible that when you omit the -w flag from iperf the window size is being dynamically scaled higher than your static values. Remember that maximum theoretical throughput is rwin/delay.

Based on that graph however it appears that your bandwidth caps out at 70Mbit or so and raising the window size isn't having any impact. As window size doesn't appear to be restricting the bandwidth, is there anything else which could be consuming it - are you performing these tests over Wifi? Alternatively, can you double check that when you set a window size greater than 256 with the -w flag, that it does actually get sent "on the wire" by verifying with tcpdump/Wireshark?

Improve this answer
6
  • I am running the tests on a virtual machine and send data to another virtual machine on another physical host. When running iperf3 without the -w flag the CPU usage is at 100%, so it seems that the CPU limits the throughput. But with the -w flag, the CPU usage is just around 30%. No idea what else could be limiting the throughput.
    – arne.z
    Commented May 16, 2016 at 19:00
  • Interesting. Can you confirm the latency between your two devices?
    – Mark Riddell
    Commented May 16, 2016 at 19:54
  • The latency is very stable at 17.7ms.
    – arne.z
    Commented May 16, 2016 at 20:18
  • So the max theoretical throughput would be ~ 240 Mbits/s, right? Which is about what I get when not using the -w flag.
    – arne.z
    Commented May 16, 2016 at 20:25
  • 1
    Your maximum throughput will be whatever is the maximum throughput on your network media - WiFi, Gig ethernet etc, DSL, Broadband. Latency and Window Size will determine how much of this throughput you are able to use. The higher the latency, the larger the window size required to "fill the pipe" and utilize all your available bandwidth.
    – Mark Riddell
    Commented May 16, 2016 at 20:30
3

That's not a shock. These day we automatically tune the amount of RAM assigned to TCP buffers (aka the maximum TCP window size), and by specifying the window size you are turning off the automated tuning.

The flat-topping at around where the bandwidth-delay product would be about 64KB suggests to me that a suitable Window Scale Option isn't being negotiated, therefore the effective TCP buffer is held at 64KB.

As an aside, you have a serious throughput issue if you are only getting 250Mbps between VMs on the same host under the most favourable circumstances. You should make sure you are using paravirtualised virtio network interfaces, and that the VMs are on the same bridge. To reason about network performance you really need to get the best case performance up to the point where hardware is the limitation.

Improve this answer
3
  • Thank you for your answer. I updated my answer with some tcpdump output that includes the window sizes. I am measuring the throughput between two VMs that are located on different physical hosts.
    – arne.z
    Commented May 17, 2016 at 18:10
  • 2
    The Window Scale Option is negotiated in the packets at the start of the connection (they have the Syn flag set). That's not the packets in your tcpdump. Also note that you can pull that dumped file into Wireshark, and it has some good graphs of window behaviour.
    – vk5tu
    Commented May 17, 2016 at 23:47
  • Okay, I uploaded the dumped files and added the link to my question. One file captured the traffic with -w option and one file captured the traffic without. I already examined the files in WireShark by myself but still can't figure out what TCP window size was actually used.
    – arne.z
    Commented May 19, 2016 at 17:38
3

As Marco mentioned,

Receive window sizes (rwin) will increase and decrease automatically during a TCP session depending on how lossy the connection ends up being. It is therefore possible that when you omit the -w flag from iperf the window size is being dynamically scaled higher than your static values. Remember that maximum theoretical throughput is rwin/delay.

When you consider this and the take TCP Window Scaling into consideration, your 1,024 KB window is considerably lower than the maximum size of 1,073,725,440 bytes (about 1 GB). That's not to say that iperf would reach that size, but it is highly likely it would go well above 1,024 KB when the -w flag isn't specified.

Your best bet is to redo the test and run a packet capture at the same time. That way you'll be able to see the window size's value.

Improve this answer
3
  • I ran tcpdump and added the output to my question. Without defining the window size in iperf3, tcpdump captures a window size of 229, when defining a window size of 512K or 1024K, the tcpdump captures a window size of 7300. :-/
    – arne.z
    Commented May 17, 2016 at 18:08
  • Can you please upload a complete dump Cloud Shark or provide a link to where it can be downloaded? The dump should start before the iperf test begins and end after the iperf test has finished.
    – OzNetNerd
    Commented May 19, 2016 at 2:16
  • I uploaded two tcpdump files. Hope that also works. One captured the traffic of iperf with the TCP window size set to 1024, the other file captured the traffic without a specified TCP window size.
    – arne.z
    Commented May 19, 2016 at 17:40
0

Look at the RTT and CWND to get information about actual runtime window size. Also, use -i 0.1 to get sampled values as these are dynamic. If these values are larger when you're not setting -w, you know the answer, i.e. your -w is too small.

With iperf 2.0.9, -e on the client will give the RTT and CWND (Linux only). Other ways to get these is using tcpdump and tcptrace.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.