12

In a WLAN iperf TCP throughput test, multiple parallel streams will give me higher throughput than 1 stream. I tried increasing the TCP window size, but I still cannot achieve the max throughput with just 1 stream. Is there something else in the TCP layer that is preventing the full link capacity from being used?

  • How much difference do you observe? Ideally, if one TCP stream provides a throughput of T, then two should individually provide a throughput of T/2 each. – Manoj Pandey Sep 16 '13 at 19:55
  • Note that full link capacity regardless of number of streams will never be reached. IPv4 with minimal [IP+TCP] headers will yield ~95% channel efficiency. See the excellent Protocol Overhead posting at sd.wareonearth.com/~phil/net/overhead. – generalnetworkerror Sep 17 '13 at 8:34
  • @ManojPandey, I'm not sure he is seeing an ideal case... especially since he's using wifi... I suspect he has some packet loss... – Mike Pennington Sep 17 '13 at 11:03
  • TCP sucks over Wi-Fi, deal with it. If you have to use it and you are seeing layer 3 packet losses, i would suggest to increase the layer 2 maximum retry count, since TCP is not designed to handle lossy links without destroying performance. – BatchyX Sep 18 '13 at 7:11
14

In a WLAN iperf TCP throughput test, multiple parallel streams will give me higher throughput than 1 stream. I tried increasing the TCP window size, but I still cannot achieve the max throughput with just 1 stream. Is there something else in the TCP layer that is preventing the full link capacity from being used?

In my experience, if you see significantly different results between 1 TCP stream and multiple TCP streams the issue is normally packet loss; so the "something else" in the TCP layer is retransmission (due to lower-layer packet loss).

An example I cooked up to illustrate how packet loss affects single-stream throughput...

                   [Wifi||LAN-0.0%-Loss||LAN-2.0%-Loss]
+--------------+                                        +--------------+
|              |                                        |              |
| Thinkpad-T61 |----------------------------------------| Linux Server |
|              |                                        | Tsunami      |
+--------------+                                        +--------------+
iperf client             ------------------>            iperf server
                             Pushes data

This is a table that summarizes the test results of a 60 second iperf test between a client and server... you might see a little variation in iperf results from RTT jitter (i.e higher RTT standard deviation); however, the most significant difference came when I simulated 2% loss leaving the client wired NIC. 172.16.1.56 and 172.16.1.80 are the same laptop (running Ubuntu). The server is 172.16.1.5, running Debian. I used netem on the client wired NIC to simulate packet loss...

Client IP       Transport    Loss     avg RTT    RTT StdDev    TCP Streams   Tput
-----------     ----------   ----     -------    ----------    -----------   ----------
172.16.1.56     802.11g      0.0%      0.016s          42.0              1     19.5Mbps
172.16.1.56     802.11g      0.0%      0.016s          42.0              5     20.5Mbps
172.16.1.80     1000BaseT    0.0%     0.0002s           0.0              1    937  Mbps
172.16.1.80     1000BaseT    0.0%     0.0002s           0.0              5    937  Mbps
172.16.1.80     1000BaseT    2.0%     0.0002s           0.0              1    730  Mbps <---
172.16.1.80     1000BaseT    2.0%     0.0002s           0.0              5    937  Mbps

EDIT for comment responses:

Can you explain what is happening in the last scenario (1000BaseT, 5 streams, 2.0% loss)? Even though there is packet loss, the total throughput is still saturated at 937 Mbits/sec.

Most TCP implementations decrease their congestion window when packet loss is detected. Since we're using netem to force 2% packet loss from the client to the server, some of the client's data gets dropped. The net effect of netem in this example is a single-stream average transmit rate of 730Mbps. Adding multiple streams means that the individual TCP streams can work together to saturate the link.

My goal is to achieve the highest TCP throughput possible over WiFi. As I understand it, I should increase the number of streams in order to counter the decrease in throughput caused by packet loss. Is this correct?

Yes

In addition, at what point will too many streams start to negatively impact throughput? Would this be caused by limited memory and/or processing power?

I can't really answer that without more experiments, but for 1GE links, I have never had a problem saturating a link with 5 parallel streams. To give you an idea of how scalable TCP is, linux servers can handle over 1500 concurrent TCP sockets under the right circumstances. This is another SO discussion that is relevant to scaling concurrent TCP sockets, but in my opinion anything above 20 parallel sockets would be overkill if you're merely trying to saturate a link.

I must have a misunderstanding that iperf uses the -w window size as a maximum as it sounds like you're saying it grew beyond that 21K initial value

I didn't use iperf -w, so I think there is a misunderstanding. Since you have so many questions about the wifi case, I'm including a wireshark graph of TCP throughput for the wifi single TCP stream case.

Wifi TCP single-stream throughput


Test Data

I am also including raw test data in case you'd like to see how I measured these things...

802.11g, 1 TCP Stream

mpenning@mpenning-ThinkPad-T61:~$ mtr --no-dns --report \
  --report-cycles 60 172.16.1.5
HOST: mpenning-ThinkPad-T61       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.16.1.5                 0.0%    60    0.8  16.0   0.7 189.4  42.0
mpenning@mpenning-ThinkPad-T61:~$

[mpenning@tsunami]$ iperf -s -p 9000 -B 172.16.1.5

mpenning@mpenning-ThinkPad-T61:~$ iperf -c 172.16.1.5 -p 9000 -t 60 -P 1
------------------------------------------------------------
Client connecting to 172.16.1.5, TCP port 9000
TCP window size: 21.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.56 port 40376 connected with 172.16.1.5 port 9000
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.1 sec   139 MBytes  19.5 Mbits/sec
mpenning@mpenning-ThinkPad-T61:~$

802.11g, 5 TCP Streams

[mpenning@tsunami]$ iperf -s -p 9001 -B 172.16.1.5

mpenning@mpenning-ThinkPad-T61:~$ iperf -c 172.16.1.5 -p 9001 -t 60 -P 5
------------------------------------------------------------
Client connecting to 172.16.1.5, TCP port 9001
TCP window size: 21.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.56 port 37162 connected with 172.16.1.5 port 9001
[  5] local 172.16.1.56 port 37165 connected with 172.16.1.5 port 9001
[  7] local 172.16.1.56 port 37167 connected with 172.16.1.5 port 9001
[  4] local 172.16.1.56 port 37163 connected with 172.16.1.5 port 9001
[  6] local 172.16.1.56 port 37166 connected with 172.16.1.5 port 9001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  28.0 MBytes  3.91 Mbits/sec
[  5]  0.0-60.1 sec  28.8 MBytes  4.01 Mbits/sec
[  4]  0.0-60.3 sec  28.1 MBytes  3.91 Mbits/sec
[  6]  0.0-60.4 sec  34.0 MBytes  4.72 Mbits/sec
[  7]  0.0-61.0 sec  30.5 MBytes  4.20 Mbits/sec
[SUM]  0.0-61.0 sec   149 MBytes  20.5 Mbits/sec
mpenning@mpenning-ThinkPad-T61:~$

1000BaseT, 1 Stream, 0.0% loss

mpenning@mpenning-ThinkPad-T61:~$ mtr --no-dns --report \
>   --report-cycles 60 172.16.1.5
HOST: mpenning-ThinkPad-T61       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.16.1.5                 0.0%    60    0.2   0.2   0.2   0.2   0.0
mpenning@mpenning-ThinkPad-T61:~$

[mpenning@tsunami]$ iperf -s -p 9002 -B 172.16.1.5

mpenning@mpenning-ThinkPad-T61:~$ iperf -c 172.16.1.5 -p 9002 -t 60 -P 1
------------------------------------------------------------
Client connecting to 172.16.1.5, TCP port 9002
TCP window size: 21.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.80 port 49878 connected with 172.16.1.5 port 9002
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  6.54 GBytes   937 Mbits/sec
mpenning@mpenning-ThinkPad-T61:~$

1000BaseT, 5 Streams, 0.0% loss

[mpenning@tsunami]$ iperf -s -p 9003 -B 172.16.1.5

mpenning@mpenning-ThinkPad-T61:~$ iperf -c 172.16.1.5 -p 9003 -t 60 -P 5
------------------------------------------------------------
Client connecting to 172.16.1.5, TCP port 9003
TCP window size: 21.0 KByte (default)
------------------------------------------------------------
[  7] local 172.16.1.80 port 47047 connected with 172.16.1.5 port 9003
[  3] local 172.16.1.80 port 47043 connected with 172.16.1.5 port 9003
[  4] local 172.16.1.80 port 47044 connected with 172.16.1.5 port 9003
[  5] local 172.16.1.80 port 47045 connected with 172.16.1.5 port 9003
[  6] local 172.16.1.80 port 47046 connected with 172.16.1.5 port 9003
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-60.0 sec  1.28 GBytes   184 Mbits/sec
[  5]  0.0-60.0 sec  1.28 GBytes   184 Mbits/sec
[  3]  0.0-60.0 sec  1.28 GBytes   183 Mbits/sec
[  6]  0.0-60.0 sec  1.35 GBytes   193 Mbits/sec
[  7]  0.0-60.0 sec  1.35 GBytes   193 Mbits/sec
[SUM]  0.0-60.0 sec  6.55 GBytes   937 Mbits/sec
mpenning@mpenning-ThinkPad-T61:~$

1000BaseT, 1 Streams, 2.0% loss

mpenning@mpenning-ThinkPad-T61:~$ sudo tc qdisc add dev eth0 root netem corrupt 2.0%

mpenning@mpenning-ThinkPad-T61:~$ mtr --no-dns --report   --report-cycles 60 172.16.1.5
HOST: mpenning-ThinkPad-T61       Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 172.16.1.5                 1.7%    60    0.2   0.2   0.2   0.2   0.0
mpenning@mpenning-ThinkPad-T61:~$

[mpenning@tsunami]$ iperf -s -p 9004 -B 172.16.1.5

mpenning@mpenning-ThinkPad-T61:~$ iperf -c 172.16.1.5 -p 9004 -t 60 -P 1
------------------------------------------------------------
Client connecting to 172.16.1.5, TCP port 9004
TCP window size: 42.0 KByte (default)
------------------------------------------------------------
[  3] local 172.16.1.80 port 48910 connected with 172.16.1.5 port 9004
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-64.1 sec  5.45 GBytes   730 Mbits/sec
mpenning@mpenning-ThinkPad-T61:~$

1000BaseT, 5 Streams, 2.0% loss

[mpenning@tsunami]$ iperf -s -p 9005 -B 172.16.1.5

mpenning@mpenning-ThinkPad-T61:~$ iperf -c 172.16.1.5 -p 9005 -t 60 -P 5
------------------------------------------------------------
Client connecting to 172.16.1.5, TCP port 9005
TCP window size: 21.0 KByte (default)
------------------------------------------------------------
[  7] local 172.16.1.80 port 50616 connected with 172.16.1.5 port 9005
[  3] local 172.16.1.80 port 50613 connected with 172.16.1.5 port 9005
[  5] local 172.16.1.80 port 50614 connected with 172.16.1.5 port 9005
[  4] local 172.16.1.80 port 50612 connected with 172.16.1.5 port 9005
[  6] local 172.16.1.80 port 50615 connected with 172.16.1.5 port 9005
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-60.0 sec  1.74 GBytes   250 Mbits/sec
[  7]  0.0-60.0 sec   711 MBytes  99.3 Mbits/sec
[  4]  0.0-60.0 sec  1.28 GBytes   183 Mbits/sec
[  6]  0.0-60.0 sec  1.59 GBytes   228 Mbits/sec
[  5]  0.0-60.0 sec  1.24 GBytes   177 Mbits/sec
[SUM]  0.0-60.0 sec  6.55 GBytes   937 Mbits/sec
mpenning@mpenning-ThinkPad-T61:~$

Remove packet loss simulation

mpenning@mpenning-ThinkPad-T61:~$ sudo tc qdisc del dev eth0 root
  • Can you explain what is happening in the last scenario (1000BaseT, 5 streams, 2.0% loss)? Even though there is packet loss, the total throughput is still saturated at 937 Mbits/sec. My goal is to achieve the highest TCP throughput possible over WiFi. As I understand it, I should increase the number of streams in order to counter the decrease in throughput caused by packet loss. Is this correct? In addition, at what point will too many streams start to negatively impact throughput? Would this be caused by limited memory and/or processing power? – elin05 Sep 17 '13 at 18:59
  • @elin05: Using multiple streams spreads the packet loss on several streams, so when a packet loss occurs, only one stream will reduce the size of its TCP window, leaving the other streams unaffected. – BatchyX Sep 18 '13 at 7:17
  • Doesn't the 802.11g (54Mbps) BDP call for a window size of 54KB with a 8ms (16ms RTT / 2) delay to keep the pipe full with in-flight packets? What's the window size on the server-side? – generalnetworkerror Sep 18 '13 at 8:17
  • @generalnetworkerror, TCP windows are not static... they change based on the needs of TCP... during that capture, the maximum window size Tsunami advertised was 1177600 bytes; Tsunami's average window was 1045083 bytes and the average RTT over that 60 second test was 32.2ms. – Mike Pennington Sep 18 '13 at 11:44
  • @MikePennington: I must have a misunderstanding that iperf uses the -w window size as a maximum as it sounds like you're saying it grew beyond that 21K initial value. – generalnetworkerror Sep 24 '13 at 7:46
2

Here's the calculation for the maximum throughput of a single tcp stream.

(TCP Window [bytes]/RTT[seconds]) * 8 [bits/byte] = Max single tcp throughput in (bps)

So you have a bottleneck and latency plays a large role.

0

It's probably due to multiple processes vs one process. with iperf 2.0.9 one can test this via -P 2 on the client. This will fork two threads instead of one. Most modern CPUs have multiple cores so using multiple thread will be able to leverage them.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.