14

Background

We rented a few layer 2 leased lines, i.e. you have one big pipe at the datacenter and the remote sites have smaller pipes. Inside the layer 2 network you can do whatever you like. Probably they use 802.1ad to give each customer their separate network inside their network. AFAICS most sites are connected via plain VDSL.

We decided to put a router at each site, and give each site its own VLAN. The firewall at the DC thus has as many VLANs defined as there are sites. Each site thus uses its on address range in its own VLAN.

Network diagram:

network diagram

Problem

Now, we are faced with throughput problems:

  • Running an FTP transfer from site to DC works fine at about 10Mb/s which is line speed.
  • Running an FTP transfer from DC to site does not work fine at 6Mb/s or less.

It does not matter which side initiates the transfer. The only consistent thing is that one direction is not working well. Too bad it is the direction toward the site because that would be the bandwidth we need most as we'd like to use terminal server clients.

About 10 seconds into the transfer, the throughput drops. We see DUP ACKs when sniffing. Which maybe leads me to rate limiting at the provider's end?? (Currently, they don't have a clue, and I like to make sure we're not at fault before escalating)

NOTE

The remote sites are limited to 10Mb somehow. Setting the switch-to-Metro-port to 10Mb doesn't help either. In fact it's the worst then (max. 30 KB/s). Setting to 100Mb works fine but already starts to produce the outlined problem. Same for 1G.

Captures of the problem can be downloaded here:

* http://178.63.11.6/dc-to-remote_dc-side.pcapng
* http://178.63.11.6/dc-to-remote_remote-side.pcapng

Diagnostics

In the image you see the Wireshark IO Graph with some error details:

  • on the left: FTP transfer from DC to site
  • on the right: FTP transfer from site to DC

duplicate acks

In case the other side initiates the transfer (i.e. put from dc, instead of get from remote), the problem remains unchanged.

Please indulge me with what you think could be the problem here.

UPDATE #1 (integrated above)

UPDATE #2 (UPDATED)

This must be a congestion control thing.

Note that from DC to remote we have 10G->1G->100M->10M->1G links. <-- not working

In the other direction we thus have the inverse: 1G->10M->100M->1G->10G. <-- just fine

The first "1G->10M" is the "invisible" 10M at the remote site, where everything including the uplink port speed is set at 1G, even though there's only 10M behind it (being sold).

However the 100Mbps at the DC are real 100Mbps, the interface is configured at 100Mbps on the physical layer.

Now using iperf:

  • TCP tests work fine only in one direction (client=DC, server=remote)
./iperf -c 192.168.x -i2 -t 60 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.x, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.x port 38195 connected with 192.168.x port 5001
[  3]  0.0- 2.0 sec  1.44 MBytes  6.03 Mbits/sec
[  3]  2.0- 4.0 sec  2.23 MBytes  9.37 Mbits/sec
[  3]  4.0- 6.0 sec  2.28 MBytes  9.57 Mbits/sec
[  3]  6.0- 8.0 sec  1.88 MBytes  7.90 Mbits/sec
[  3]  8.0-10.0 sec  1.00 MBytes  4.19 Mbits/sec
[  3] 10.0-12.0 sec  1.30 MBytes  5.47 Mbits/sec
[  3] 12.0-14.0 sec    688 KBytes  2.82 Mbits/sec
[  3] 14.0-16.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 16.0-18.0 sec  1.03 MBytes  4.33 Mbits/sec
[  3] 18.0-20.0 sec  1.01 MBytes  4.23 Mbits/sec
[  3] 20.0-22.0 sec  1.03 MBytes  4.33 Mbits/sec
[  3] 22.0-24.0 sec  1.18 MBytes  4.95 Mbits/sec
[  3] 24.0-26.0 sec    904 KBytes  3.70 Mbits/sec
[  3] 26.0-28.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 28.0-30.0 sec    936 KBytes  3.83 Mbits/sec
[  3] 30.0-32.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 32.0-34.0 sec    960 KBytes  3.93 Mbits/sec
[  3] 34.0-36.0 sec    752 KBytes  3.08 Mbits/sec
[  3] 36.0-38.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 38.0-40.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 40.0-42.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 42.0-44.0 sec  1.27 MBytes  5.34 Mbits/sec
[  3] 44.0-46.0 sec  1.16 MBytes  4.85 Mbits/sec
[  3] 46.0-48.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 48.0-50.0 sec    960 KBytes  3.93 Mbits/sec
[  3] 50.0-52.0 sec  1.28 MBytes  5.37 Mbits/sec
[  3] 52.0-54.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 54.0-56.0 sec    992 KBytes  4.06 Mbits/sec
[  3] 56.0-58.0 sec  1.00 MBytes  4.19 Mbits/sec
[  3] 58.0-60.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3]  0.0-60.2 sec  33.9 MBytes  4.73 Mbits/sec
[  5] local 10.x port 5001 connected with 192.168.x port 10965
[  5]  0.0- 2.0 sec  1.85 MBytes  7.75 Mbits/sec
[  5]  2.0- 4.0 sec  1.90 MBytes  7.98 Mbits/sec
[  5]  4.0- 6.0 sec  1.89 MBytes  7.93 Mbits/sec
[  5]  6.0- 8.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5]  8.0-10.0 sec  1.91 MBytes  8.02 Mbits/sec
[  5] 10.0-12.0 sec  1.83 MBytes  7.69 Mbits/sec
[  5] 12.0-14.0 sec  1.86 MBytes  7.78 Mbits/sec
[  5] 14.0-16.0 sec  1.79 MBytes  7.52 Mbits/sec
[  5] 16.0-18.0 sec  1.79 MBytes  7.52 Mbits/sec
[  5] 18.0-20.0 sec  1.89 MBytes  7.91 Mbits/sec
[  5] 20.0-22.0 sec  1.91 MBytes  8.00 Mbits/sec
[  5] 22.0-24.0 sec  1.88 MBytes  7.91 Mbits/sec
[  5] 24.0-26.0 sec  1.95 MBytes  8.16 Mbits/sec
[  5] 26.0-28.0 sec  1.90 MBytes  7.99 Mbits/sec
[  5] 28.0-30.0 sec  1.87 MBytes  7.84 Mbits/sec
[  5] 30.0-32.0 sec  1.85 MBytes  7.77 Mbits/sec
[  5] 32.0-34.0 sec  1.55 MBytes  6.49 Mbits/sec
[  5] 34.0-36.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5] 36.0-38.0 sec  1.90 MBytes  7.99 Mbits/sec
[  5] 38.0-40.0 sec  1.84 MBytes  7.73 Mbits/sec
[  5] 40.0-42.0 sec  1.66 MBytes  6.95 Mbits/sec
[  5] 42.0-44.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5] 44.0-46.0 sec  1.91 MBytes  7.99 Mbits/sec
[  5] 46.0-48.0 sec  1.90 MBytes  7.98 Mbits/sec
[  5] 48.0-50.0 sec  1.84 MBytes  7.70 Mbits/sec
[  5] 50.0-52.0 sec  1.93 MBytes  8.09 Mbits/sec
[  5] 52.0-54.0 sec  1.80 MBytes  7.54 Mbits/sec
[  5] 54.0-56.0 sec  1.83 MBytes  7.67 Mbits/sec
[  5] 56.0-58.0 sec  1.88 MBytes  7.86 Mbits/sec
[  5] 58.0-60.0 sec  1.85 MBytes  7.78 Mbits/sec
[  5]  0.0-60.3 sec  56.0 MBytes  7.79 Mbits/sec
  • To get to the bottom of it, here are UDP tests from two hosts in the same VLAN yet using the Metro Connection, 200=remote, 201=DC

We see the packet loss increasing with incrementing bandwidth (when approaching 10Mbps we have 0.93%, starts to be critical... and would also explain why TCP has problem performing)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61759 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  1.0- 2.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  2.0- 3.0 sec   129 KBytes  1.06 Mbits/sec
[  4]  3.0- 4.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  4.0- 5.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  5.0- 6.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  6.0- 7.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  7.0- 8.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  8.0- 9.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  9.0-10.0 sec   129 KBytes  1.06 Mbits/sec
[  4] 10.0-11.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 11.0-12.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 12.0-13.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 13.0-14.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 14.0-15.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 15.0-16.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 16.0-17.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 17.0-18.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 18.0-19.0 sec   131 KBytes  1.07 Mbits/sec
[  4] 19.0-20.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  0.0-20.0 sec  2.50 MBytes  1.05 Mbits/sec
[  4] Sent 1785 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  2.50 MBytes  1.05 Mbits/sec   0.257 ms    0/ 1785 (0%)
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50749
[  3]  0.0- 1.0 sec   128 KBytes  1.05 Mbits/sec   0.285 ms    0/   89 (0%)
[  3]  1.0- 2.0 sec   128 KBytes  1.05 Mbits/sec   0.313 ms    0/   89 (0%)
[  3]  2.0- 3.0 sec   128 KBytes  1.05 Mbits/sec   0.278 ms    0/   89 (0%)
[  3]  3.0- 4.0 sec   128 KBytes  1.05 Mbits/sec   0.241 ms    0/   89 (0%)
[  3]  4.0- 5.0 sec   128 KBytes  1.05 Mbits/sec   0.266 ms    0/   89 (0%)
[  3]  5.0- 6.0 sec   128 KBytes  1.05 Mbits/sec   0.293 ms    0/   89 (0%)
[  3]  6.0- 7.0 sec   128 KBytes  1.05 Mbits/sec   0.314 ms    0/   89 (0%)
[  3]  7.0- 8.0 sec   128 KBytes  1.05 Mbits/sec   0.280 ms    0/   89 (0%)
[  3]  8.0- 9.0 sec   128 KBytes  1.05 Mbits/sec   0.242 ms    0/   89 (0%)
[  3]  9.0-10.0 sec   129 KBytes  1.06 Mbits/sec   0.250 ms    0/   90 (0%)
[  3] 10.0-11.0 sec   128 KBytes  1.05 Mbits/sec   0.275 ms    0/   89 (0%)
[  3] 11.0-12.0 sec   128 KBytes  1.05 Mbits/sec   0.299 ms    0/   89 (0%)
[  3] 12.0-13.0 sec   128 KBytes  1.05 Mbits/sec   0.327 ms    0/   89 (0%)
[  3] 13.0-14.0 sec   128 KBytes  1.05 Mbits/sec   0.290 ms    0/   89 (0%)
[  3] 14.0-15.0 sec   128 KBytes  1.05 Mbits/sec   0.251 ms    0/   89 (0%)
[  3] 15.0-16.0 sec   128 KBytes  1.05 Mbits/sec   0.275 ms    0/   89 (0%)
[  3] 16.0-17.0 sec   128 KBytes  1.05 Mbits/sec   0.303 ms    0/   89 (0%)
[  3] 17.0-18.0 sec   128 KBytes  1.05 Mbits/sec   0.333 ms    0/   89 (0%)
[  3] 18.0-19.0 sec   128 KBytes  1.05 Mbits/sec   0.294 ms    0/   89 (0%)
[  3] 19.0-20.0 sec   131 KBytes  1.07 Mbits/sec   0.281 ms    0/   91 (0%)
[  3]  0.0-20.0 sec  2.50 MBytes  1.05 Mbits/sec   0.305 ms    0/ 1785 (0%)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 5m
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61760 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  1.0- 2.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  2.0- 3.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  3.0- 4.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  4.0- 5.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  5.0- 6.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  6.0- 7.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  7.0- 8.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  8.0- 9.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  9.0-10.0 sec   619 KBytes  5.07 Mbits/sec
[  4] 10.0-11.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 11.0-12.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 12.0-13.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 13.0-14.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 14.0-15.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 15.0-16.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 16.0-17.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 17.0-18.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 18.0-19.0 sec   619 KBytes  5.07 Mbits/sec
[  4] 19.0-20.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  0.0-20.0 sec  11.9 MBytes  5.00 Mbits/sec
[  4] Sent 8504 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  11.9 MBytes  4.99 Mbits/sec   0.000 ms   12/ 8503 (0.14%)
[  4]  0.0-20.0 sec  1 datagrams received out-of-order
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50750
[  3]  0.0- 1.0 sec   606 KBytes  4.96 Mbits/sec   2.238 ms    1/  423 (0.24%)
[  3]  1.0- 2.0 sec   610 KBytes  5.00 Mbits/sec   2.739 ms    0/  425 (0%)
[  3]  2.0- 3.0 sec   609 KBytes  4.99 Mbits/sec   3.089 ms    1/  425 (0.24%)
[  3]  3.0- 4.0 sec   609 KBytes  4.99 Mbits/sec   3.605 ms    0/  424 (0%)
[  3]  4.0- 5.0 sec   607 KBytes  4.97 Mbits/sec   1.954 ms    0/  423 (0%)
[  3]  5.0- 6.0 sec   612 KBytes  5.01 Mbits/sec   2.666 ms    0/  426 (0%)
[  3]  6.0- 7.0 sec   607 KBytes  4.97 Mbits/sec   2.602 ms    0/  423 (0%)
[  3]  7.0- 8.0 sec   612 KBytes  5.01 Mbits/sec   2.960 ms    0/  426 (0%)
[  3]  8.0- 9.0 sec   609 KBytes  4.99 Mbits/sec   2.512 ms    0/  424 (0%)
[  3]  9.0-10.0 sec   619 KBytes  5.07 Mbits/sec   2.133 ms    0/  431 (0%)
[  3] 10.0-11.0 sec   609 KBytes  4.99 Mbits/sec   3.605 ms    1/  425 (0.24%)
[  3] 11.0-12.0 sec   609 KBytes  4.99 Mbits/sec   2.509 ms    0/  424 (0%)
[  3] 12.0-13.0 sec   610 KBytes  5.00 Mbits/sec   3.570 ms    0/  425 (0%)
[  3] 13.0-14.0 sec   609 KBytes  4.99 Mbits/sec   3.077 ms    1/  425 (0.24%)
[  3] 14.0-15.0 sec   609 KBytes  4.99 Mbits/sec   2.679 ms    0/  424 (0%)
[  3] 15.0-16.0 sec   609 KBytes  4.99 Mbits/sec   1.887 ms    0/  424 (0%)
[  3] 16.0-17.0 sec   610 KBytes  5.00 Mbits/sec   2.651 ms    0/  425 (0%)
[  3] 17.0-18.0 sec   609 KBytes  4.99 Mbits/sec   3.390 ms    0/  424 (0%)
[  3] 18.0-19.0 sec   617 KBytes  5.06 Mbits/sec   2.601 ms    0/  430 (0%)
[  3] 19.0-20.0 sec   612 KBytes  5.01 Mbits/sec   3.525 ms    0/  426 (0%)
[  3]  0.0-20.0 sec  11.9 MBytes  4.99 Mbits/sec   3.156 ms    3/ 8503 (0.035%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 9m
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61761 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  1.0- 2.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  2.0- 3.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  3.0- 4.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  4.0- 5.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  5.0- 6.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  6.0- 7.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  7.0- 8.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  8.0- 9.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  9.0-10.0 sec  1.09 MBytes  9.14 Mbits/sec
[  4] 10.0-11.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4] 11.0-12.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 12.0-13.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 13.0-14.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4] 14.0-15.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 15.0-16.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4] 16.0-17.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 17.0-18.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 18.0-19.0 sec  1.09 MBytes  9.14 Mbits/sec
[  4] 19.0-20.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  0.0-20.0 sec  21.5 MBytes  9.00 Mbits/sec
[  4] Sent 15315 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  21.3 MBytes  8.94 Mbits/sec   0.104 ms   96/15314 (0.63%)  !!!!!!!!!!
[  4]  0.0-20.0 sec  1 datagrams received out-of-order
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50751
[  3]  0.0- 1.0 sec  1.06 MBytes  8.89 Mbits/sec   2.405 ms    0/  756 (0%)
[  3]  1.0- 2.0 sec  1.07 MBytes  9.00 Mbits/sec   2.308 ms    0/  765 (0%)
[  3]  2.0- 3.0 sec  1.07 MBytes  9.00 Mbits/sec   2.305 ms    0/  765 (0%)
[  3]  3.0- 4.0 sec  1.07 MBytes  8.97 Mbits/sec   2.290 ms    1/  764 (0.13%)
[  3]  4.0- 5.0 sec  1.07 MBytes  8.98 Mbits/sec   2.271 ms    1/  765 (0.13%)
[  3]  5.0- 6.0 sec  1.07 MBytes  8.98 Mbits/sec   2.313 ms    0/  764 (0%)
[  3]  6.0- 7.0 sec  1.07 MBytes  9.00 Mbits/sec   2.191 ms    0/  765 (0%)
[  3]  7.0- 8.0 sec  1.07 MBytes  8.95 Mbits/sec   2.314 ms    3/  764 (0.39%)
[  3]  8.0- 9.0 sec  1.07 MBytes  8.98 Mbits/sec   2.232 ms    1/  765 (0.13%)
[  3]  9.0-10.0 sec  1.09 MBytes  9.13 Mbits/sec   2.257 ms    0/  776 (0%)
[  3] 10.0-11.0 sec  1.07 MBytes  8.98 Mbits/sec   2.365 ms    0/  764 (0%)
[  3] 11.0-12.0 sec  1.07 MBytes  8.98 Mbits/sec   2.301 ms    1/  765 (0.13%)
[  3] 12.0-13.0 sec  1.07 MBytes  8.98 Mbits/sec   2.277 ms    0/  764 (0%)
[  3] 13.0-14.0 sec  1.07 MBytes  9.00 Mbits/sec   2.323 ms    0/  765 (0%)
[  3] 14.0-15.0 sec  1.07 MBytes  9.00 Mbits/sec   2.176 ms    0/  765 (0%)
[  3] 15.0-16.0 sec  1.07 MBytes  8.96 Mbits/sec   2.273 ms    2/  764 (0.26%)
[  3] 16.0-17.0 sec  1.07 MBytes  8.98 Mbits/sec   2.313 ms    0/  764 (0%)
[  3] 17.0-18.0 sec  1.07 MBytes  8.98 Mbits/sec   2.247 ms    1/  765 (0.13%)
[  3] 18.0-19.0 sec  1.09 MBytes  9.11 Mbits/sec   2.276 ms    1/  776 (0.13%)
[  3] 19.0-20.0 sec  1.07 MBytes  8.97 Mbits/sec   2.394 ms    1/  764 (0.13%)
[  3]  0.0-20.0 sec  21.5 MBytes  8.99 Mbits/sec   2.659 ms   11/15314 (0.072%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 9850k
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61762 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  1.0- 2.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  2.0- 3.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  3.0- 4.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  4.0- 5.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  5.0- 6.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4]  6.0- 7.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  7.0- 8.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  8.0- 9.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  9.0-10.0 sec  1.19 MBytes  10.0 Mbits/sec
[  4] 10.0-11.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4] 11.0-12.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4] 12.0-13.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4] 13.0-14.0 sec  1.17 MBytes  9.85 Mbits/sec
[  4] 14.0-15.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4] 15.0-16.0 sec  1.17 MBytes  9.85 Mbits/sec
[  4] 16.0-17.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4] 17.0-18.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4] 18.0-19.0 sec  1.19 MBytes  10.0 Mbits/sec
[  4] 19.0-20.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  0.0-20.0 sec  23.5 MBytes  9.85 Mbits/sec
[  4] Sent 16765 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  23.3 MBytes  9.74 Mbits/sec   3.421 ms  156/16764 (0.93%) !!!!!!!!!!
[  4]  0.0-20.0 sec  1 datagrams received out-of-order
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50752
[  3]  0.0- 1.0 sec  1.16 MBytes  9.74 Mbits/sec   2.131 ms    0/  828 (0%)
[  3]  1.0- 2.0 sec  1.17 MBytes  9.84 Mbits/sec   2.140 ms    0/  837 (0%)
[  3]  2.0- 3.0 sec  1.17 MBytes  9.83 Mbits/sec   2.099 ms    1/  837 (0.12%)
[  3]  3.0- 4.0 sec  1.17 MBytes  9.84 Mbits/sec   2.113 ms    0/  837 (0%)
[  3]  4.0- 5.0 sec  1.17 MBytes  9.84 Mbits/sec   2.105 ms    0/  837 (0%)
[  3]  5.0- 6.0 sec  1.17 MBytes  9.83 Mbits/sec   2.058 ms    1/  837 (0.12%)
[  3]  6.0- 7.0 sec  1.17 MBytes  9.82 Mbits/sec   2.165 ms    1/  836 (0.12%)
[  3]  7.0- 8.0 sec  1.17 MBytes  9.84 Mbits/sec   2.156 ms    0/  837 (0%)
[  3]  8.0- 9.0 sec  1.17 MBytes  9.82 Mbits/sec   2.135 ms    2/  837 (0.24%)
[  3]  9.0-10.0 sec  1.19 MBytes  9.97 Mbits/sec   2.152 ms    2/  850 (0.24%)
[  3] 10.0-11.0 sec  1.17 MBytes  9.83 Mbits/sec   2.153 ms    1/  837 (0.12%)
[  3] 11.0-12.0 sec  1.17 MBytes  9.84 Mbits/sec   2.127 ms    0/  837 (0%)
[  3] 12.0-13.0 sec  1.17 MBytes  9.83 Mbits/sec   2.136 ms    1/  837 (0.12%)
[  3] 13.0-14.0 sec  1.17 MBytes  9.82 Mbits/sec   2.087 ms    2/  837 (0.24%)
[  3] 14.0-15.0 sec  1.17 MBytes  9.83 Mbits/sec   2.061 ms    1/  837 (0.12%)
[  3] 15.0-16.0 sec  1.17 MBytes  9.84 Mbits/sec   2.045 ms    0/  837 (0%)
[  3] 16.0-17.0 sec  1.17 MBytes  9.82 Mbits/sec   2.203 ms    1/  836 (0.12%)
[  3] 17.0-18.0 sec  1.17 MBytes  9.84 Mbits/sec   2.165 ms    0/  837 (0%)
[  3] 18.0-19.0 sec  1.17 MBytes  9.83 Mbits/sec   2.154 ms    1/  837 (0.12%)
[  3] 19.0-20.0 sec  1.19 MBytes  9.98 Mbits/sec   2.209 ms    0/  849 (0%)
[  3]  0.0-20.0 sec  23.5 MBytes  9.84 Mbits/sec   2.548 ms   13/16764 (0.078%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

The real question remains

We are not oversubscribing the DC link since it's at 100Mbps and cannot send more than 100Mbps. However the remote sites are at 10Mbps.

  • Are the buffers at the remote side overflowing and dropping packets?
  • Is the provider's traffic shaper doing something to the traffic? (Would the traffic coming from another node be influenced by the ISPs traffic shaper or only the traffic ingressing the node (from outside))...... You see what I mean?

Why can't TCP handle that all on its own?

Update #3

I tried the following scenario:


    Laptop ------- ... LAN ... --- DC switch --- Metro-Eth --- Laptop (directly connected)
    NIC@10Mbps                       100Mbps                  NIC@10Mbps

Here is the packet loss in the DC->remote direction: (iperf 9 Mbps UDP test)

    [  3] local 192.168.191.200 port 5001 connected with 192.168.191.201 port 55236
    [ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
    [  3]  0.0- 1.0 sec   912 KBytes  7.47 Mbits/sec   2.713 ms    0/  635 (0%)
    [  3]  1.0- 2.0 sec  1001 KBytes  8.20 Mbits/sec   2.168 ms    0/  697 (0%)
    [  3]  2.0- 3.0 sec  1001 KBytes  8.20 Mbits/sec   2.478 ms    0/  697 (0%)
    [  3]  3.0- 4.0 sec   999 KBytes  8.18 Mbits/sec   0.933 ms    0/  696 (0%)
    [  3]  4.0- 5.0 sec  1001 KBytes  8.20 Mbits/sec   2.620 ms    0/  697 (0%)
    [  3]  5.0- 6.0 sec  1001 KBytes  8.20 Mbits/sec   2.721 ms    0/  697 (0%)
    [  3]  6.0- 7.0 sec  1001 KBytes  8.20 Mbits/sec   2.089 ms    0/  697 (0%)
    [  3]  7.0- 8.0 sec   999 KBytes  8.18 Mbits/sec   2.641 ms    0/  696 (0%)
    [  3]  8.0- 9.0 sec  1002 KBytes  8.21 Mbits/sec   0.896 ms    0/  698 (0%)
    [  3]  9.0-10.0 sec  1015 KBytes  8.31 Mbits/sec   2.557 ms    0/  707 (0%)
    [  3] 10.0-11.0 sec   999 KBytes  8.18 Mbits/sec   2.822 ms    1/  697 (0.14%)
    [  3] 11.0-12.0 sec   999 KBytes  8.18 Mbits/sec   1.551 ms    1/  697 (0.14%)
    [  3] 12.0-13.0 sec   998 KBytes  8.17 Mbits/sec   2.504 ms    2/  697 (0.29%)
    [  3] 13.0-14.0 sec   995 KBytes  8.15 Mbits/sec   2.038 ms    3/  696 (0.43%)
    [  3] 14.0-15.0 sec   991 KBytes  8.11 Mbits/sec   2.539 ms    7/  697 (1%)
    [  3] 15.0-16.0 sec   992 KBytes  8.13 Mbits/sec   2.759 ms    6/  697 (0.86%)
    [  3] 16.0-17.0 sec   998 KBytes  8.17 Mbits/sec   2.229 ms    2/  697 (0.29%)
    [  3] 17.0-18.0 sec   993 KBytes  8.14 Mbits/sec   2.723 ms    4/  696 (0.57%)
    [  3] 18.0-19.0 sec   998 KBytes  8.17 Mbits/sec   2.038 ms    2/  697 (0.29%)
    [  3] 19.0-20.0 sec  1012 KBytes  8.29 Mbits/sec   2.575 ms    3/  708 (0.42%)
    [  3]  0.0-20.0 sec  19.5 MBytes  8.15 Mbits/sec   2.775 ms   31/13917 (0.22%)
    [  3]  0.0-20.0 sec  1 datagrams received out-of-order

The other direction is fine. However, when running a TCP test the remote->DC direction does not perform much better than the DC->remote direction (about 5Mbps)...

I'm not sure we reached the bottom of this.

Improve this question
3
  • Not really an answer but my recommendation would be to get a JDSU and test this circuit. If they are policing you, then make sure you get the policer, "regulator", settings... If they have a small CBS then they are confining your TCP traffic to essentially a smaller window size. You can test for this via a back-2-back test. I have spent a lot of time doing the back and forth with providers on L2 circuits to know that when we get a new circuit test it thoroughly not only at the CIR but at the CBS...
    – matak
    Jul 30, 2014 at 13:13
  • Also, just a quick side note. The TCP throughput that is seen from a Windows OS vs Linux is going to be different because the TCP settings are going to be different; ie. buffer size, algorithm, etc. You can view the settings for you Linux machine via sysctl not sure about about Windows... maybe netsh. If I was going to take a guess for what is wrong with your circuit I would say that the CPE at the spoke site is setup with a larger CBS than the hub side... which is usually the other way around. Again, the JDSU will punt the ball back to them or let you refocus on what the issue is.
    – matak
    Jul 30, 2014 at 13:45
  • @matak Why not make an additional answer of your remarks? When we talk about the shaper, where do I imagine this device? At the DC there is a RJ45 plug with no (visible) CPE. At the remote sites I mostly have a VDSL modem and some kind of MPLS-capable router. Not sure if they use MPLS though. And furthermore Which direction of the traffic does the shaper shape? We can shape ingress@spoke (from site), egress@spoke (toward ISP's cloud), ingress@hub (from DC), egress@hub (toward ISP's cloud)... I'm probably missing the big picture. Can you illustrate why the issue with the CBS would be a problem?
    – Marki
    Aug 5, 2014 at 20:46

1 Answer 1

Reset to default
3

While discussing this problem was very interesting, the ISP has in the meantime started exchanging the DSL modems on the different sites by another brand. Some packet fragmentation issue they say. And hey, 9.5 Mbps in both directions without any problems or special settings.

Improve this answer
1
  • "Packet fragmentation" is bogus for an L2 connection... I guess a hardwired VDSL profile was the problem.
    – Zac67 ♦
    Oct 1, 2021 at 13:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.