14

The setup

We have rented a few leased lines that present themselves as a layer 2 network, i.e. you have one big pipe at the datacenter and the remote sites have smaller pipes. Inside the layer 2 network you can do whatever you like. Probably they use 802.1ad to give each customer their separate network inside their network. AFAICS most sites are connected via plain VDSL.

We decided to put a router at each site, and give each site its own VLAN. The firewall at the DC thus has as many VLANs defined as there are sites. Each site thus uses its on address range in its own VLAN.

Network diagram:

network diagram

The problem

Now, we are faced with throughput problems:

  • Running an FTP transfer from site to DC works fine at about 10Mb/s which is line speed.
  • Running an FTP transfer from DC to site does not work fine at 6Mb/s or less.

It does not matter which side initiates the transfer. The only consistent thing is that one direction is not working well. Too bad it is the direction toward the site because that would be the bandwidth we need most as we'd like to use terminal server clients.

About 10 seconds into the transfer, the throughput drops. We see DUP ACKs when sniffing. Which maybe leads me to rate limiting at the provider's end?? (Currently, they don't have a clue, and I like to make sure we're not at fault before escalating)

NOTE The remote sites are limited to 10Mb somehow. Setting the switch-to-Metro-port to 10Mb doesn't help either. In fact it's the worst then (max. 30 KB/s). Setting to 100Mb works fine but already starts to produce the outlined problem. Same for 1G.

Captures of the problem can be downloaded here:

* http://178.63.11.6/dc-to-remote_dc-side.pcapng
* http://178.63.11.6/dc-to-remote_remote-side.pcapng

Diagnostics

In the image you see the Wireshark IO Graph with some error details:

  • on the left: FTP transfer from DC to site
  • on the right: FTP transfer from site to DC

duplicate acks

In case the other side initiates the transfer (i.e. put from dc, instead of get from remote), the problem remains unchanged.

Please indulge me with what you think could be the problem here.


UPDATE #1 (integrated above)


UPDATE #2 (UPDATED)

This must be a congestion control thing.

Note that from DC to remote we have 10G->1G->100M->10M->1G links. <-- not working

In the other direction we thus have the inverse: 1G->10M->100M->1G->10G. <-- just fine

The first "1G->10M" is the "invisible" 10M at the remote site, where everything including the uplink port speed is set at 1G, even though there's only 10M behind it (being sold).

However the 100Mbps at the DC are real 100Mbps, the interface is configured at 100Mbps on the physical layer.

I now used iperf:

  • TCP tests work fine only in one direction (client=DC, server=remote)
./iperf -c 192.168.x -i2 -t 60 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.x, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.x port 38195 connected with 192.168.x port 5001
[  3]  0.0- 2.0 sec  1.44 MBytes  6.03 Mbits/sec
[  3]  2.0- 4.0 sec  2.23 MBytes  9.37 Mbits/sec
[  3]  4.0- 6.0 sec  2.28 MBytes  9.57 Mbits/sec
[  3]  6.0- 8.0 sec  1.88 MBytes  7.90 Mbits/sec
[  3]  8.0-10.0 sec  1.00 MBytes  4.19 Mbits/sec
[  3] 10.0-12.0 sec  1.30 MBytes  5.47 Mbits/sec
[  3] 12.0-14.0 sec    688 KBytes  2.82 Mbits/sec
[  3] 14.0-16.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 16.0-18.0 sec  1.03 MBytes  4.33 Mbits/sec
[  3] 18.0-20.0 sec  1.01 MBytes  4.23 Mbits/sec
[  3] 20.0-22.0 sec  1.03 MBytes  4.33 Mbits/sec
[  3] 22.0-24.0 sec  1.18 MBytes  4.95 Mbits/sec
[  3] 24.0-26.0 sec    904 KBytes  3.70 Mbits/sec
[  3] 26.0-28.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 28.0-30.0 sec    936 KBytes  3.83 Mbits/sec
[  3] 30.0-32.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 32.0-34.0 sec    960 KBytes  3.93 Mbits/sec
[  3] 34.0-36.0 sec    752 KBytes  3.08 Mbits/sec
[  3] 36.0-38.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 38.0-40.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 40.0-42.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 42.0-44.0 sec  1.27 MBytes  5.34 Mbits/sec
[  3] 44.0-46.0 sec  1.16 MBytes  4.85 Mbits/sec
[  3] 46.0-48.0 sec    840 KBytes  3.44 Mbits/sec
[  3] 48.0-50.0 sec    960 KBytes  3.93 Mbits/sec
[  3] 50.0-52.0 sec  1.28 MBytes  5.37 Mbits/sec
[  3] 52.0-54.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3] 54.0-56.0 sec    992 KBytes  4.06 Mbits/sec
[  3] 56.0-58.0 sec  1.00 MBytes  4.19 Mbits/sec
[  3] 58.0-60.0 sec  1.09 MBytes  4.59 Mbits/sec
[  3]  0.0-60.2 sec  33.9 MBytes  4.73 Mbits/sec
[  5] local 10.x port 5001 connected with 192.168.x port 10965
[  5]  0.0- 2.0 sec  1.85 MBytes  7.75 Mbits/sec
[  5]  2.0- 4.0 sec  1.90 MBytes  7.98 Mbits/sec
[  5]  4.0- 6.0 sec  1.89 MBytes  7.93 Mbits/sec
[  5]  6.0- 8.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5]  8.0-10.0 sec  1.91 MBytes  8.02 Mbits/sec
[  5] 10.0-12.0 sec  1.83 MBytes  7.69 Mbits/sec
[  5] 12.0-14.0 sec  1.86 MBytes  7.78 Mbits/sec
[  5] 14.0-16.0 sec  1.79 MBytes  7.52 Mbits/sec
[  5] 16.0-18.0 sec  1.79 MBytes  7.52 Mbits/sec
[  5] 18.0-20.0 sec  1.89 MBytes  7.91 Mbits/sec
[  5] 20.0-22.0 sec  1.91 MBytes  8.00 Mbits/sec
[  5] 22.0-24.0 sec  1.88 MBytes  7.91 Mbits/sec
[  5] 24.0-26.0 sec  1.95 MBytes  8.16 Mbits/sec
[  5] 26.0-28.0 sec  1.90 MBytes  7.99 Mbits/sec
[  5] 28.0-30.0 sec  1.87 MBytes  7.84 Mbits/sec
[  5] 30.0-32.0 sec  1.85 MBytes  7.77 Mbits/sec
[  5] 32.0-34.0 sec  1.55 MBytes  6.49 Mbits/sec
[  5] 34.0-36.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5] 36.0-38.0 sec  1.90 MBytes  7.99 Mbits/sec
[  5] 38.0-40.0 sec  1.84 MBytes  7.73 Mbits/sec
[  5] 40.0-42.0 sec  1.66 MBytes  6.95 Mbits/sec
[  5] 42.0-44.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5] 44.0-46.0 sec  1.91 MBytes  7.99 Mbits/sec
[  5] 46.0-48.0 sec  1.90 MBytes  7.98 Mbits/sec
[  5] 48.0-50.0 sec  1.84 MBytes  7.70 Mbits/sec
[  5] 50.0-52.0 sec  1.93 MBytes  8.09 Mbits/sec
[  5] 52.0-54.0 sec  1.80 MBytes  7.54 Mbits/sec
[  5] 54.0-56.0 sec  1.83 MBytes  7.67 Mbits/sec
[  5] 56.0-58.0 sec  1.88 MBytes  7.86 Mbits/sec
[  5] 58.0-60.0 sec  1.85 MBytes  7.78 Mbits/sec
[  5]  0.0-60.3 sec  56.0 MBytes  7.79 Mbits/sec
  • To get to the bottom of it, here are UDP tests from two hosts in the same VLAN yet using the Metro Connection, 200=remote, 201=DC

We see the packet loss increasing with incrementing bandwidth (when approaching 10Mbps we have 0.93%, starts to be critical... and would also explain why TCP has problem performing)


++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61759 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  1.0- 2.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  2.0- 3.0 sec   129 KBytes  1.06 Mbits/sec
[  4]  3.0- 4.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  4.0- 5.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  5.0- 6.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  6.0- 7.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  7.0- 8.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  8.0- 9.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  9.0-10.0 sec   129 KBytes  1.06 Mbits/sec
[  4] 10.0-11.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 11.0-12.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 12.0-13.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 13.0-14.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 14.0-15.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 15.0-16.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 16.0-17.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 17.0-18.0 sec   128 KBytes  1.05 Mbits/sec
[  4] 18.0-19.0 sec   131 KBytes  1.07 Mbits/sec
[  4] 19.0-20.0 sec   128 KBytes  1.05 Mbits/sec
[  4]  0.0-20.0 sec  2.50 MBytes  1.05 Mbits/sec
[  4] Sent 1785 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  2.50 MBytes  1.05 Mbits/sec   0.257 ms    0/ 1785 (0%)
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50749
[  3]  0.0- 1.0 sec   128 KBytes  1.05 Mbits/sec   0.285 ms    0/   89 (0%)
[  3]  1.0- 2.0 sec   128 KBytes  1.05 Mbits/sec   0.313 ms    0/   89 (0%)
[  3]  2.0- 3.0 sec   128 KBytes  1.05 Mbits/sec   0.278 ms    0/   89 (0%)
[  3]  3.0- 4.0 sec   128 KBytes  1.05 Mbits/sec   0.241 ms    0/   89 (0%)
[  3]  4.0- 5.0 sec   128 KBytes  1.05 Mbits/sec   0.266 ms    0/   89 (0%)
[  3]  5.0- 6.0 sec   128 KBytes  1.05 Mbits/sec   0.293 ms    0/   89 (0%)
[  3]  6.0- 7.0 sec   128 KBytes  1.05 Mbits/sec   0.314 ms    0/   89 (0%)
[  3]  7.0- 8.0 sec   128 KBytes  1.05 Mbits/sec   0.280 ms    0/   89 (0%)
[  3]  8.0- 9.0 sec   128 KBytes  1.05 Mbits/sec   0.242 ms    0/   89 (0%)
[  3]  9.0-10.0 sec   129 KBytes  1.06 Mbits/sec   0.250 ms    0/   90 (0%)
[  3] 10.0-11.0 sec   128 KBytes  1.05 Mbits/sec   0.275 ms    0/   89 (0%)
[  3] 11.0-12.0 sec   128 KBytes  1.05 Mbits/sec   0.299 ms    0/   89 (0%)
[  3] 12.0-13.0 sec   128 KBytes  1.05 Mbits/sec   0.327 ms    0/   89 (0%)
[  3] 13.0-14.0 sec   128 KBytes  1.05 Mbits/sec   0.290 ms    0/   89 (0%)
[  3] 14.0-15.0 sec   128 KBytes  1.05 Mbits/sec   0.251 ms    0/   89 (0%)
[  3] 15.0-16.0 sec   128 KBytes  1.05 Mbits/sec   0.275 ms    0/   89 (0%)
[  3] 16.0-17.0 sec   128 KBytes  1.05 Mbits/sec   0.303 ms    0/   89 (0%)
[  3] 17.0-18.0 sec   128 KBytes  1.05 Mbits/sec   0.333 ms    0/   89 (0%)
[  3] 18.0-19.0 sec   128 KBytes  1.05 Mbits/sec   0.294 ms    0/   89 (0%)
[  3] 19.0-20.0 sec   131 KBytes  1.07 Mbits/sec   0.281 ms    0/   91 (0%)
[  3]  0.0-20.0 sec  2.50 MBytes  1.05 Mbits/sec   0.305 ms    0/ 1785 (0%)

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 5m
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61760 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  1.0- 2.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  2.0- 3.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  3.0- 4.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  4.0- 5.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  5.0- 6.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  6.0- 7.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  7.0- 8.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  8.0- 9.0 sec   610 KBytes  5.00 Mbits/sec
[  4]  9.0-10.0 sec   619 KBytes  5.07 Mbits/sec
[  4] 10.0-11.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 11.0-12.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 12.0-13.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 13.0-14.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 14.0-15.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 15.0-16.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 16.0-17.0 sec   609 KBytes  4.99 Mbits/sec
[  4] 17.0-18.0 sec   610 KBytes  5.00 Mbits/sec
[  4] 18.0-19.0 sec   619 KBytes  5.07 Mbits/sec
[  4] 19.0-20.0 sec   609 KBytes  4.99 Mbits/sec
[  4]  0.0-20.0 sec  11.9 MBytes  5.00 Mbits/sec
[  4] Sent 8504 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  11.9 MBytes  4.99 Mbits/sec   0.000 ms   12/ 8503 (0.14%)
[  4]  0.0-20.0 sec  1 datagrams received out-of-order
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50750
[  3]  0.0- 1.0 sec   606 KBytes  4.96 Mbits/sec   2.238 ms    1/  423 (0.24%)
[  3]  1.0- 2.0 sec   610 KBytes  5.00 Mbits/sec   2.739 ms    0/  425 (0%)
[  3]  2.0- 3.0 sec   609 KBytes  4.99 Mbits/sec   3.089 ms    1/  425 (0.24%)
[  3]  3.0- 4.0 sec   609 KBytes  4.99 Mbits/sec   3.605 ms    0/  424 (0%)
[  3]  4.0- 5.0 sec   607 KBytes  4.97 Mbits/sec   1.954 ms    0/  423 (0%)
[  3]  5.0- 6.0 sec   612 KBytes  5.01 Mbits/sec   2.666 ms    0/  426 (0%)
[  3]  6.0- 7.0 sec   607 KBytes  4.97 Mbits/sec   2.602 ms    0/  423 (0%)
[  3]  7.0- 8.0 sec   612 KBytes  5.01 Mbits/sec   2.960 ms    0/  426 (0%)
[  3]  8.0- 9.0 sec   609 KBytes  4.99 Mbits/sec   2.512 ms    0/  424 (0%)
[  3]  9.0-10.0 sec   619 KBytes  5.07 Mbits/sec   2.133 ms    0/  431 (0%)
[  3] 10.0-11.0 sec   609 KBytes  4.99 Mbits/sec   3.605 ms    1/  425 (0.24%)
[  3] 11.0-12.0 sec   609 KBytes  4.99 Mbits/sec   2.509 ms    0/  424 (0%)
[  3] 12.0-13.0 sec   610 KBytes  5.00 Mbits/sec   3.570 ms    0/  425 (0%)
[  3] 13.0-14.0 sec   609 KBytes  4.99 Mbits/sec   3.077 ms    1/  425 (0.24%)
[  3] 14.0-15.0 sec   609 KBytes  4.99 Mbits/sec   2.679 ms    0/  424 (0%)
[  3] 15.0-16.0 sec   609 KBytes  4.99 Mbits/sec   1.887 ms    0/  424 (0%)
[  3] 16.0-17.0 sec   610 KBytes  5.00 Mbits/sec   2.651 ms    0/  425 (0%)
[  3] 17.0-18.0 sec   609 KBytes  4.99 Mbits/sec   3.390 ms    0/  424 (0%)
[  3] 18.0-19.0 sec   617 KBytes  5.06 Mbits/sec   2.601 ms    0/  430 (0%)
[  3] 19.0-20.0 sec   612 KBytes  5.01 Mbits/sec   3.525 ms    0/  426 (0%)
[  3]  0.0-20.0 sec  11.9 MBytes  4.99 Mbits/sec   3.156 ms    3/ 8503 (0.035%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 9m
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61761 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  1.0- 2.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  2.0- 3.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  3.0- 4.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  4.0- 5.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  5.0- 6.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  6.0- 7.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  7.0- 8.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  8.0- 9.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4]  9.0-10.0 sec  1.09 MBytes  9.14 Mbits/sec
[  4] 10.0-11.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4] 11.0-12.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 12.0-13.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 13.0-14.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4] 14.0-15.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 15.0-16.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4] 16.0-17.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 17.0-18.0 sec  1.07 MBytes  8.98 Mbits/sec
[  4] 18.0-19.0 sec  1.09 MBytes  9.14 Mbits/sec
[  4] 19.0-20.0 sec  1.07 MBytes  9.00 Mbits/sec
[  4]  0.0-20.0 sec  21.5 MBytes  9.00 Mbits/sec
[  4] Sent 15315 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  21.3 MBytes  8.94 Mbits/sec   0.104 ms   96/15314 (0.63%)  !!!!!!!!!!
[  4]  0.0-20.0 sec  1 datagrams received out-of-order
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50751
[  3]  0.0- 1.0 sec  1.06 MBytes  8.89 Mbits/sec   2.405 ms    0/  756 (0%)
[  3]  1.0- 2.0 sec  1.07 MBytes  9.00 Mbits/sec   2.308 ms    0/  765 (0%)
[  3]  2.0- 3.0 sec  1.07 MBytes  9.00 Mbits/sec   2.305 ms    0/  765 (0%)
[  3]  3.0- 4.0 sec  1.07 MBytes  8.97 Mbits/sec   2.290 ms    1/  764 (0.13%)
[  3]  4.0- 5.0 sec  1.07 MBytes  8.98 Mbits/sec   2.271 ms    1/  765 (0.13%)
[  3]  5.0- 6.0 sec  1.07 MBytes  8.98 Mbits/sec   2.313 ms    0/  764 (0%)
[  3]  6.0- 7.0 sec  1.07 MBytes  9.00 Mbits/sec   2.191 ms    0/  765 (0%)
[  3]  7.0- 8.0 sec  1.07 MBytes  8.95 Mbits/sec   2.314 ms    3/  764 (0.39%)
[  3]  8.0- 9.0 sec  1.07 MBytes  8.98 Mbits/sec   2.232 ms    1/  765 (0.13%)
[  3]  9.0-10.0 sec  1.09 MBytes  9.13 Mbits/sec   2.257 ms    0/  776 (0%)
[  3] 10.0-11.0 sec  1.07 MBytes  8.98 Mbits/sec   2.365 ms    0/  764 (0%)
[  3] 11.0-12.0 sec  1.07 MBytes  8.98 Mbits/sec   2.301 ms    1/  765 (0.13%)
[  3] 12.0-13.0 sec  1.07 MBytes  8.98 Mbits/sec   2.277 ms    0/  764 (0%)
[  3] 13.0-14.0 sec  1.07 MBytes  9.00 Mbits/sec   2.323 ms    0/  765 (0%)
[  3] 14.0-15.0 sec  1.07 MBytes  9.00 Mbits/sec   2.176 ms    0/  765 (0%)
[  3] 15.0-16.0 sec  1.07 MBytes  8.96 Mbits/sec   2.273 ms    2/  764 (0.26%)
[  3] 16.0-17.0 sec  1.07 MBytes  8.98 Mbits/sec   2.313 ms    0/  764 (0%)
[  3] 17.0-18.0 sec  1.07 MBytes  8.98 Mbits/sec   2.247 ms    1/  765 (0.13%)
[  3] 18.0-19.0 sec  1.09 MBytes  9.11 Mbits/sec   2.276 ms    1/  776 (0.13%)
[  3] 19.0-20.0 sec  1.07 MBytes  8.97 Mbits/sec   2.394 ms    1/  764 (0.13%)
[  3]  0.0-20.0 sec  21.5 MBytes  8.99 Mbits/sec   2.659 ms   11/15314 (0.072%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 9850k
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
------------------------------------------------------------
Server listening on UDP port 5001
Receiving 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.191.200, UDP port 5001
Sending 1470 byte datagrams
UDP buffer size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.191.201 port 61762 connected with 192.168.191.200 port 5001
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0- 1.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  1.0- 2.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  2.0- 3.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  3.0- 4.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  4.0- 5.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  5.0- 6.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4]  6.0- 7.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  7.0- 8.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  8.0- 9.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  9.0-10.0 sec  1.19 MBytes  10.0 Mbits/sec
[  4] 10.0-11.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4] 11.0-12.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4] 12.0-13.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4] 13.0-14.0 sec  1.17 MBytes  9.85 Mbits/sec
[  4] 14.0-15.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4] 15.0-16.0 sec  1.17 MBytes  9.85 Mbits/sec
[  4] 16.0-17.0 sec  1.17 MBytes  9.83 Mbits/sec
[  4] 17.0-18.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4] 18.0-19.0 sec  1.19 MBytes  10.0 Mbits/sec
[  4] 19.0-20.0 sec  1.17 MBytes  9.84 Mbits/sec
[  4]  0.0-20.0 sec  23.5 MBytes  9.85 Mbits/sec
[  4] Sent 16765 datagrams
[  4] Server Report:
[  4]  0.0-20.0 sec  23.3 MBytes  9.74 Mbits/sec   3.421 ms  156/16764 (0.93%) !!!!!!!!!!
[  4]  0.0-20.0 sec  1 datagrams received out-of-order
[  3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50752
[  3]  0.0- 1.0 sec  1.16 MBytes  9.74 Mbits/sec   2.131 ms    0/  828 (0%)
[  3]  1.0- 2.0 sec  1.17 MBytes  9.84 Mbits/sec   2.140 ms    0/  837 (0%)
[  3]  2.0- 3.0 sec  1.17 MBytes  9.83 Mbits/sec   2.099 ms    1/  837 (0.12%)
[  3]  3.0- 4.0 sec  1.17 MBytes  9.84 Mbits/sec   2.113 ms    0/  837 (0%)
[  3]  4.0- 5.0 sec  1.17 MBytes  9.84 Mbits/sec   2.105 ms    0/  837 (0%)
[  3]  5.0- 6.0 sec  1.17 MBytes  9.83 Mbits/sec   2.058 ms    1/  837 (0.12%)
[  3]  6.0- 7.0 sec  1.17 MBytes  9.82 Mbits/sec   2.165 ms    1/  836 (0.12%)
[  3]  7.0- 8.0 sec  1.17 MBytes  9.84 Mbits/sec   2.156 ms    0/  837 (0%)
[  3]  8.0- 9.0 sec  1.17 MBytes  9.82 Mbits/sec   2.135 ms    2/  837 (0.24%)
[  3]  9.0-10.0 sec  1.19 MBytes  9.97 Mbits/sec   2.152 ms    2/  850 (0.24%)
[  3] 10.0-11.0 sec  1.17 MBytes  9.83 Mbits/sec   2.153 ms    1/  837 (0.12%)
[  3] 11.0-12.0 sec  1.17 MBytes  9.84 Mbits/sec   2.127 ms    0/  837 (0%)
[  3] 12.0-13.0 sec  1.17 MBytes  9.83 Mbits/sec   2.136 ms    1/  837 (0.12%)
[  3] 13.0-14.0 sec  1.17 MBytes  9.82 Mbits/sec   2.087 ms    2/  837 (0.24%)
[  3] 14.0-15.0 sec  1.17 MBytes  9.83 Mbits/sec   2.061 ms    1/  837 (0.12%)
[  3] 15.0-16.0 sec  1.17 MBytes  9.84 Mbits/sec   2.045 ms    0/  837 (0%)
[  3] 16.0-17.0 sec  1.17 MBytes  9.82 Mbits/sec   2.203 ms    1/  836 (0.12%)
[  3] 17.0-18.0 sec  1.17 MBytes  9.84 Mbits/sec   2.165 ms    0/  837 (0%)
[  3] 18.0-19.0 sec  1.17 MBytes  9.83 Mbits/sec   2.154 ms    1/  837 (0.12%)
[  3] 19.0-20.0 sec  1.19 MBytes  9.98 Mbits/sec   2.209 ms    0/  849 (0%)
[  3]  0.0-20.0 sec  23.5 MBytes  9.84 Mbits/sec   2.548 ms   13/16764 (0.078%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

The real question remains:

We are not oversubscribing the DC link since it's at 100Mbps and cannot send more than 100Mbps. However the remote sites are at 10Mbps.

  • Are the buffers at the remote side overflowing and dropping packets?
  • Is the provider's traffic shaper doing something to the traffic? (Would the traffic coming from another node be influenced by the ISPs traffic shaper or only the traffic ingressing the node (from outside))...... You see what I mean?

Why can't TCP handle that all on its own?


Update #3 I have now used the following scenario:

Laptop ------- ... LAN ... --- DC switch --- Metro-Eth --- Laptop (directly connected)
NIC@10Mbps                       100Mbps                  NIC@10Mbps

Here is the packet loss in the DC->remote direction: (iperf 9 Mbps UDP test)

[  3] local 192.168.191.200 port 5001 connected with 192.168.191.201 port 55236
[ ID] Interval       Transfer     Bandwidth        Jitter   Lost/Total Datagrams
[  3]  0.0- 1.0 sec   912 KBytes  7.47 Mbits/sec   2.713 ms    0/  635 (0%)
[  3]  1.0- 2.0 sec  1001 KBytes  8.20 Mbits/sec   2.168 ms    0/  697 (0%)
[  3]  2.0- 3.0 sec  1001 KBytes  8.20 Mbits/sec   2.478 ms    0/  697 (0%)
[  3]  3.0- 4.0 sec   999 KBytes  8.18 Mbits/sec   0.933 ms    0/  696 (0%)
[  3]  4.0- 5.0 sec  1001 KBytes  8.20 Mbits/sec   2.620 ms    0/  697 (0%)
[  3]  5.0- 6.0 sec  1001 KBytes  8.20 Mbits/sec   2.721 ms    0/  697 (0%)
[  3]  6.0- 7.0 sec  1001 KBytes  8.20 Mbits/sec   2.089 ms    0/  697 (0%)
[  3]  7.0- 8.0 sec   999 KBytes  8.18 Mbits/sec   2.641 ms    0/  696 (0%)
[  3]  8.0- 9.0 sec  1002 KBytes  8.21 Mbits/sec   0.896 ms    0/  698 (0%)
[  3]  9.0-10.0 sec  1015 KBytes  8.31 Mbits/sec   2.557 ms    0/  707 (0%)
[  3] 10.0-11.0 sec   999 KBytes  8.18 Mbits/sec   2.822 ms    1/  697 (0.14%)
[  3] 11.0-12.0 sec   999 KBytes  8.18 Mbits/sec   1.551 ms    1/  697 (0.14%)
[  3] 12.0-13.0 sec   998 KBytes  8.17 Mbits/sec   2.504 ms    2/  697 (0.29%)
[  3] 13.0-14.0 sec   995 KBytes  8.15 Mbits/sec   2.038 ms    3/  696 (0.43%)
[  3] 14.0-15.0 sec   991 KBytes  8.11 Mbits/sec   2.539 ms    7/  697 (1%)
[  3] 15.0-16.0 sec   992 KBytes  8.13 Mbits/sec   2.759 ms    6/  697 (0.86%)
[  3] 16.0-17.0 sec   998 KBytes  8.17 Mbits/sec   2.229 ms    2/  697 (0.29%)
[  3] 17.0-18.0 sec   993 KBytes  8.14 Mbits/sec   2.723 ms    4/  696 (0.57%)
[  3] 18.0-19.0 sec   998 KBytes  8.17 Mbits/sec   2.038 ms    2/  697 (0.29%)
[  3] 19.0-20.0 sec  1012 KBytes  8.29 Mbits/sec   2.575 ms    3/  708 (0.42%)
[  3]  0.0-20.0 sec  19.5 MBytes  8.15 Mbits/sec   2.775 ms   31/13917 (0.22%)
[  3]  0.0-20.0 sec  1 datagrams received out-of-order

The other direction is fine. However, when running a TCP test the remote->DC direction does not perform much better than the DC->remote direction (about 5Mbps).......

I'm not sure we reached the bottom of this.

  • Not really an answer but my recommendation would be to get a JDSU and test this circuit. If they are policing you, then make sure you get the policer, "regulator", settings... If they have a small CBS then they are confining your TCP traffic to essentially a smaller window size. You can test for this via a back-2-back test. I have spent a lot of time doing the back and forth with providers on L2 circuits to know that when we get a new circuit test it thoroughly not only at the CIR but at the CBS... – matak Jul 30 '14 at 13:13
  • Also, just a quick side note. The TCP throughput that is seen from a Windows OS vs Linux is going to be different because the TCP settings are going to be different; ie. buffer size, algorithm, etc. You can view the settings for you Linux machine via sysctl not sure about about Windows... maybe netsh. If I was going to take a guess for what is wrong with your circuit I would say that the CPE at the spoke site is setup with a larger CBS than the hub side... which is usually the other way around. Again, the JDSU will punt the ball back to them or let you refocus on what the issue is. – matak Jul 30 '14 at 13:45
  • @matak Why not make an additional answer of your remarks? When we talk about the shaper, where do I imagine this device? At the DC there is a RJ45 plug with no (visible) CPE. At the remote sites I mostly have a VDSL modem and some kind of MPLS-capable router. Not sure if they use MPLS though. And furthermore Which direction of the traffic does the shaper shape? We can shape ingress@spoke (from site), egress@spoke (toward ISP's cloud), ingress@hub (from DC), egress@hub (toward ISP's cloud)... I'm probably missing the big picture. Can you illustrate why the issue with the CBS would be a problem? – Marki Aug 5 '14 at 20:46
20
+50

Referencing our Stack Exchange chat...

Short story, you need to control the speed mismatches on both sides of your metro Ethernet links... I redrew your diagram for the sake of clarity... Note 1

Problem Diagram

  • The DC (shown in green) transitions from 10GE to 100M very quickly... this is a 100-fold speed transition, and you commonly need to implement some form of qos (such as shaping) to mitigate such a large transition. See the bottom of this answer for evidence that the DC needs shaping (per site)...
  • The remote side transitions from 1GE to a 10M CIR very quickly... this again, is a 100-fold speed transition. Shaping or other qos workarounds are typically required.
  • There also seems to be a speed mismatch between the DC UNI (100M) and the remote UNI (10M); this itself begs for a per-site bandwidth-management solution.

FYI, if your provider implements MEF-compliant services, they are not shaping, they are policing. TCP traffic tends to perform better with shaping.

The need for your own QoS

You seem to question the need for qos, so I will quote from the MEF "Understanding Carrier Ethernet Throughput" Whitepaper, page 9... by way of review, the customer in the MEF Whitepaper's Figure 2 has a better situation than you do... they purchased a 50Mbps CIR, but their UNI is delivered on a 1GE... your remote site has a 10Mbps CIR on a 1GE UNI.

The transition from legacy services such as T1, T3, Frame Relay and ATM
to Carrier Ethernet has created some unintended consequences. Not all customers have 
conforming equipment facing the network which properly limits/shapes the traffic outbound
to the network, with deleterious results.  For instance, on the 1 GigE interface of
Figure 2, if the customer’s equipment accidentally transmits long bursts of data at 
150 Mbits instead of the SLA’s Committed Information Rate of 50 Mbits, 67% of the data 
may be lost and network breakdown will likely result.

Responding to other TCP questions in an edit...

We are not oversubscribing the DC link since it's at 100Mbps and cannot send more than 100Mbps...

I disagree, you can send microbursts at 10GE because your DC has 10GE links, but the metro UNI is 100Mbps. One open question is how much buffering you have on your Enterasys LAN switch (Switch A) when you make the transition from 10GE to 100M.

Why can't TCP handle that all on its own?

TCP handles things by slowing down when it sees packet loss... it really slows down (and may abort the connection) for serious packet loss. So, TCP is doing what it should... as a network engineer, your goal is to build a network with conditions that make TCP happy.

Other TCP questions from chat

Marki said: I don't understand what is being dropped where and by whom and why and why TCP doesn't simply handle the fact that there is (real) 100Mb at one end, and only 10Mbps at the other.

Regarding TCP's need for buffering, and the consequences of no buffers:

Fact number 1: TCP needs buffering for speed transitions because it's designed as a feedback-control system.

Using a driving analogy: as good drivers we always leave a few seconds worth of space between us and the car in front of us; in some ways, that space between cars is roughly analogous to a network buffer. If the person in front of us slams on the brakes when an animal runs in front of them, the space between our cars (hopefully) prevents us from hitting their car. We leave space because it takes time for our eyes to see the brake lights, our foot to react, and the brakes to dissipate enough heat; our eyes give us a visual feedback control system.

Likewise, when an FTP session is blasted out at 10GE, bursts of traffic could be up to 4MB long (in your case) due to of TCP scaled window-size before the socket must stop and wait for a TCP ACK. Meanwhile if the 10GE stream of traffic suddenly hits a "Fast Ethernet", TCP needs to gradually slow down. Deep buffers in network gear allow TCP to drop far fewer packets when it makes speed transitions; however, if you have no buffers, you could drop perhaps 99% of that 4MB TCP window when it's throttled from 10GE to 100M. Think of that severe 99% loss as a TCP socket crash; TCP reacts predictably to relatively gradual packet loss. TCP to reacts much less predictably to ongoing, severe packet loss Note 3.

To the question of why you shouldn't use an asymmetric Metro Ethernet CIR with 100M at the DC and 10M on the remote, ask yourself a rhetorical question "who is buffering that 100Mbps traffic burst when it hits the cheap 10Mbps Ethernet NID that your metro-ethernet provider gave you?"... (hint: nobody is buffering).

If nobody is buffering the large (see Note 2) speed transitions then those points are potential places to intermittently drop traffic.

What is being dropped by whom:

Egress traffic drops from the DC

When TCP traffic leaves the data center, there are three places it could be dropped:

  • At D1: because LAN switches rarely have deep enough buffering for a 100:1 speed transition
  • At D2: if the NID ever negotiated the UNI link at a higher speed than the CIR; that's not the case right now, so I don't expect drops there.
  • At D3: for all the reasons I just described about asymmetric Metro Ethernet CIRs.

When TCP traffic goes to the data center...

Ingress traffic drops to the DC

  • At D4: because you have a 1GE UNI and a 10M CIR; this is the pathological case of D2 that I mentioned above.

How to mitigate the speed mismatches:

An example EVPL solution: EVPL with point-to-point EVC solution

  • In a switched topology like this, an EVPL with point to point EVCs from the DC to each Remote is probably your best option (see the diagram above). This would apply an individual CIR to each EVC. Note: all other QoS guidance in this answer applies... i.e. avoid large speed transitionsNote 2 without testing whether your equipment will handle that well enough.
  • Alternatively, you could consider buying metroe services which have symmetric rates between the DC and the remote; although I would concede that might not be the most practical guidance.
  • FYI, the classic solution to this problem for routed services is to buy routers that support shaping at the required speeds and then shape your metroe traffic to the appropriate CIR (per remote site). FYI, the remote side could get away with a reasonably small router, since it's only a 1GE input and a 10Mbps CIR... Months ago, when we talked about the design of this service, I recommended routing if you were comfortable with the technologies...
  • If you have no extra money to spend and cannot re-engineer your metro-ethernet service, you could massage the speed mismatches more gradually. I have never done this, but in principle you could try to make the speed transitions 10 to 1, instead of 100 to 1 (which is what you currently have at both the DC and remote):

    • Instead of buying a router to shape the remote to 10M, you could try to force the remote UNI to auto-negotiate at 100M instead of 1GE; GigabitEthernet requires all pins in a Cat5e cable, so you could effectively force it to 100M with an RJ45 mod-plug that merely connect pins 1, 2, 3 and 6.
    • Instead of buying a router to shape the DC to 100M, use your Enterasys to police the 10GE link to 1GE when sending traffic towards the 100M link

Analyzing your iperf results...

There are two key points to remember about iperf (all information based on iperf version 2):

  • iperf measures the TCP or UDP payload bandwidth; in other words it ignores the overhead of TCP, UDP, IP and Ethernet headers. Since you have an ethernet service, remember to modify the iperf results accordingly
  • The iperf client sends data to the server during tests.

As such, the following output shows that the DC machine (in iperf -c mode) connects to the iperf server at the remote site (192.168.x) and pushes data from the DC (100M UNI) to the remote site (10M UNI)...

./iperf -c 192.168.x -i2 -t 60 -r
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.x, TCP port 5001
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  3] local 10.x port 38195 connected with 192.168.x port 5001
[  3]  0.0- 2.0 sec  1.44 MBytes  6.03 Mbits/sec
[  3]  2.0- 4.0 sec  2.23 MBytes  9.37 Mbits/sec
[  3]  4.0- 6.0 sec  2.28 MBytes  9.57 Mbits/sec
[  3]  6.0- 8.0 sec  1.88 MBytes  7.90 Mbits/sec
[  3]  8.0-10.0 sec  1.00 MBytes  4.19 Mbits/sec
[  3] 10.0-12.0 sec  1.30 MBytes  5.47 Mbits/sec
[  3] 12.0-14.0 sec    688 KBytes  2.82 Mbits/sec

The output above clearly shows problems in the DC to remote direction; we should expect to see 9Mbps or more when things work well (i.e you expect at least 90% of capacity - 10Mbps at the remote site). Now, let's look at traffic in the opposite direction (when iperf pushes data from the remote site to the DC)...

[  5] local 10.x port 5001 connected with 192.168.x port 10965
[  5]  0.0- 2.0 sec  1.85 MBytes  7.75 Mbits/sec
[  5]  2.0- 4.0 sec  1.90 MBytes  7.98 Mbits/sec
[  5]  4.0- 6.0 sec  1.89 MBytes  7.93 Mbits/sec
[  5]  6.0- 8.0 sec  1.92 MBytes  8.07 Mbits/sec
[  5]  8.0-10.0 sec  1.91 MBytes  8.02 Mbits/sec
[  5] 10.0-12.0 sec  1.83 MBytes  7.69 Mbits/sec
[  5] 12.0-14.0 sec  1.86 MBytes  7.78 Mbits/sec

You're able to send around 80% of your remote CIR's capacity, but that's still less than what I expect.

Illustration of the DC speed mismatch (10Gbps -> 100Mbps)

marki said: Don't forget, the problem only shows itself when the flow is 100Mb->10Mb, not the other way round.

The problem shows itself in both directions, but the iperf symptoms seem to be worse in the DC -> remote direction. See my analysis of the iperf output above.

To make this concrete, let's look at your FTP pcap when pushing a file from your DC FTP server (130.1.6.4) to the remote site (192.168.191.2). The transfer from the 100M metro ethernet side gets limited at several points during the transfer. You can see this if you look at the dc-to-remote_remote-side.pcapng pcap, and filter on expert.message contains "segment not captured"

enter image description here


End Notes:

Note 1 I choose CBS values of 25KB per 1Mbps MetroEthernet CIR; this is a common ratio used by providers... YMMV
Note 2 My personal rule: "large" is a speed transition that is significantly larger than a 10:1 speed transition
Note 3 I can't give hard numbers for what is and isn't too much packet loss for TCP. If the loss is bad enough for your applications to suffer, then it's too much. My personal rule: When dealing with a wired corporate network completely under my own control, any (unintentional) packet loss is too much. That said, there are some switch models that cut corners on buffering; these switches may occasionally drop packets... it’s a judgement call as to whether you have to live with the problem or buy better switches. FYI: It's not always obvious, but TCP periodically increases the transmission rate of a socket to ensure that it's getting as much throughput as possible; many TCP implementations know they are going too fast when they see packet drops.

  • Note that the DC's PHY speed (Metro Ethernet port) is already at 100Mb. But I can't send at 100M either because the other side is max 10Mb... Right now it's still unclear to me where exactly the shaping should have to take place. Oh and did you mean "the iperf symptoms seem to be worse in the DC -> remote direction"? – Marki Aug 5 '14 at 11:51
  • I updated the answer, yes "remote -> DC" was a typo in the original answer. – Mike Pennington Aug 6 '14 at 11:35
  • I agree with Mike here, depending on who your provider is, if you ask them they'll tell you the line rate on their end, make that match your physical interfaces hanging off your metro-E. As for WHERE to QoS, I'd do at your biggest entry points, so your 10Gb devices, before they go up to the smaller upstream devices. I spend more time firewalling and routing than switching though, but hopefully Mike can corroborate my claims! – A L Aug 6 '14 at 15:02
  • 3
    @MikePennington - Egress blocking due to speed mismatches is something I run into a lot with P2P microwave links. Great answer, lot of good information in your post. Thanks! – matak Aug 6 '14 at 15:38
  • 1
    Also check for duplex mismatch, this can cause uni-directional speed issues. – cpt_fink Oct 3 '14 at 7:45
1

While discussing this problem was very interesting, the ISP has in the meantime started exchanging the DSL modems on the different sites by another brand. Some packet fragmentation issue they say. And hey, 9.5 Mbps in both directions without any problems or special settings.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.