Background
We rented a few layer 2 leased lines, i.e. you have one big pipe at the datacenter and the remote sites have smaller pipes. Inside the layer 2 network you can do whatever you like. Probably they use 802.1ad to give each customer their separate network inside their network. AFAICS most sites are connected via plain VDSL.
We decided to put a router at each site, and give each site its own VLAN. The firewall at the DC thus has as many VLANs defined as there are sites. Each site thus uses its on address range in its own VLAN.
Network diagram:
Problem
Now, we are faced with throughput problems:
- Running an FTP transfer from site to DC works fine at about 10Mb/s which is line speed.
- Running an FTP transfer from DC to site does not work fine at 6Mb/s or less.
It does not matter which side initiates the transfer. The only consistent thing is that one direction is not working well. Too bad it is the direction toward the site because that would be the bandwidth we need most as we'd like to use terminal server clients.
About 10 seconds into the transfer, the throughput drops. We see DUP ACKs when sniffing. Which maybe leads me to rate limiting at the provider's end?? (Currently, they don't have a clue, and I like to make sure we're not at fault before escalating)
NOTE
The remote sites are limited to 10Mb somehow. Setting the switch-to-Metro-port to 10Mb doesn't help either. In fact it's the worst then (max. 30 KB/s). Setting to 100Mb works fine but already starts to produce the outlined problem. Same for 1G.
Captures of the problem can be downloaded here:
* http://178.63.11.6/dc-to-remote_dc-side.pcapng
* http://178.63.11.6/dc-to-remote_remote-side.pcapng
Diagnostics
In the image you see the Wireshark IO Graph with some error details:
- on the left: FTP transfer from DC to site
- on the right: FTP transfer from site to DC
In case the other side initiates the transfer (i.e. put from dc, instead of get from remote), the problem remains unchanged.
Please indulge me with what you think could be the problem here.
UPDATE #1 (integrated above)
UPDATE #2 (UPDATED)
This must be a congestion control thing.
Note that from DC to remote we have 10G->1G->100M->10M->1G links. <-- not working
In the other direction we thus have the inverse: 1G->10M->100M->1G->10G. <-- just fine
The first "1G->10M" is the "invisible" 10M at the remote site, where everything including the uplink port speed is set at 1G, even though there's only 10M behind it (being sold).
However the 100Mbps at the DC are real 100Mbps, the interface is configured at 100Mbps on the physical layer.
Now using iperf:
- TCP tests work fine only in one direction (client=DC, server=remote)
./iperf -c 192.168.x -i2 -t 60 -r ------------------------------------------------------------ Server listening on TCP port 5001 TCP window size: 85.3 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 192.168.x, TCP port 5001 TCP window size: 16.0 KByte (default) ------------------------------------------------------------ [ 3] local 10.x port 38195 connected with 192.168.x port 5001 [ 3] 0.0- 2.0 sec 1.44 MBytes 6.03 Mbits/sec [ 3] 2.0- 4.0 sec 2.23 MBytes 9.37 Mbits/sec [ 3] 4.0- 6.0 sec 2.28 MBytes 9.57 Mbits/sec [ 3] 6.0- 8.0 sec 1.88 MBytes 7.90 Mbits/sec [ 3] 8.0-10.0 sec 1.00 MBytes 4.19 Mbits/sec [ 3] 10.0-12.0 sec 1.30 MBytes 5.47 Mbits/sec [ 3] 12.0-14.0 sec 688 KBytes 2.82 Mbits/sec [ 3] 14.0-16.0 sec 840 KBytes 3.44 Mbits/sec [ 3] 16.0-18.0 sec 1.03 MBytes 4.33 Mbits/sec [ 3] 18.0-20.0 sec 1.01 MBytes 4.23 Mbits/sec [ 3] 20.0-22.0 sec 1.03 MBytes 4.33 Mbits/sec [ 3] 22.0-24.0 sec 1.18 MBytes 4.95 Mbits/sec [ 3] 24.0-26.0 sec 904 KBytes 3.70 Mbits/sec [ 3] 26.0-28.0 sec 840 KBytes 3.44 Mbits/sec [ 3] 28.0-30.0 sec 936 KBytes 3.83 Mbits/sec [ 3] 30.0-32.0 sec 1.09 MBytes 4.59 Mbits/sec [ 3] 32.0-34.0 sec 960 KBytes 3.93 Mbits/sec [ 3] 34.0-36.0 sec 752 KBytes 3.08 Mbits/sec [ 3] 36.0-38.0 sec 1.09 MBytes 4.59 Mbits/sec [ 3] 38.0-40.0 sec 1.09 MBytes 4.59 Mbits/sec [ 3] 40.0-42.0 sec 840 KBytes 3.44 Mbits/sec [ 3] 42.0-44.0 sec 1.27 MBytes 5.34 Mbits/sec [ 3] 44.0-46.0 sec 1.16 MBytes 4.85 Mbits/sec [ 3] 46.0-48.0 sec 840 KBytes 3.44 Mbits/sec [ 3] 48.0-50.0 sec 960 KBytes 3.93 Mbits/sec [ 3] 50.0-52.0 sec 1.28 MBytes 5.37 Mbits/sec [ 3] 52.0-54.0 sec 1.09 MBytes 4.59 Mbits/sec [ 3] 54.0-56.0 sec 992 KBytes 4.06 Mbits/sec [ 3] 56.0-58.0 sec 1.00 MBytes 4.19 Mbits/sec [ 3] 58.0-60.0 sec 1.09 MBytes 4.59 Mbits/sec [ 3] 0.0-60.2 sec 33.9 MBytes 4.73 Mbits/sec [ 5] local 10.x port 5001 connected with 192.168.x port 10965 [ 5] 0.0- 2.0 sec 1.85 MBytes 7.75 Mbits/sec [ 5] 2.0- 4.0 sec 1.90 MBytes 7.98 Mbits/sec [ 5] 4.0- 6.0 sec 1.89 MBytes 7.93 Mbits/sec [ 5] 6.0- 8.0 sec 1.92 MBytes 8.07 Mbits/sec [ 5] 8.0-10.0 sec 1.91 MBytes 8.02 Mbits/sec [ 5] 10.0-12.0 sec 1.83 MBytes 7.69 Mbits/sec [ 5] 12.0-14.0 sec 1.86 MBytes 7.78 Mbits/sec [ 5] 14.0-16.0 sec 1.79 MBytes 7.52 Mbits/sec [ 5] 16.0-18.0 sec 1.79 MBytes 7.52 Mbits/sec [ 5] 18.0-20.0 sec 1.89 MBytes 7.91 Mbits/sec [ 5] 20.0-22.0 sec 1.91 MBytes 8.00 Mbits/sec [ 5] 22.0-24.0 sec 1.88 MBytes 7.91 Mbits/sec [ 5] 24.0-26.0 sec 1.95 MBytes 8.16 Mbits/sec [ 5] 26.0-28.0 sec 1.90 MBytes 7.99 Mbits/sec [ 5] 28.0-30.0 sec 1.87 MBytes 7.84 Mbits/sec [ 5] 30.0-32.0 sec 1.85 MBytes 7.77 Mbits/sec [ 5] 32.0-34.0 sec 1.55 MBytes 6.49 Mbits/sec [ 5] 34.0-36.0 sec 1.92 MBytes 8.07 Mbits/sec [ 5] 36.0-38.0 sec 1.90 MBytes 7.99 Mbits/sec [ 5] 38.0-40.0 sec 1.84 MBytes 7.73 Mbits/sec [ 5] 40.0-42.0 sec 1.66 MBytes 6.95 Mbits/sec [ 5] 42.0-44.0 sec 1.92 MBytes 8.07 Mbits/sec [ 5] 44.0-46.0 sec 1.91 MBytes 7.99 Mbits/sec [ 5] 46.0-48.0 sec 1.90 MBytes 7.98 Mbits/sec [ 5] 48.0-50.0 sec 1.84 MBytes 7.70 Mbits/sec [ 5] 50.0-52.0 sec 1.93 MBytes 8.09 Mbits/sec [ 5] 52.0-54.0 sec 1.80 MBytes 7.54 Mbits/sec [ 5] 54.0-56.0 sec 1.83 MBytes 7.67 Mbits/sec [ 5] 56.0-58.0 sec 1.88 MBytes 7.86 Mbits/sec [ 5] 58.0-60.0 sec 1.85 MBytes 7.78 Mbits/sec [ 5] 0.0-60.3 sec 56.0 MBytes 7.79 Mbits/sec
- To get to the bottom of it, here are UDP tests from two hosts in the same VLAN yet using the Metro Connection, 200=remote, 201=DC
We see the packet loss increasing with incrementing bandwidth (when approaching 10Mbps we have 0.93%, starts to be critical... and would also explain why TCP has problem performing)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 192.168.191.200, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.191.201 port 61759 connected with 192.168.191.200 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 2.0- 3.0 sec 129 KBytes 1.06 Mbits/sec [ 4] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 4.0- 5.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 5.0- 6.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 6.0- 7.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 7.0- 8.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 9.0-10.0 sec 129 KBytes 1.06 Mbits/sec [ 4] 10.0-11.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 11.0-12.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 12.0-13.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 13.0-14.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 14.0-15.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 15.0-16.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 16.0-17.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 17.0-18.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 18.0-19.0 sec 131 KBytes 1.07 Mbits/sec [ 4] 19.0-20.0 sec 128 KBytes 1.05 Mbits/sec [ 4] 0.0-20.0 sec 2.50 MBytes 1.05 Mbits/sec [ 4] Sent 1785 datagrams [ 4] Server Report: [ 4] 0.0-20.0 sec 2.50 MBytes 1.05 Mbits/sec 0.257 ms 0/ 1785 (0%) [ 3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50749 [ 3] 0.0- 1.0 sec 128 KBytes 1.05 Mbits/sec 0.285 ms 0/ 89 (0%) [ 3] 1.0- 2.0 sec 128 KBytes 1.05 Mbits/sec 0.313 ms 0/ 89 (0%) [ 3] 2.0- 3.0 sec 128 KBytes 1.05 Mbits/sec 0.278 ms 0/ 89 (0%) [ 3] 3.0- 4.0 sec 128 KBytes 1.05 Mbits/sec 0.241 ms 0/ 89 (0%) [ 3] 4.0- 5.0 sec 128 KBytes 1.05 Mbits/sec 0.266 ms 0/ 89 (0%) [ 3] 5.0- 6.0 sec 128 KBytes 1.05 Mbits/sec 0.293 ms 0/ 89 (0%) [ 3] 6.0- 7.0 sec 128 KBytes 1.05 Mbits/sec 0.314 ms 0/ 89 (0%) [ 3] 7.0- 8.0 sec 128 KBytes 1.05 Mbits/sec 0.280 ms 0/ 89 (0%) [ 3] 8.0- 9.0 sec 128 KBytes 1.05 Mbits/sec 0.242 ms 0/ 89 (0%) [ 3] 9.0-10.0 sec 129 KBytes 1.06 Mbits/sec 0.250 ms 0/ 90 (0%) [ 3] 10.0-11.0 sec 128 KBytes 1.05 Mbits/sec 0.275 ms 0/ 89 (0%) [ 3] 11.0-12.0 sec 128 KBytes 1.05 Mbits/sec 0.299 ms 0/ 89 (0%) [ 3] 12.0-13.0 sec 128 KBytes 1.05 Mbits/sec 0.327 ms 0/ 89 (0%) [ 3] 13.0-14.0 sec 128 KBytes 1.05 Mbits/sec 0.290 ms 0/ 89 (0%) [ 3] 14.0-15.0 sec 128 KBytes 1.05 Mbits/sec 0.251 ms 0/ 89 (0%) [ 3] 15.0-16.0 sec 128 KBytes 1.05 Mbits/sec 0.275 ms 0/ 89 (0%) [ 3] 16.0-17.0 sec 128 KBytes 1.05 Mbits/sec 0.303 ms 0/ 89 (0%) [ 3] 17.0-18.0 sec 128 KBytes 1.05 Mbits/sec 0.333 ms 0/ 89 (0%) [ 3] 18.0-19.0 sec 128 KBytes 1.05 Mbits/sec 0.294 ms 0/ 89 (0%) [ 3] 19.0-20.0 sec 131 KBytes 1.07 Mbits/sec 0.281 ms 0/ 91 (0%) [ 3] 0.0-20.0 sec 2.50 MBytes 1.05 Mbits/sec 0.305 ms 0/ 1785 (0%) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 5m ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 192.168.191.200, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.191.201 port 61760 connected with 192.168.191.200 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 1.0- 2.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 2.0- 3.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 3.0- 4.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 4.0- 5.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 5.0- 6.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 6.0- 7.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 7.0- 8.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 8.0- 9.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 9.0-10.0 sec 619 KBytes 5.07 Mbits/sec [ 4] 10.0-11.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 11.0-12.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 12.0-13.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 13.0-14.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 14.0-15.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 15.0-16.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 16.0-17.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 17.0-18.0 sec 610 KBytes 5.00 Mbits/sec [ 4] 18.0-19.0 sec 619 KBytes 5.07 Mbits/sec [ 4] 19.0-20.0 sec 609 KBytes 4.99 Mbits/sec [ 4] 0.0-20.0 sec 11.9 MBytes 5.00 Mbits/sec [ 4] Sent 8504 datagrams [ 4] Server Report: [ 4] 0.0-20.0 sec 11.9 MBytes 4.99 Mbits/sec 0.000 ms 12/ 8503 (0.14%) [ 4] 0.0-20.0 sec 1 datagrams received out-of-order [ 3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50750 [ 3] 0.0- 1.0 sec 606 KBytes 4.96 Mbits/sec 2.238 ms 1/ 423 (0.24%) [ 3] 1.0- 2.0 sec 610 KBytes 5.00 Mbits/sec 2.739 ms 0/ 425 (0%) [ 3] 2.0- 3.0 sec 609 KBytes 4.99 Mbits/sec 3.089 ms 1/ 425 (0.24%) [ 3] 3.0- 4.0 sec 609 KBytes 4.99 Mbits/sec 3.605 ms 0/ 424 (0%) [ 3] 4.0- 5.0 sec 607 KBytes 4.97 Mbits/sec 1.954 ms 0/ 423 (0%) [ 3] 5.0- 6.0 sec 612 KBytes 5.01 Mbits/sec 2.666 ms 0/ 426 (0%) [ 3] 6.0- 7.0 sec 607 KBytes 4.97 Mbits/sec 2.602 ms 0/ 423 (0%) [ 3] 7.0- 8.0 sec 612 KBytes 5.01 Mbits/sec 2.960 ms 0/ 426 (0%) [ 3] 8.0- 9.0 sec 609 KBytes 4.99 Mbits/sec 2.512 ms 0/ 424 (0%) [ 3] 9.0-10.0 sec 619 KBytes 5.07 Mbits/sec 2.133 ms 0/ 431 (0%) [ 3] 10.0-11.0 sec 609 KBytes 4.99 Mbits/sec 3.605 ms 1/ 425 (0.24%) [ 3] 11.0-12.0 sec 609 KBytes 4.99 Mbits/sec 2.509 ms 0/ 424 (0%) [ 3] 12.0-13.0 sec 610 KBytes 5.00 Mbits/sec 3.570 ms 0/ 425 (0%) [ 3] 13.0-14.0 sec 609 KBytes 4.99 Mbits/sec 3.077 ms 1/ 425 (0.24%) [ 3] 14.0-15.0 sec 609 KBytes 4.99 Mbits/sec 2.679 ms 0/ 424 (0%) [ 3] 15.0-16.0 sec 609 KBytes 4.99 Mbits/sec 1.887 ms 0/ 424 (0%) [ 3] 16.0-17.0 sec 610 KBytes 5.00 Mbits/sec 2.651 ms 0/ 425 (0%) [ 3] 17.0-18.0 sec 609 KBytes 4.99 Mbits/sec 3.390 ms 0/ 424 (0%) [ 3] 18.0-19.0 sec 617 KBytes 5.06 Mbits/sec 2.601 ms 0/ 430 (0%) [ 3] 19.0-20.0 sec 612 KBytes 5.01 Mbits/sec 3.525 ms 0/ 426 (0%) [ 3] 0.0-20.0 sec 11.9 MBytes 4.99 Mbits/sec 3.156 ms 3/ 8503 (0.035%) [ 3] 0.0-20.0 sec 1 datagrams received out-of-order ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 9m ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 192.168.191.200, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.191.201 port 61761 connected with 192.168.191.200 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 1.0- 2.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 2.0- 3.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 3.0- 4.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 4.0- 5.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 5.0- 6.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 6.0- 7.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 7.0- 8.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 8.0- 9.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 9.0-10.0 sec 1.09 MBytes 9.14 Mbits/sec [ 4] 10.0-11.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 11.0-12.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 12.0-13.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 13.0-14.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 14.0-15.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 15.0-16.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 16.0-17.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 17.0-18.0 sec 1.07 MBytes 8.98 Mbits/sec [ 4] 18.0-19.0 sec 1.09 MBytes 9.14 Mbits/sec [ 4] 19.0-20.0 sec 1.07 MBytes 9.00 Mbits/sec [ 4] 0.0-20.0 sec 21.5 MBytes 9.00 Mbits/sec [ 4] Sent 15315 datagrams [ 4] Server Report: [ 4] 0.0-20.0 sec 21.3 MBytes 8.94 Mbits/sec 0.104 ms 96/15314 (0.63%) !!!!!!!!!! [ 4] 0.0-20.0 sec 1 datagrams received out-of-order [ 3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50751 [ 3] 0.0- 1.0 sec 1.06 MBytes 8.89 Mbits/sec 2.405 ms 0/ 756 (0%) [ 3] 1.0- 2.0 sec 1.07 MBytes 9.00 Mbits/sec 2.308 ms 0/ 765 (0%) [ 3] 2.0- 3.0 sec 1.07 MBytes 9.00 Mbits/sec 2.305 ms 0/ 765 (0%) [ 3] 3.0- 4.0 sec 1.07 MBytes 8.97 Mbits/sec 2.290 ms 1/ 764 (0.13%) [ 3] 4.0- 5.0 sec 1.07 MBytes 8.98 Mbits/sec 2.271 ms 1/ 765 (0.13%) [ 3] 5.0- 6.0 sec 1.07 MBytes 8.98 Mbits/sec 2.313 ms 0/ 764 (0%) [ 3] 6.0- 7.0 sec 1.07 MBytes 9.00 Mbits/sec 2.191 ms 0/ 765 (0%) [ 3] 7.0- 8.0 sec 1.07 MBytes 8.95 Mbits/sec 2.314 ms 3/ 764 (0.39%) [ 3] 8.0- 9.0 sec 1.07 MBytes 8.98 Mbits/sec 2.232 ms 1/ 765 (0.13%) [ 3] 9.0-10.0 sec 1.09 MBytes 9.13 Mbits/sec 2.257 ms 0/ 776 (0%) [ 3] 10.0-11.0 sec 1.07 MBytes 8.98 Mbits/sec 2.365 ms 0/ 764 (0%) [ 3] 11.0-12.0 sec 1.07 MBytes 8.98 Mbits/sec 2.301 ms 1/ 765 (0.13%) [ 3] 12.0-13.0 sec 1.07 MBytes 8.98 Mbits/sec 2.277 ms 0/ 764 (0%) [ 3] 13.0-14.0 sec 1.07 MBytes 9.00 Mbits/sec 2.323 ms 0/ 765 (0%) [ 3] 14.0-15.0 sec 1.07 MBytes 9.00 Mbits/sec 2.176 ms 0/ 765 (0%) [ 3] 15.0-16.0 sec 1.07 MBytes 8.96 Mbits/sec 2.273 ms 2/ 764 (0.26%) [ 3] 16.0-17.0 sec 1.07 MBytes 8.98 Mbits/sec 2.313 ms 0/ 764 (0%) [ 3] 17.0-18.0 sec 1.07 MBytes 8.98 Mbits/sec 2.247 ms 1/ 765 (0.13%) [ 3] 18.0-19.0 sec 1.09 MBytes 9.11 Mbits/sec 2.276 ms 1/ 776 (0.13%) [ 3] 19.0-20.0 sec 1.07 MBytes 8.97 Mbits/sec 2.394 ms 1/ 764 (0.13%) [ 3] 0.0-20.0 sec 21.5 MBytes 8.99 Mbits/sec 2.659 ms 11/15314 (0.072%) [ 3] 0.0-20.0 sec 1 datagrams received out-of-order ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ C:\iperf-2.0.5-2-win32>iperf -c 192.168.191.200 -i 1 -t 20 -r -u -b 9850k ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ ------------------------------------------------------------ Server listening on UDP port 5001 Receiving 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ ------------------------------------------------------------ Client connecting to 192.168.191.200, UDP port 5001 Sending 1470 byte datagrams UDP buffer size: 64.0 KByte (default) ------------------------------------------------------------ [ 4] local 192.168.191.201 port 61762 connected with 192.168.191.200 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0- 1.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 1.0- 2.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 2.0- 3.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 3.0- 4.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 4.0- 5.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 5.0- 6.0 sec 1.17 MBytes 9.83 Mbits/sec [ 4] 6.0- 7.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 7.0- 8.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 8.0- 9.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 9.0-10.0 sec 1.19 MBytes 10.0 Mbits/sec [ 4] 10.0-11.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 11.0-12.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 12.0-13.0 sec 1.17 MBytes 9.83 Mbits/sec [ 4] 13.0-14.0 sec 1.17 MBytes 9.85 Mbits/sec [ 4] 14.0-15.0 sec 1.17 MBytes 9.83 Mbits/sec [ 4] 15.0-16.0 sec 1.17 MBytes 9.85 Mbits/sec [ 4] 16.0-17.0 sec 1.17 MBytes 9.83 Mbits/sec [ 4] 17.0-18.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 18.0-19.0 sec 1.19 MBytes 10.0 Mbits/sec [ 4] 19.0-20.0 sec 1.17 MBytes 9.84 Mbits/sec [ 4] 0.0-20.0 sec 23.5 MBytes 9.85 Mbits/sec [ 4] Sent 16765 datagrams [ 4] Server Report: [ 4] 0.0-20.0 sec 23.3 MBytes 9.74 Mbits/sec 3.421 ms 156/16764 (0.93%) !!!!!!!!!! [ 4] 0.0-20.0 sec 1 datagrams received out-of-order [ 3] local 192.168.191.201 port 5001 connected with 192.168.191.200 port 50752 [ 3] 0.0- 1.0 sec 1.16 MBytes 9.74 Mbits/sec 2.131 ms 0/ 828 (0%) [ 3] 1.0- 2.0 sec 1.17 MBytes 9.84 Mbits/sec 2.140 ms 0/ 837 (0%) [ 3] 2.0- 3.0 sec 1.17 MBytes 9.83 Mbits/sec 2.099 ms 1/ 837 (0.12%) [ 3] 3.0- 4.0 sec 1.17 MBytes 9.84 Mbits/sec 2.113 ms 0/ 837 (0%) [ 3] 4.0- 5.0 sec 1.17 MBytes 9.84 Mbits/sec 2.105 ms 0/ 837 (0%) [ 3] 5.0- 6.0 sec 1.17 MBytes 9.83 Mbits/sec 2.058 ms 1/ 837 (0.12%) [ 3] 6.0- 7.0 sec 1.17 MBytes 9.82 Mbits/sec 2.165 ms 1/ 836 (0.12%) [ 3] 7.0- 8.0 sec 1.17 MBytes 9.84 Mbits/sec 2.156 ms 0/ 837 (0%) [ 3] 8.0- 9.0 sec 1.17 MBytes 9.82 Mbits/sec 2.135 ms 2/ 837 (0.24%) [ 3] 9.0-10.0 sec 1.19 MBytes 9.97 Mbits/sec 2.152 ms 2/ 850 (0.24%) [ 3] 10.0-11.0 sec 1.17 MBytes 9.83 Mbits/sec 2.153 ms 1/ 837 (0.12%) [ 3] 11.0-12.0 sec 1.17 MBytes 9.84 Mbits/sec 2.127 ms 0/ 837 (0%) [ 3] 12.0-13.0 sec 1.17 MBytes 9.83 Mbits/sec 2.136 ms 1/ 837 (0.12%) [ 3] 13.0-14.0 sec 1.17 MBytes 9.82 Mbits/sec 2.087 ms 2/ 837 (0.24%) [ 3] 14.0-15.0 sec 1.17 MBytes 9.83 Mbits/sec 2.061 ms 1/ 837 (0.12%) [ 3] 15.0-16.0 sec 1.17 MBytes 9.84 Mbits/sec 2.045 ms 0/ 837 (0%) [ 3] 16.0-17.0 sec 1.17 MBytes 9.82 Mbits/sec 2.203 ms 1/ 836 (0.12%) [ 3] 17.0-18.0 sec 1.17 MBytes 9.84 Mbits/sec 2.165 ms 0/ 837 (0%) [ 3] 18.0-19.0 sec 1.17 MBytes 9.83 Mbits/sec 2.154 ms 1/ 837 (0.12%) [ 3] 19.0-20.0 sec 1.19 MBytes 9.98 Mbits/sec 2.209 ms 0/ 849 (0%) [ 3] 0.0-20.0 sec 23.5 MBytes 9.84 Mbits/sec 2.548 ms 13/16764 (0.078%) [ 3] 0.0-20.0 sec 1 datagrams received out-of-order
The real question remains
We are not oversubscribing the DC link since it's at 100Mbps and cannot send more than 100Mbps. However the remote sites are at 10Mbps.
- Are the buffers at the remote side overflowing and dropping packets?
- Is the provider's traffic shaper doing something to the traffic? (Would the traffic coming from another node be influenced by the ISPs traffic shaper or only the traffic ingressing the node (from outside))...... You see what I mean?
Why can't TCP handle that all on its own?
Update #3
I tried the following scenario:
Laptop ------- ... LAN ... --- DC switch --- Metro-Eth --- Laptop (directly connected) NIC@10Mbps 100Mbps NIC@10Mbps Here is the packet loss in the DC->remote direction: (iperf 9 Mbps UDP test) [ 3] local 192.168.191.200 port 5001 connected with 192.168.191.201 port 55236 [ ID] Interval Transfer Bandwidth Jitter Lost/Total Datagrams [ 3] 0.0- 1.0 sec 912 KBytes 7.47 Mbits/sec 2.713 ms 0/ 635 (0%) [ 3] 1.0- 2.0 sec 1001 KBytes 8.20 Mbits/sec 2.168 ms 0/ 697 (0%) [ 3] 2.0- 3.0 sec 1001 KBytes 8.20 Mbits/sec 2.478 ms 0/ 697 (0%) [ 3] 3.0- 4.0 sec 999 KBytes 8.18 Mbits/sec 0.933 ms 0/ 696 (0%) [ 3] 4.0- 5.0 sec 1001 KBytes 8.20 Mbits/sec 2.620 ms 0/ 697 (0%) [ 3] 5.0- 6.0 sec 1001 KBytes 8.20 Mbits/sec 2.721 ms 0/ 697 (0%) [ 3] 6.0- 7.0 sec 1001 KBytes 8.20 Mbits/sec 2.089 ms 0/ 697 (0%) [ 3] 7.0- 8.0 sec 999 KBytes 8.18 Mbits/sec 2.641 ms 0/ 696 (0%) [ 3] 8.0- 9.0 sec 1002 KBytes 8.21 Mbits/sec 0.896 ms 0/ 698 (0%) [ 3] 9.0-10.0 sec 1015 KBytes 8.31 Mbits/sec 2.557 ms 0/ 707 (0%) [ 3] 10.0-11.0 sec 999 KBytes 8.18 Mbits/sec 2.822 ms 1/ 697 (0.14%) [ 3] 11.0-12.0 sec 999 KBytes 8.18 Mbits/sec 1.551 ms 1/ 697 (0.14%) [ 3] 12.0-13.0 sec 998 KBytes 8.17 Mbits/sec 2.504 ms 2/ 697 (0.29%) [ 3] 13.0-14.0 sec 995 KBytes 8.15 Mbits/sec 2.038 ms 3/ 696 (0.43%) [ 3] 14.0-15.0 sec 991 KBytes 8.11 Mbits/sec 2.539 ms 7/ 697 (1%) [ 3] 15.0-16.0 sec 992 KBytes 8.13 Mbits/sec 2.759 ms 6/ 697 (0.86%) [ 3] 16.0-17.0 sec 998 KBytes 8.17 Mbits/sec 2.229 ms 2/ 697 (0.29%) [ 3] 17.0-18.0 sec 993 KBytes 8.14 Mbits/sec 2.723 ms 4/ 696 (0.57%) [ 3] 18.0-19.0 sec 998 KBytes 8.17 Mbits/sec 2.038 ms 2/ 697 (0.29%) [ 3] 19.0-20.0 sec 1012 KBytes 8.29 Mbits/sec 2.575 ms 3/ 708 (0.42%) [ 3] 0.0-20.0 sec 19.5 MBytes 8.15 Mbits/sec 2.775 ms 31/13917 (0.22%) [ 3] 0.0-20.0 sec 1 datagrams received out-of-order
The other direction is fine. However, when running a TCP test the remote->DC direction does not perform much better than the DC->remote direction (about 5Mbps)...
I'm not sure we reached the bottom of this.
sysctl
not sure about about Windows... maybenetsh
. If I was going to take a guess for what is wrong with your circuit I would say that the CPE at the spoke site is setup with a larger CBS than the hub side... which is usually the other way around. Again, the JDSU will punt the ball back to them or let you refocus on what the issue is.