Random intercontinental TCP streams are slow

Question

I have a home server in New Zealand and a remote server in Amsterdam of the following configuration:

I have a simple test script:

#!/bin/bash

for i in `seq 1 20`;
do
    scp root@remoteserver:~/testfile.txt .
done

Example output:

testfile.txt                                  100%   10MB 764.9KB/s   00:13
testfile.txt                                  100%   10MB 810.6KB/s   00:12
testfile.txt                                  100%   10MB 810.5KB/s   00:12
testfile.txt                                   48% 4960KB  11.6KB/s   07:36 ETA

testfile.txt is 10MB of random data. It seems that approximately 1/4 of the transfers will be locked in at around 10KB/s. The rest will average around 800KB/s. Now, I certainly don't expect to saturate a 1 Gigabit pipe with a 300ms RTT (and an untuned TCP stack), so the 800KB/s cap is to be expected. What isn't expected is the random streams that complete at 10KB/s. Can anyone explain this behaviour and perhaps suggest a solution?

A few bits of extra context:

• The local and remote server are both running Ubuntu. I have tried different versions to no avail.
• This has only been a problem for the last couple of weeks. Prior to that, I never encountered such strange behaviour.
• Either the RTT or the length of the link seem to have an impact on this. The same behaviour is not witnessed on a server operating out of Australia (i.e. ~60ms RTT).
• Switching to a different European country doesn't seem to make a difference (i.e. data centers in France and Germany have the same problem).

I'm asking on Network Engineering because I believe this may come down to the infrastructure of the internet or my network as opposed to some configuration issue at either endpoint.

Answer 1

One of my clients has traffic between UK, Ireland, and Singapore; frequently there are spectacular slowdowns of the kind you describe. My understanding is that a lot of traffic between Europe and Australian and New Zealand goes through Singapore, and there can be considerable congestion there. (Even if not the case for you, you may have analogous issues.)

1. Routing: One problem was caused by poor routing of the UK site. Routes transitting through NTT were taking approx 190 ms RTT and with relatively low variation, through PCCW were taking 330 ms with very high variation up to about 1000 ms. Additionally there apparent bandwidth limitations. (For comparison, AWS Singapore to AWS London 165 ms; theoretical minimum 105 ms.) Only sometimes was that related to this client's traffic. Attached graph for illustration (measuring from AWS Singapore to client's site in UK). Suggestion: log RTT and look for patterns, examine route (with traceroute) carefully

2. Server throttling: Another problem has to do with the servers in Europe. Some rented virtual private servers have limits which are triggered by usage. I've certainly seen very severe throttling on very cheap VPS, on network traffic, on CPU usage, and on "disk" throughput. Suggestion: repeat tests on expensive VPS or physical hardware.

Perhaps these suggestions will give you a direction to try.