0

I'm troubleshooting a situation where clients in far off locations with high latency connections to our network are experiencing severely adverse network transfer speeds.

If I run a speed test from our gear in Seattle to a speed test server in Seattle, the performance is nearly full gigabit:

[root@sharedsea1 ~]# speedtest -s 5904
   Speedtest by Ookla
     Server: Metapeer - Seattle, WA (id = 5904)
    Latency:     0.76 ms   (0.03 ms jitter)
   Download:   928.12 Mbps (data used: 419.7 MB)
     Upload:   925.95 Mbps (data used: 417.0 MB)
Packet Loss:     0.0%
[root@sharedsea1 ~]#

However, if I run the same test to Hong Kong, the transfer rate drops more than you'd expect to see:

[root@sharedsea1 ~]# speedtest -s 1536
   Speedtest by Ookla
     Server: STC - Hong Kong (id = 1536)
    Latency:   157.25 ms   (0.07 ms jitter)
   Download:   323.54 Mbps (data used: 451.4 MB)
     Upload:   269.72 Mbps (data used: 434.4 MB)
Packet Loss: Not available.
[root@sharedsea1 ~]#

And if I run the test again to an even further location (e.g., India), the transfer rate just becomes terrible:

[root@sharedsea1 ~]# speedtest -s 18976
   Speedtest by Ookla
     Server: Airtel Broadband - Bangalore (id = 18976)
    Latency:   252.47 ms   (0.14 ms jitter)
   Download:    35.78 Mbps (data used: 48.6 MB)
     Upload:   217.25 Mbps (data used: 338.7 MB)
Packet Loss: Not available.
[root@sharedsea1 ~]#

We BGP peer with two different upstream carriers. We've tried depeering from each carrier one at a time to test speeds in isolation. The poor speeds occur with both of them. And both ISPs have provided speed tests to me showing vastly better transfer rates just one hop away from our network. For example:

Speedtest by Ookla
 Server: STC - Hong Kong (id = 1536)
Latency:   167.73 ms   (0.06 ms jitter)
Download: 941.36 Mbps (data used: 1.4 GB)
Upload: 366.29 Mbps (data used: 570.3 MB)
Packet Loss: Not available.

Lastly, I tried to eliminate the hardware as a possible factor. We have two Brocade routers at our network edge each with a 10G connection to an upstream carrier. I ran the speed test from a server connected to one of those edge routers and received the same poor performance. I then ran the same speed test from another server connected to the other edge router -- same poor result.

We have no congestion on our network -- utilization on our 10G ports at the network edge are close to 0.5% and the utilization on the 1G ports on our test servers is 0% (i.e., only used for testing). There is also no packet loss between our network and the various test servers.

Does anyone have any ideas for what might cause low speeds for high latency connections but have no impact on speeds on low latency connections?

Improve this question
4
  • Google Delay-bandwidth product
    – Ron Trunk
    Dec 3, 2020 at 21:54
  • @RonTrunk I'm aware that high latency connections in general should receive far slower speeds than low latency connections. However, in this case, bandwidth delay product doesn't seem to explain the root cause. My upstream carriers, just a couple hops out from my network (~10ms difference in latency), are able to achieve vastly better speeds against the same speed test servers.
    – Elliot B.
    Dec 3, 2020 at 21:56
  • First, do not put much faith in the speed test servers. They are mostly to mollify home users as they are on networks run by ISPs, so the ISP can point the home users to a server on their own network. Next, we really cannot help with Internet problems, as the Internet is a bunch of networks not under your direct control, and one requirement for questions here is that the question must be about a network under your direct control.
    – Ron Maupin ♦
    Dec 3, 2020 at 23:00
  • If you want help on your network, we need more information, such as a good network description or diagram, the network device models, and the network device configurations. You can refer to the Network Engineering Question Checklist for guidance, then edit your question.
    – Ron Maupin ♦
    Dec 3, 2020 at 23:02

1 Answer 1

Reset to default
1

High latency directly impacts throughput when the TCP send window doesn't scale up to what is required to "fill the pipe": bandwidth * RTT (the delay-bandwidth product). That very quickly requires the window scaling option which is widely supported today.

However, high latency (especially with considerable jitter) can also be a sign of congestion. If there's a bottleneck on the path, throughput will obviously be lower than expected (depending on time of day, for instance).

Also, congestion increases packet loss, triggering TCP's congestion detection and reducing the effictive window size.

Effectively, there's very little you can do short of looking for a better carrier with better peering (which might not even exist).

Improve this answer
1
  • Thank you for the answer -- very helpful information. In our tests jitter is low, no packet loss and just two hops out of our network our upstream carriers are able to achieve far faster results with effectively the same level of latency. I understand that high latency reduces speeds, but what we're seeing seems very severe, far slower than one would expect.
    – Elliot B.
    Dec 3, 2020 at 22:19

Not the answer you're looking for? Browse other questions tagged or ask your own question.