High latency during downloads

Question

I am having the same issue on my business connection 5Mbps as in another posting on this site. As soon as any computer starts a download the latency on the first hop past our DFG provided by our ISP (Bell) goes off the chart. This first hop is likely in our same building and is 1ms constantly, start a download, eg windows update, and it jumps to 200-1000ms.

I have spent hours on the phone with support all saying you have reach the max available bandwidth, it is normal for your latency to spike. But my reading tells me they are breaking something with TCP. I have run tests on a home Shaw connection and even on a Rogers LTE running downloads and reaching the max Mbps for my account but the latency does not go through the roof.

Am I right in my understanding that Bell is doing something to break TCP's built-in technologies to manage its rate based on the available bandwidth between the 2 end points?

Answer 1

Bell is telling you the truth. When you try to push 5Mbps (or more) into a 5Mbps connection, everything files into a neat little order (read: queue.) Your ping goes out without delay because there's no backlog. The reply, however, is now at the end of the queue. TCP is doing exactly what it's supposed to here -- the sender is filling the allowed receive window.

There are things you can do on your side of the line (QoS, WRED, etc.) to help reduce the effects, but this is the sort of thing you're going to see when there's a large difference between sender and receiver bandwidth. I've lived with it for years (T1, 6Mbps DS3, even 10Mbps cablemodem) You could ask the ISP to reduce the queue size on their side, but they're not likely to do it, as it'll result in packet drops.

Answer 2

Most forms of "QoS" these days do not include AQM as vendors found it too hard to configure RED automagically without doing harm. This leads to the horrendous delays you see on many common devices today, notably cable modems and wireless. So merely recommending "turning qos on"... doesn't help. In fact on at least one of Netgear's products, turning on the rate limiter for "QoS" leads to vastly worse results....

Recently a new fair queueing + AQM algorithm has appear that does appear to work extremely well, and better, requires almost no configuration besides setting the rate limiter. It's called fq_codel, and it is now widely available in most Linux's and has been ported to BSD as well. It's part of the default "QoS" in openwrt barrier breaker, cerowrt, and gargoyle uses a (pretty good) earlier version called sfqred with an innovative automatic rate adjusting scheme called ACC.

So you can slam a box based on this in front of your misbehaving link, turn on their QoS rate limiter (set just slightly below your providers inbound and outbound settings so you take control) + fq_codel, and get much better performance for everybody using it. I mean astoundingly better: see the ietf demo below, the report to the iccrg working group at the ietf, etc.

For way more detail on the bufferbloat problem and the fixes for it, see:

http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos

We are (of course) trying to convince various ISP CPE vendors to pay attention, as is cablelabs, which published a wonderful study of this new stuff a few months back, which also contains some detail on the current misbehavior of cable modems in particular.

http://www.cablelabs.com/downloads/pubs/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf

Answer 3

What you are seeing is entirely typical. Many service providers will rate limit and/or use a QoS mechanism to lower the priority of ICMP (which includes traditional ping and traceroute) as it has been used in denial of service attacks at times.

While a link isn't congested, the lowered priority doesn't affect anything as no traffic is being queued. During these periods, your latency remains low because the ICMP packets will be forwarded immediately and not be delayed at all.

When the link is congested, higher priority queues get more attention. Depending on the queuing mechanism, it might forward multiple packets from a higher priority queue for each packet from a lower priority queue or even forward only when there is nothing in a higher priority queue. In any case, a packet relegated to a lower priority queue will generally be held back longer than on a link with no congestion, increasing the latency.

Answer 4

You're probably suffering from bufferbloat and you want AQM (Active Queue Management). I have written a script for Linux which makes this pretty easy:

#!/bin/bash
# Traffic shaping script (AQM, fq_codel+tbf)
# Copyright 2018 Mikko Rantalainen <[email protected]>
# License: MIT (X11)
# Usage:
#   21/0.8 Mbps connection (ADSL2): DOWNLINK_RATE=21.7Mbit UPLINK_RATE=0.8Mbit TBF_LATENCY=500ms bin/traffic-shaping start
#   100/100 Mbps connection: ./traffic-shaping
#   1/1 GBps connection: DOWNLINK_RATE=1Gbit UPLINK_RATE=1Gbit TBF_LATENCY=10ms bin/traffic-shaping start
# Note that using low TBF_LATENCY will require powerful CPU.
#   

set -e

DEV="${DEV:=$(ip route | grep "^default " | grep -Po "(?<=dev )[^ ]+")}"

# ingress:
DOWNLINK_RATE="${DOWNLINK_RATE:=104000kbit}" # or e.g. "21.5Mbit"
# egress:
UPLINK_RATE="${UPLINK_RATE:=105000kbit}"

CODEL_INTERVAL="${CODEL_INTERVAL:=100ms}" # usually 100ms, high speed links with low latency may need lower values
CODEL_TARGET="${CODEL_TARGET:=5ms}" # unit "us" is also available, usually 5%-10% of CODEL_INTERVAL
CODEL_LIMIT="${CODEL_LIMIT:=1001}" # decrease to reduce latency, too low values will limit throughput
CODEL_FLOWS="${CODEL_FLOWS:=1024}"

# set burst as high as possible without causing dropped packets at the start of the connections
DOWNLINK_BURST="${DOWNLINK_BURST:=6500}"
UPLINK_BURST="${UPLINK_BURST:=6500}"

TBF_LATENCY="${TBF_LATENCY:=14ms}" # set to lower latency to improve control over bandwidth limiting, UPLINK_BURST bytes must be able to be sent in this time

IFB="$DEV.ingress"

INITCWND="${INITCWND:=20}"
INITRWND="${INITRWND:=20}"

configure_shaping()
{
    # EGRESS (outgoing traffic, "uploads"):

    # setup bandwidth limiting:
    tc qdisc add dev "$DEV" root handle 1: tbf rate "$UPLINK_RATE" burst "$UPLINK_BURST" latency "$TBF_LATENCY"

    # setup fq_codel for bandwidth shaping
    tc qdisc add dev "$DEV" parent 1: fq_codel limit "$CODEL_LIMIT" target "$CODEL_TARGET" interval "$CODEL_INTERVAL" flows "$CODEL_FLOWS" noecn


    # INGRESS (incoming traffic, "downloads"):

    # setup bandwidth limiting (ingress limiting needs IFB or Intermediate Functional Block, see https://wiki.linuxfoundation.org/networking/ifb):
    tc qdisc add dev "$DEV" handle ffff: ingress
    ip link add name "$IFB" type ifb
    tc qdisc add dev "$IFB" root handle 1: tbf rate "$DOWNLINK_RATE" burst "$DOWNLINK_BURST" latency "$TBF_LATENCY"

    # setup fq_codel for bandwidth shaping
    tc qdisc add dev "$IFB" parent 1: fq_codel limit "$CODEL_LIMIT" target "$CODEL_TARGET" interval "$CODEL_INTERVAL" flows "$CODEL_FLOWS" ecn
    ip link set dev "$IFB" up

    # connect ingress filtering to actual WAN device
    tc filter add dev "$DEV" parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev "$IFB"

    # configure initcwnd and initrwnd
    ip route change $(ip route | grep ^default) initcwnd "$INITCWND" initrwnd "$INITRWND"
}

remove_shaping()
{
    tc qdisc list | grep -q "ingress" && tc qdisc del dev "$DEV" ingress || true
    tc qdisc list | grep -q "codel" && tc qdisc del dev "$DEV" root || true
    ip link show | grep -q "$IFB" && ip link del "$IFB" || true
}

status()
{
        echo "─── queue discipline configuration: ──────────────────"
        tc qdisc list
        echo "   TIP: use e.g. 'sudo tc qdisc del dev $DEV ingress' to remove ingress filtering"
        echo "   TIP: use e.g. 'sudo tc qdisc del dev $DEV root' to remove egress filtering"
        echo "─── ip link show: ────────────────────────────────────"
        ip link show
        echo "   TIP: use e.g. 'sudo ip link del $IFB' to remove ingress device"
}

color_status()
{
    status | grep --color=auto -E "^|$DEV|$IFB|rate [^ ]+"
}

# handle parameters

ACTION="$1"
shift || true

while [ ! -z "$1" ]
do
    case "$1" in
        -v|--verbose)
            echo "Device: $DEV"
            echo "Downlink rate (ingress): $DOWNLINK_RATE"
            echo "Uplink rate (egress): $UPLINK_RATE"
            set -x
            ;;
        *)
            if [ ! -z "$2" ]; then
                echo "Unknown parameter: '$2'" 1>&2
                exit 1
            fi
            ;;
    esac
    shift
done

case "$ACTION" in
    start)
        remove_shaping
        configure_shaping
        ;;
    stop)
        remove_shaping
        ;;
    status)
        color_status
        ;;
    restart)
        remove_shaping
        configure_shaping
        ;;
    *)
        echo "Unknown action: $1" 1>&2
        echo "Usage: $0 <start|stop|restart|status> [--verbose|-v]" 1>&2
        exit 1
esac

You simply save the script as traffic-shaping and chmod a+x it and run it as root (after reading the source code, obviously).

For your use case, I'd suggest

DOWNLINK_RATE=5.0Mbit UPLINK_RATE=5.0Mbit TBF_LATENCY=500ms ./traffic-shaping start