9

I am having the same issue on my business connection 5Mbps as in another posting on this site. As soon as any computer starts a download the latency on the first hop past our DFG provided by our ISP (Bell) goes off the chart. This first hop is likely in our same building and is 1ms constantly, start a download, eg windows update, and it jumps to 200-1000ms.

I have spent hours on the phone with support all saying you have reach the max available bandwidth, it is normal for your latency to spike. But my reading tells me they are breaking something with TCP. I have run tests on a home Shaw connection and even on a Rogers LTE running downloads and reaching the max Mbps for my account but the latency does not go through the roof.

Am I right in my understanding that Bell is doing something to break TCP's built-in technologies to manage its rate based on the available bandwidth between the 2 end points?

2
  • I'd guess a simple ready made solution would be to purchase Edgerouter X ui.com/edgemax/edgerouter-x and configure adaptive queue management for 4.9 Mbps. That should get rid of the bufferbloat as long as ALL your connections to internet go through this box. If I remember correctly, Edgerouter X is able to handle adaptive queue management up to 130 Mbps level. If you need adaptive queue management with higher limits, you need a better box. Aug 19, 2020 at 10:35
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer.
    – Ron Maupin
    Jan 3, 2021 at 5:41

4 Answers 4

12

Bell is telling you the truth. When you try to push 5Mbps (or more) into a 5Mbps connection, everything files into a neat little order (read: queue.) Your ping goes out without delay because there's no backlog. The reply, however, is now at the end of the queue. TCP is doing exactly what it's supposed to here -- the sender is filling the allowed receive window.

There are things you can do on your side of the line (QoS, WRED, etc.) to help reduce the effects, but this is the sort of thing you're going to see when there's a large difference between sender and receiver bandwidth. I've lived with it for years (T1, 6Mbps DS3, even 10Mbps cablemodem) You could ask the ISP to reduce the queue size on their side, but they're not likely to do it, as it'll result in packet drops.

3
  • 5
    200-1000ms (85-420 packets, 1500B@5Mbps) is in en.wikipedia.org/wiki/Bufferbloat domain, as TCP depends on packet loss occurring to correctly and rapidly set window size, it should be net win to reduce it to maybe 10 packets (25ms). I fully agree that operator is unlikely to change this in their product unless lot of customers complain, likely easier to just order QoS product to the business connection, it should have saner buffer values and they should be orderable to customer demands. Interestingly Google's QUIC optionally can slow down transmit rate when latency starts going up.
    – ytti
    Jul 24, 2013 at 7:08
  • Thanks Ricky, I hear what you are saying but after more reading, shouldnt TCP's Flow Control see the backlog and adjust the window to the rate the receiver can handle? Thus not overloading the client or router(s) (hop 2 on Bells network) along the way? To me it seems like your reference to bufferbloat that I have also read which describes the scenario exactly.
    – SA-KRP
    Jul 24, 2013 at 18:35
  • 3
    TCP cannot detect any bottleneck without packet drops (or ECN.) If the router queues are deep enough and your receive window is large enough, you can create a huge backlog. RFC1323 timestamps might help, but I've seen significant problems allowing windows to "use" TS. (it attempts to "negotiate" TS by sending an initial TS=0)
    – Ricky
    Jul 24, 2013 at 18:57
5

Most forms of "QoS" these days do not include AQM as vendors found it too hard to configure RED automagically without doing harm. This leads to the horrendous delays you see on many common devices today, notably cable modems and wireless. So merely recommending "turning qos on"... doesn't help. In fact on at least one of Netgear's products, turning on the rate limiter for "QoS" leads to vastly worse results....

Recently a new fair queueing + AQM algorithm has appear that does appear to work extremely well, and better, requires almost no configuration besides setting the rate limiter. It's called fq_codel, and it is now widely available in most Linux's and has been ported to BSD as well. It's part of the default "QoS" in openwrt barrier breaker, cerowrt, and gargoyle uses a (pretty good) earlier version called sfqred with an innovative automatic rate adjusting scheme called ACC.

So you can slam a box based on this in front of your misbehaving link, turn on their QoS rate limiter (set just slightly below your providers inbound and outbound settings so you take control) + fq_codel, and get much better performance for everybody using it. I mean astoundingly better: see the ietf demo below, the report to the iccrg working group at the ietf, etc.

For way more detail on the bufferbloat problem and the fixes for it, see:

http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos

We are (of course) trying to convince various ISP CPE vendors to pay attention, as is cablelabs, which published a wonderful study of this new stuff a few months back, which also contains some detail on the current misbehavior of cable modems in particular.

http://www.cablelabs.com/downloads/pubs/Active_Queue_Management_Algorithms_DOCSIS_3_0.pdf

1

What you are seeing is entirely typical. Many service providers will rate limit and/or use a QoS mechanism to lower the priority of ICMP (which includes traditional ping and traceroute) as it has been used in denial of service attacks at times.

While a link isn't congested, the lowered priority doesn't affect anything as no traffic is being queued. During these periods, your latency remains low because the ICMP packets will be forwarded immediately and not be delayed at all.

When the link is congested, higher priority queues get more attention. Depending on the queuing mechanism, it might forward multiple packets from a higher priority queue for each packet from a lower priority queue or even forward only when there is nothing in a higher priority queue. In any case, a packet relegated to a lower priority queue will generally be held back longer than on a link with no congestion, increasing the latency.

1
  • 1
    Thanks YLearn for your reply. I do get the priority of ICMP but we can see other traffic affected and ICMP was just to illustrate the issue. As I was trying convey to Ricky, in my comment is Flow Control is why TCP works, as any sender with a higher bandwidth than the receiver would take him offline DOS if Flow Control was not working right. Thats why a dial-up can communicate with a 1000Mbps connection? Am I wrong thinking, if things are running to proper standards latency during a file transfer maintain a resonalble level and not go through the roof?
    – SA-KRP
    Jul 24, 2013 at 18:50
1

You're probably suffering from bufferbloat and you want AQM (Active Queue Management). I have written a script for Linux which makes this pretty easy:

#!/bin/bash
# Traffic shaping script (AQM, fq_codel+tbf)
# Copyright 2018 Mikko Rantalainen <[email protected]>
# License: MIT (X11)
# Usage:
#   21/0.8 Mbps connection (ADSL2): DOWNLINK_RATE=21.7Mbit UPLINK_RATE=0.8Mbit TBF_LATENCY=500ms bin/traffic-shaping start
#   100/100 Mbps connection: ./traffic-shaping
#   1/1 GBps connection: DOWNLINK_RATE=1Gbit UPLINK_RATE=1Gbit TBF_LATENCY=10ms bin/traffic-shaping start
# Note that using low TBF_LATENCY will require powerful CPU.
#   

set -e

DEV="${DEV:=$(ip route | grep "^default " | grep -Po "(?<=dev )[^ ]+")}"

# ingress:
DOWNLINK_RATE="${DOWNLINK_RATE:=104000kbit}" # or e.g. "21.5Mbit"
# egress:
UPLINK_RATE="${UPLINK_RATE:=105000kbit}"

CODEL_INTERVAL="${CODEL_INTERVAL:=100ms}" # usually 100ms, high speed links with low latency may need lower values
CODEL_TARGET="${CODEL_TARGET:=5ms}" # unit "us" is also available, usually 5%-10% of CODEL_INTERVAL
CODEL_LIMIT="${CODEL_LIMIT:=1001}" # decrease to reduce latency, too low values will limit throughput
CODEL_FLOWS="${CODEL_FLOWS:=1024}"

# set burst as high as possible without causing dropped packets at the start of the connections
DOWNLINK_BURST="${DOWNLINK_BURST:=6500}"
UPLINK_BURST="${UPLINK_BURST:=6500}"

TBF_LATENCY="${TBF_LATENCY:=14ms}" # set to lower latency to improve control over bandwidth limiting, UPLINK_BURST bytes must be able to be sent in this time

IFB="$DEV.ingress"

INITCWND="${INITCWND:=20}"
INITRWND="${INITRWND:=20}"

configure_shaping()
{
    # EGRESS (outgoing traffic, "uploads"):

    # setup bandwidth limiting:
    tc qdisc add dev "$DEV" root handle 1: tbf rate "$UPLINK_RATE" burst "$UPLINK_BURST" latency "$TBF_LATENCY"

    # setup fq_codel for bandwidth shaping
    tc qdisc add dev "$DEV" parent 1: fq_codel limit "$CODEL_LIMIT" target "$CODEL_TARGET" interval "$CODEL_INTERVAL" flows "$CODEL_FLOWS" noecn


    # INGRESS (incoming traffic, "downloads"):

    # setup bandwidth limiting (ingress limiting needs IFB or Intermediate Functional Block, see https://wiki.linuxfoundation.org/networking/ifb):
    tc qdisc add dev "$DEV" handle ffff: ingress
    ip link add name "$IFB" type ifb
    tc qdisc add dev "$IFB" root handle 1: tbf rate "$DOWNLINK_RATE" burst "$DOWNLINK_BURST" latency "$TBF_LATENCY"

    # setup fq_codel for bandwidth shaping
    tc qdisc add dev "$IFB" parent 1: fq_codel limit "$CODEL_LIMIT" target "$CODEL_TARGET" interval "$CODEL_INTERVAL" flows "$CODEL_FLOWS" ecn
    ip link set dev "$IFB" up

    # connect ingress filtering to actual WAN device
    tc filter add dev "$DEV" parent ffff: protocol all prio 10 u32 match u32 0 0 flowid 1:1 action mirred egress redirect dev "$IFB"

    # configure initcwnd and initrwnd
    ip route change $(ip route | grep ^default) initcwnd "$INITCWND" initrwnd "$INITRWND"
}

remove_shaping()
{
    tc qdisc list | grep -q "ingress" && tc qdisc del dev "$DEV" ingress || true
    tc qdisc list | grep -q "codel" && tc qdisc del dev "$DEV" root || true
    ip link show | grep -q "$IFB" && ip link del "$IFB" || true
}

status()
{
        echo "─── queue discipline configuration: ──────────────────"
        tc qdisc list
        echo "   TIP: use e.g. 'sudo tc qdisc del dev $DEV ingress' to remove ingress filtering"
        echo "   TIP: use e.g. 'sudo tc qdisc del dev $DEV root' to remove egress filtering"
        echo "─── ip link show: ────────────────────────────────────"
        ip link show
        echo "   TIP: use e.g. 'sudo ip link del $IFB' to remove ingress device"
}

color_status()
{
    status | grep --color=auto -E "^|$DEV|$IFB|rate [^ ]+"
}

# handle parameters

ACTION="$1"
shift || true

while [ ! -z "$1" ]
do
    case "$1" in
        -v|--verbose)
            echo "Device: $DEV"
            echo "Downlink rate (ingress): $DOWNLINK_RATE"
            echo "Uplink rate (egress): $UPLINK_RATE"
            set -x
            ;;
        *)
            if [ ! -z "$2" ]; then
                echo "Unknown parameter: '$2'" 1>&2
                exit 1
            fi
            ;;
    esac
    shift
done

case "$ACTION" in
    start)
        remove_shaping
        configure_shaping
        ;;
    stop)
        remove_shaping
        ;;
    status)
        color_status
        ;;
    restart)
        remove_shaping
        configure_shaping
        ;;
    *)
        echo "Unknown action: $1" 1>&2
        echo "Usage: $0 <start|stop|restart|status> [--verbose|-v]" 1>&2
        exit 1
esac

You simply save the script as traffic-shaping and chmod a+x it and run it as root (after reading the source code, obviously).

For your use case, I'd suggest

DOWNLINK_RATE=5.0Mbit UPLINK_RATE=5.0Mbit TBF_LATENCY=500ms ./traffic-shaping start
4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.