I've been studying TC for a while.

I came across this answer containing a script.

In it we set the egress limit as:

# setup bandwidth limiting:
tc qdisc add dev "$DEV" root handle 1: tbf rate "$UPLINK_RATE" burst "$UPLINK_BURST" latency "$TBF_LATENCY"

# setup fq_codel for bandwidth shaping
tc qdisc add dev "$DEV" parent 1: fq_codel limit "$CODEL_LIMIT" target "$CODEL_TARGET" interval "$CODEL_INTERVAL" flows "$CODEL_FLOWS" noecn

I am not really sure why do we need the second line, defining the fq_codel?

How does the tbf qdisc and the fq_codel qdisc work together?

  • Unfortunately, questions about host/server configurations are off-topic here. You could try to ask this question on Super User.
    – Ron Maupin
    Sep 13 at 12:22
  • could we keep the question here though? I am thinking how to formulate it so it falls under "design or theory of protocols used to operate a network (e.g. ..., TCP ....);" and is somehow linked to the original question
    – Effie
    Sep 13 at 21:57

there is really no short and simple answer to this question. Here is a general idea, but to answer this question it might be necessary to understand exactly how a number of things work and how they interact.

First, tbf is equivalent to having an network interface with bandwidth set to of tbf. It also sets a buffer size (it can be calculated from the rate and latency).

So, what is this queue. Outgoing interfaces have so called buffers. Buffer is a memory. In simple case this memory is organized as a queue, which is, well, a data structure called queue (first in first out). If a packet has to be sent on the outgoing interface and the interface is busy (sending other packets) than this packet is queued, i.e, put in the back of the queue. When the interface can send next packet, it takes the packet from the front of the queue and sends it. If a packet should be queued and there is no memory left, then the packet is dropped.

Every device in a packet switched network uses queues on outgoing interfaces. They are there to handle temporary bursts of packets (i.e., a short mismatch between rate of incoming packets (packets to be sent via this interface) and interface bandwidth. These bursts are an artifact of packet switched network. If for some reason incoming packets rate is higher over a longer period of time, this buffer will get full over this long period of time. Now, sitting in a buffer adds latency to the packet. The bigger the queue (fuller the buffer) was when the packet was queued, the more time the packet sits in the queue, and hence higher the latency. If the buffer is big (in bytes/number of packets it can queue), this latency can become significant, compared to the delay it takes to transmit packet over the distance from sender to receiver (i.e., not counting queues). This is referred to as bufferbloat.

So, what we can do to alleviate the situation can be coarsely classified into schedulers and active queue management (AQM). FQ-Codel is actually both: it is a combination of FQ scheduler and Codel active queue management scheme.

What does a scheduler do. It organizes this buffer memory in a more sophisticated structure than a single queue. Let's use the bottom part of this image from wikipedia as an example.

example of weighted round robin

So, basically instead of one queue there are several queues and packets land in different queues based on some criteria (the process is usually called classification). For example, packets of a single flow (e.g., single tcp flow) land in the same (sub-) queue, but different flows land in different queues. When an outgoing interface can send next packet, scheduler decides from which queue to take the packet. What this does: it allocates bandwidth between queues, so that each queue gets roughly equal share of the bandwidth, while if one queue does not use it's share, it can be reallocated to other queues. You can read more about "round robin", "fair queuing", "deficit round robin", and "FQ" scheduler/qdisc.

In the example between a download and a ping, the scheduler can ensure that ping packets can go through while download packets which arrived before them are still queued.

Now what does active queue management scheme (AQM) do. As i said, if there is no space left in buffer, the packet is dropped. An AQM drops some packets before the buffer becomes full. This is done do that TCP will react to this packet loss and send less packets. The intention is to have large buffer, which can accommodate large short term bursts, but at the same time keep the average queue occupancy (i.e., the number of packets in it) low, to keep the latency low.

Now, AQMs are there because they interact with TCP congestion control. Congestion control is something that hosts need to do at layer 4 and above, to ensure that they do not overload the network, i.e., their sending rate for each flow does not exceed the available bandwidth of the path for that flow. There is no simple explanation on how exactly TCP congestion control and AQMs interact. TCP congestion control schemes TCP Reno and TCP Cubic are loss based, i.e., they treat packet losses as an indication of that they have to do something. The way Reno and Cubic work results in them filling buffer full before they actually react. AQM drops packets before the queue is full, so that TCP can react sooner. (Note, that AQM does not have to drop packets, it can mark packets instead - see explicit congestion notification (ECN)).

In the example of that question, AQM can be used to signal the download flow to reduce its rate.

Not the answer you're looking for? Browse other questions tagged or ask your own question.