5

I work for a VoIP provider, and one of the biggest hassles I deal with is downstream traffic management.

Our flagship product is QoS-enabled bonded T1 or SHDSL bundles configured to support our own VoIP service. We build QoS policies at our edge router that sort our voice traffic into the highest priority queue, and that product is very effective. When the circuit is built correctly, there's nothing the customer can do to interfere with their audio quality.

What's bothering me is that I've gotten a LOT of flack from other network engineers, IT techs, etc. who think that it's possible to make a service like this work without edge router QoS. I don't feel like this practical. I'd like to explain my understanding, and I hope somebody has the time to poke holes in it, if there are any. Pardon the lengthy question.

Let's take a Comcast circuit. I don't work for Comcast so I'm speculating on their network build, but I imagine it's something like this:

Comcast's fiber backs up to a Cisco edge router, and from there connects to their cable headend. Each customers IP is built out on the edge router with a given downstream bandwidth - say, 30mbit - which allocates them a queue of 30 megabits plus some percentage of overhead (I don't know how much this typically is).

Let's say the subscriber goes to download a 40mbyte file from a large enterprise server. They hit download, the TCP connection establishes and starts the slow-start process, but after a few seconds the server is pushing well over 30mbit of data. This fills up the queue on the edge router and it starts dropping packets.

At this point TCP congestion avoidance kicks in and starts slowing down the transfer to fit within the available bandwidth, but obviously voice is ruined already.

Now, I've been told that routers with inbound QoS policies are capable of leveraging the TCP congestion avoidance mechanism by dropping ACKs in order to rate-shape inbound traffic. I've never seen this done in practice, or at least never seen it work; is this a real thing? If so, is it effective? Wouldn't it result in a run of dropped packets every time that a new connection ramped up to speed, before the congestion algorithm could start slowing things down?

In addition to that, however, what if the traffic utilizing the circuit is largely UDP? If a request for a large amount of UDP data is sent to a server, it's naturally going to send the data as quickly as feasible, flood the queue and cause dropped voice packets - but maybe I'm overreacting to that scenario. Is UDP used for any high-throughput internet technologies these days?

More specifically, have you ever personally seen a QoS method that let VoIP work without constant quality problems on a frequently-saturated internet connection without ISP-provided QoS? Have you ever heard of a DSL or cable provider implementing edge router QoS for an application like this? I've only ever heard of that offered as part of MPLS products, is that pretty much a necessity if someone wants downstream QoS?

  • 2
    This is a pretty general question. I work for a vendor of a VoIP solution and we have always run into issues when QoS isn't applied basically everywhere. Fortunately we don't really deal with Internet connectivity but even on a shared private lan/wan you can run into trouble when you are saturating links. – Acyclic Tau Jan 11 '15 at 1:55
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer. – Ron Maupin Aug 10 '17 at 23:13
5

Now, I've been told that routers with inbound QoS policies are capable of leveraging the TCP congestion avoidance mechanism by dropping ACKs in order to rate-shape inbound traffic. I've never seen this done in practice, or at least never seen it work; is this a real thing? If so, is it effective? Wouldn't it result in a run of dropped packets every time that a new connection ramped up to speed, before the congestion algorithm could start slowing things down?

I doubt it, certainly for queues as these are relatively TCP friendly anway. Some routers might have tried this strategy to build tcp-friendly policers but I doubt this would be a decent solution. First of all, an ACK can be accompanied by data so you can't really only drop 'ACK's. Secondly, even if that is true for some streams, why would you drop only on the return path? For the TCP stream it's just as good to simply drop the original data and that way you don't waste bandwidth and processing resources between your edge router and receiver.

More likely is to have H-QoS, with a 30Mbps queue/scheduler on the circuit/user level and then multiple smaller-tier queues feeding into that. A simple setup could be to let both TCP and UDP feed into two different 25Mbps queues, so TCP can still leave 5Mbps for UDP and vice versa. Of course more complex variations exist.

Also, consider that ssthresh is usually adapted fairly quick towards your available bandwidth (here 30Mbps - your voice BW). While TCP does not stop increasing cwnd at ssthresh, it will only increase slowly afterwards and disturbing your voice stream only slightly when congestion is reached. However, at the start of the TCP connection, before ssthresh is adapted to a realistic value this will probably significantly disturb your voice flow (albeit shortly). Finally, while it might work okay with only a few TCP streams, when somebody starts 100s of TCP connections (e.g. P2P app) it gets more difficult. It's unlikely that all ssthresh and cwnd values stabilize (quickly) to guarantee fairness, even more if those connections are quite volatile (as is often the case with P2P).

Also, next to drops notice that a saturated connection also has a negative impact on your jitter, this might sometimes be just as problematic (or even worse) than a few packet drops.

In addition to that, however, what if the traffic utilizing the circuit is largely UDP? If a request for a large amount of UDP data is sent to a server, it's naturally going to send the data as quickly as feasible, flood the queue and cause dropped voice packets - but maybe I'm overreacting to that scenario. Is UDP used for any high-throughput internet technologies these days?

You're 100% correct, UDP has no fairness built-in. Many real-time multimedia applications (voip, video conferencing, ...) still use UDP. Tunnels (e.g. for VPN) also usually use UDP to avoid TCP-in-TCP retransmit timer conflicts. So yes, you certainly can have quite some high-throughput UDP traffic on a connection.

More specifically, have you ever personally seen a QoS method that let VoIP work without constant quality problems on a frequently-saturated internet connection without ISP-provided QoS? Have you ever heard of a DSL or cable provider implementing edge router QoS for an application like this? I've only ever heard of that offered as part of MPLS products, is that pretty much a necessity if someone wants downstream QoS?

Well, here the question is of course if many modern internet connections are frequently saturated. In my experience the available BW for a fixed connection often exceeds the average usage. There are always exceptions but usually the bulk of the people have loads of BW, it might still get saturated somewhere, but not that often on your direct access circuit. However, you might encounter it more often in mobile access as this is a relatively low-bandwidth low-reliability shared medium. Currently 4G helps a lot but that will only last this long as mobile BW usage is still growing heavily.

Yes, I have heard providers implementing QoS for multimedia applications. For which applications that might differ a lot from provider to provider. Assume they just qualify traffic as BE unless there is a good reason to do otherwise. But some might indifferently prioritize SIP/RTP.

MPLS is certainly not a necessity for QoS, any decent BNG today also includes quite significant QoS options.

| improve this answer | |
  • I think I need to clarify the first part of my question: I'm asking if there's any way for the router at the end-user premises to effectively slow down inbound traffic. In a scenario where the ISP is unwilling to build QoS policies on their edge router (depressingly common) there's no way to queue before the packets hit (and have a chance to saturate) the circuit. The theory is that by dropping ACKs at the premises you can artificially force TCP congestion avoidance to slow down how fast the server tries to send data. I just don't know if it really works. – Daniel Thompson Jan 14 '15 at 1:32
  • 2
    @DanielThompson yes of course it 'works', but as I say there is absolutely no point in dropping the ACK, dropping the original packet (that is going to be ack'd) is way more efficient. Alternatively you could try to manipulate rwnds. The main issue with all those approaches is that it is extremely hard to measure the impact of the router's action on rwnd, cwnd, ssthresh and thus the actual throughput. Without doing some very heavy per-stream analytics you would likely not achieve much more than killing user experience. – KillianDS Jan 16 '15 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.