QoS shaper, shapes traffic without dropping packets?

Question

I have a problem with my QoS policy, or I should better say, I have a problem understanding how my policy manages to achieve what ask of it without dropping a packet!

Here is the scenario:

We have an GRE point-to-point Tunnel, Crypto-Map protected with IPsec, and we shape traffic going out of it to 10Mbps. We use HQF so I have a parent shaping policy applied outgoing on the tunnel interface and a child policy with 5 classes (default included). The path towards the other end of the tunnel can handle 100Mbps all the way, and CPUs on the edge routers are monitored and within normal operation levels, so there is nothing other than my shaper limiting traffic to 10Mbps. Policy Classes “A”, “C”, “D”, “Default” are set with “queue-limit 234” (Class “B” gets the default 64 packets) therefore the sum of the max-queues of all 5 classes equals less than 1000 packets which is the default “max-queue” value on the physical interface that sources the tunnel. Ping shows RTT through the Tunnel is around 12msec.

This is how I test my policy:

I FTP a large ISO file to a host on the other side of the tunnel. This first FTP matches class "C" for example and it exceeds the classe's assigned "weight". I notice that tunnel outgoing traffic gets shaped to 10 Mbps. “show policy-map” shows a few packets queued but no drops… With the tunnel still being full (10Mbps) I start a second FTP that matches Class “D”, large ISO file again, and after a while “show policy-map” shows that the two classes are traversed by traffic in proportion to the “weights” assigned to them with the bandwidth command. Tunnel traffic remains 10Mbps and still a few packets in queue but no drops… Even more, “normal” network traffic matching the other classes (not exceeding their assigned “weights”) still went through without a problem.

The question:

How does my policy manage to shape traffic to the requested size without dropping packets?

My (obviously wrong) theory:

What I would expect was that when TCP exceeded the available BW of a class, packets would pile up on the class queue until it got full and start tail-dropping. Then TCP eventually would have to retransmit the unacknowledged data and lower the window size until it tries to enlarge it again after a while and we end up with tail-drop like before and here we go again, as a result shaping traffic to 10Mpbs.

But where are my tail-drops??? What am I missing here?

The router is a Cisco 2821 with IOS c2800nm-advsecurityk9-mz.151-4.M12a. Here is a piece of my config:

class-map match-any A
 match access-group name ACL_A
class-map match-any B
 match access-group name ACL_B
class-map match-any C
 match access-group name ACL_C
class-map match-any D
 match access-group name ACL_D
!
!
policy-map CHILD
 class A
  bandwidth percent 5
  queue-limit 234 packets
 class B
  bandwidth percent 5
 class C
  bandwidth percent 40
  queue-limit 234 packets
 class D
  bandwidth percent 40
  queue-limit 234 packets
 class class-default
  queue-limit 234 packets
  bandwidth percent 10
!
policy-map PARENT
 class class-default
  shape average 10000000
  service-policy CHILD
!
!
interface Tunnel1
 bandwidth 10000
 ip address x.x.x.x 255.255.255.252
 ip mtu 1400
 ip flow ingress
 ip flow egress
 load-interval 30
 qos pre-classify
 keepalive 3 3
 tunnel source GigabitEthernet0/1.4042
 tunnel destination z.z.z.z
 tunnel path-mtu-discovery
 service-policy output PARENT
!
!    
interface GigabitEthernet0/1
 no ip address
 load-interval 30
 duplex auto
 speed auto
!
!
interface GigabitEthernet0/1.4042
 bandwidth 10000
 encapsulation dot1Q 4042
 ip address w.w.w.w 255.255.255.252
crypto map XXXZZZ
!

Answer 1

How does my policy manage to shape traffic to the requested size without dropping packets? ... Tunnel traffic remains 10Mbps and still a few packets in queue but no drops…

Your C-class FTP and D-class FTPs are getting handled exactly as you asked for in your well-written CBWFQ policy.

The fact that you see some packets in the queues means that the policy is doing what you asked for; if you saw packet drops in this scenario, Cisco IOS would be at fault.

Since you're essentially delaying some TCP packets, the OS kernel's TCP RTT estimate gets a little higher when you have competing traffic for that 10Mbps PARENT class, but that's all that will happen in this scenario.

CBWFQ only goes active when there are packets in the queue; you asked for 40% of your queue bandwidth to be allocated to "C FTP" and 40% of the queue bandwidth to be allocated to "D FTP". Essentially you are slowing down the FTP sessions when there is competition for that same 10Mbps in the PARENT class shaper.

But where are my tail-drops??? What am I missing here?

You aren't sending enough traffic yet to exceed those 234-packet Class C and Class D queue-limits. Send a lot more TCP sessions and you'll start busting those queues and dropping packets.

[in comments] How is TCP slowing down if not dropping packets?

To know the exact reason why you're slowing traffic down, please post wireshark packet captures; the easiest place to get captures is on your ethernet switches. This avoids distortions caused by TCP Segmentation Offload, which is turned on by default in your NIC driver.

Answer 2

Following Mike’s advice I got some captures and, well, I have half an answer to my question. I read around the internet a few things, RFC 5681 about TCP Congestion Control among others, in order to better understand what I should expect from TCP, then I played a bit with wireshark and the different captures and here’s what I made out of all that.

The images I provide are from a capture made close to the sender, before my QoS policy is applied. Also the test was made as described in the question at the beginning. One FTP starts matching a class with 40% BW guarantee of the 10Mbps link, then a second one matching a different class with 40% BW guarantee. Second FTP is stopped and after a while the first one is stopped as well. For the time of the capture no other traffic was allowed on the link. This helped clear things up a bit and come up with this half of an answer. The images refer only to the first FTP stream all the way to the end.

From t=0 to t=34 TCP receivers receive window is 64KB, and sender’s Bytes-in-Flight reach that number. RTT rises up, which indicates that some delay is inserted in the communication between sender and receiver. Due to the sender hitting the receiver’s advertised window size, the sender is waiting for the receiver to acknowledge packets so that it can send more. Further ACKs are not being sent because packets sent by the sender are not received yet by the receiver because they are delayed by the QoS policy before they are allowed on the 10Mbps link. When the QoS policy permits a packet on the link an ACK is sent for it to the sender and then the sender can send the next packet. The rate at which the QoS policy allows packets on the link is therefore the rate that the sender is allowed to send further packets on the network, in effect limiting the TCP stream to the 10Mbps of the link!

I hope this was understandable so far, because this was the answer bit that I had…

I know now that a QoS policy is capable of limiting the transfer rate of a TCP stream without dropping a packet, provided of course there is enough queue depth for all the queued packets, which in my case seems to be true. (One important thing here is to notice the relation between queue depth and delay imposed by the Qos policy, but this is out of topic at the moment)

To the remaining problem now: After t=34 and up to around t=58 the second FTP is sent through the same link and I can confirm that my QoS policy hasn’t dropped a packet during this test. As one can see in the images I uploaded, competition for BW starts between the 2 TCP streams and TCP on the first stream is backing off repeatedly, as can be confirmed by the saw-tooth like graphs.

The question is the same:

Since in order for TCP congestion avoidance to come in effect a “loss event” (RTO expiration or Duplicate ACK sent by receiver) needs to be sensed by the sender, and my QoS policy isn’t dropping packets, how is it limiting the TCP stream rate to the one available on the link.

The clue is understanding the reason that triggers this back off but I am having a difficult time finding it out. Is it because of a packet not arriving at the receiver because it got dropped somewhere on the way to the receiver, or because a packet arrived out of order at the receiver, or because the receiver’s receive buffer was overrun by the sender (while I can’t see how this can happen because of the sender since it is always respecting the advertised window size…!) or because the RTO timer expired at the sender due to the excessive amount of delay by the queueing at the QoS link

To be honest in this capture there were a few “TCP previous packet not captured” occurrences during the congestion period so, I want to confirm this and make another capture, this time on the other side of the link, close to the receiver, which is a bit tricky...

In any case I have half an answer, and a lot of suspects to blame (that are not Qos drops) for the congestion portion, so I must be in the right track. Is my thinking wrong here?