Our core router is a Cisco 7206VXR with NPE-G2 running 12.4(20)T1. It has numerous interfaces connected to WAN links (DS3, ATM DS3, 2xT1, onboard Gig0/1), and feeds into the LAN core with 2xGig EtherChannel.
The LAN side clearly has more bandwidth than any of the WAN links. We address this with a service-policy on the WAN side that allocates bandwidth to each "interesting" class of service, based on an ACL or DSCP markings. So far, so good.
Our problem is that we're seeing excessive tail drops for one particular class of traffic that leaves on a serial DS3 card:
Class-map: class-symposium-unicast-acl (match-all) 1530 packets, 2153296 bytes 30 second offered rate 336000 bps, drop rate 247000 bps Match: access-group name acl-symposium-ucast Queueing queue limit 64 packets (queue depth/total drops/no-buffer drops) 0/1056/0 (pkts output/bytes output) 474/672266 bandwidth remaining 10% (4421 kbps) Exp-weight-constant: 9 (1/512) Mean queue depth: 50 packets dscp Transmitted Random drop Tail drop Minimum Maximum Mark pkts/bytes pkts/bytes pkts/bytes thresh thresh prob af31 474/672266 22/31547 1034/1449483 32 40 1/10
This is an abbreviated version of our configuration:
class-map match-all class-symposium-unicast-acl match access-group name acl-symposium-ucast ! policy-map WAN class class-symposium-unicast-acl bandwidth remaining percent 10 random-detect dscp-based class class-default ! interface Serial2/0 description PA-T3/E3-EC - 45 Mbps DS3 service-policy output WAN ! ip access-list extended acl-symposium-ucast permit udp host x.x.x.x any
hold-queue on the serial interface, or
queue-limit within the policy-map have no apparent effect, and repeatedly checking
show int or
show policy-map does not show the output queue filling up. Bandwidth should not be an issue, because the interface is very lightly loaded (under 2 Mbps output on a 45 Mbps link), and the offered rate on the class-map is well below the bandwidth provisioned. Removing the
service-policy from the serial interface causes the drops to stop, but we need to be able to reserve bandwidth for certain applications, so this is not an acceptable long-term solution.
I suspect the nature of the affected traffic has something to do with the drops we're seeing, because it's unlike anything else our network. UDP packets are emitted from a server at regular (7-second) intervals to one or more remote PCs that request it. Some of the packets are in excess of 15KB, so they have to be fragmented.
Could the excessive fragmentation be a contributing factor, and if so, would
ip virtual-reassembly on the ingress interface be of any benefit? Any other thoughts or suggestions?
EDIT: Output of "show int ser 2/0" as requested:
Serial2/0 is up, line protocol is up Hardware is PA-T3/E3-EC Description: xxxxx Internet address is x.x.x.x/30 MTU 4470 bytes, BW 44210 Kbit/sec, DLY 200 usec, reliability 255/255, txload 8/255, rxload 235/255 Encapsulation PPP, LCP Open Open: IPCP, CDPCP, crc 16, loopback not set Keepalive set (10 sec) Restart-Delay is 0 secs Last input 00:00:22, output 00:00:00, output hang never Last clearing of "show interface" counters 1d00h Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 14291 Queueing strategy: Class-based queueing Output queue: 0/1000/0 (size/max total/drops) 30 second input rate 40774000 bits/sec, 3435 packets/sec 30 second output rate 1442000 bits/sec, 1847 packets/sec 274986144 packets input, 4019985342 bytes, 0 no buffer Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 parity 6 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 6 abort 149374995 packets output, 1855187875 bytes, 0 underruns 0 output errors, 0 applique, 3 interface resets 0 unknown protocol drops 0 output buffer failures, 0 output buffers swapped out 0 carrier transitions no alarm present DSU mode 0, bandwidth 44210 Kbit, scramble 0, VC 0
I realize the input rate is close to maximum in this copy/paste, but the problem occurs even when input is next to zero.