2

Sorry if this is a noob-ish question, but I am a bit lost.

We are currently having some network issues, that seem to be manifesting in what appear to me to be odd ways. End result is I have some backup jobs that are failing due to network connection being lost.

The network that we are dealing with is a stack of 6 Cisco 3850's (running IOS-XE 03.06.04.E) doing L3 routing for our entire network. This stack, in addition to doing the L3 routing is hosting clients for this building as well as many of our servers.

While troubleshooting the backup failures, I found some odd results that I am hoping someone might be able to help me with. I performed a copy of a 20GB file from server A (port Gi6/0/44) to server B (port Gi6/0/38), both on the same VLAN. The odd thing is that the rxload of the sending port goes to 2xx/255 (sh cont util shows 99% receive) and the txload of the receiving port goes to 2xx/255 (sh cont util shows 99% transmit) which seems exactly backwards to me. Additionally, the receiving port has lots of output drops/errors. Below is the sh int for the 2 interfaces during file copy. Also, reliability for the ports seems to degrade during the transfer, going down as far as 200/255 that I have seen.

Sending port

GigabitEthernet6/0/44 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is f09e.63dc.63ac (bia f09e.63dc.63ac)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 72/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 00:01:17
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 283590000 bits/sec, 23371 packets/sec
  5 minute output rate 1897000 bits/sec, 2367 packets/sec
     5392809 packets input, 8181417474 bytes, 0 no buffer
     Received 4 broadcasts (4 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 4 multicast, 0 pause input
     0 input packets with dribble condition detected
     491872 packets output, 38490525 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Receiving Port

GigabitEthernet6/0/38 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is f09e.63dc.63a6 (bia f09e.63dc.63a6)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 249/255, txload 239/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output never, output hang never
  Last clearing of "show interface" counters 00:01:10
  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 171702
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 3667000 bits/sec, 6244 packets/sec
  30 second output rate 938277000 bits/sec, 77528 packets/sec
     440644 packets input, 30590281 bytes, 0 no buffer
     Received 15 broadcasts (1 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 1 multicast, 0 pause input
     0 input packets with dribble condition detected
     5001112 packets output, 7564904837 bytes, 0 underruns
     171702 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

I am not sure where to go or what to look for with what I am seeing.

Thanks, Stanfosd

Improve this question
4
  • 1
    The txload is from the perspective of the switch, not the server, so the receiving server port should be transmitting a lot more than it receives. You appear to be overloading the transmit queue on the port for the receiving server. There is probably other traffic than the backup trying to use that server.
    – Ron Maupin ♦
    Commented Mar 29, 2016 at 14:52
  • If the transmit queue was being overloaded, shouldnt it show something other than 0/40? Also, do i need to worry about the 171K+ output errors during this 45 second test, or is that due to overloading the queue? Since these 2 servers are both 1GB links on the same vlan, on the same switch, I am surprised by such results; should I be?
    – Scott Stanford
    Commented Mar 29, 2016 at 16:29
  • See my answer. If you are sending the full 1 Gbps from one server to the other, any traffic from any other source to the receiving server will overload the interface. Switch buffers are extremely small.
    – Ron Maupin ♦
    Commented Mar 29, 2016 at 17:02
  • Did any answer help you? if so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you could provide and accept your own answer.
    – Ron Maupin ♦
    Commented Aug 7, 2017 at 22:33

3 Answers 3

Reset to default
3

You're running the interface at 94% of capacity -- 938M out of 1000M -- without flow control. The hardware buffer space is very limited (and shared between several ports in a group) so this is expected behavior when running near the limit. There's not much that can be done here: use a better switch (deeper buffers, faster ports, etc.), move things around to be in different groups, enable flow-control (limited support on the 3850.)

Improve this answer
1

As I wrote in my comment, your thinking that the txload is backwards is because you are not looking at it from the perspective of the switch instead of the servers. The most likely cause of a large output errors number is that you are overrunning the output queue.

Cisco has many documents on troubleshooting interface errors, e.g. Troubleshooting Switch Port and Interface Problems:

output errors

Description: Cisco IOS sh interfaces counter. The sum of all errors that prevented the final transmission of datagrams out of the interface. Common Cause: This issue is due to the low Output Queue size.

Improve this answer
1

You migh also be hitting Cisco bug CSCvb65304....

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.