3

Ports in my network on random switches(connected to Windows 7 workstations) are going into err-disable because of dynamic arp inspection. Ports are going into err-disable because there are too many arp requests packets per second.

Please see output from the log:

Mar  2 13:14:35 EST: %SW_DAI-4-PACKET_RATE_EXCEEDED: 21 packets received in 922 milliseconds 
on Gi1/0/6.
Mar  2 13:14:35 EST: %PM-4-ERR_DISABLE: arp-inspection error detected on Gi1/0/6, putting 
Gi1/0/6 in err-disable state
Mar  2 13:14:36 EST: %LINEPROTO-5-UPDOWN: Line protocol on Interface 
GigabitEthernet1/0/6, changed state to down

I was able to capture port going into error disable (through SPAN). Please see attached capture: https://app.box.com/s/2ow0tlutzxwk5odk0zuqhug1kgqlykbd

The IP address of the workstation that is on port gig1/0/6 is 151.121.79.89.

As far as I see there are many ARP requests for 151.121.79.89 from random workstations at 13:14:34 and that is causing port to go into err-disable.

I am working with a Cisco WS-C2960S-48FPS-L

Port configuration is standard for all access ports:

switchport access vlan 79
switchport mode access
switchport voice vlan 3001
switchport port-security maximum 3
switchport port-security
switchport port-security aging time 2
switchport port-security violation restrict
ip arp inspection limit rate 20
srr-queue bandwidth share 1 30 35 5
queue-set 2
priority-queue out
mls qos trust device cisco-phone
mls qos trust cos
spanning-tree portfast
service-policy input AUTOQOS-SRND4-CISCOPHONE-POLICY
ip dhcp snooping limit rate 20

I'm also including the arp inspection config (the dhcp-snooping config on vlan 79 is typical):

ip arp inspection vlan 79
ip arp inspection validate src-mac ip
ip arp inspection log-buffer entries 1024
ip arp inspection log-buffer logs 1024 interval 10

I am wondering why would random workstations try to resolve ip address of workstation connected to the 1/0/6 in less than a second?

3

ip arp inspection limit rate 20

20 packets per second is really low. If there's a server on that port (or more than one system), 20 pps is nothing.

Update

I don't know what any hosts are on your network (who's talking to who, and why), but for the minute at 13:14:34, 15 hosts asked who 89 was. And 89 asked who 5 hosts were. The request ~0.1sec before this period runs the total to 21pps. [filter: arp && (eth.src == 10:1f:74:f5:be:36)]

[Note: Cisco's default is even worse at 15.]

  • yes, arp packets per second. And it's an insanely low limit. Upon reflection, Cisco's entire logic here is flawed: the rate applies to all arp packets, not just requests. For an anti-DoS feature, it's a total fail - any two hosts in the network can kill every access port on the switch. (send 11 requests for the same MAC in less than a second, each. poof port where that MAC is attached is disabled.) – Ricky Beam Mar 7 '15 at 1:47
  • I was aware that rate limit is 20. What value would you recommend? I tried setting up to 50 and still had some ports exceeded that rate. I had 20 configured for last 8 months and haven't had problems. Ports started to go to err-disable recently. Is there a way to see why are different hosts trying to arp for .89? There is no reason for hosts to talk to each other as far as I know in this network. Thank you all for the answers! – user4608 Mar 7 '15 at 20:56
  • And absolutely nothing changed for 8 months? I doubt that -- more machines, more users, software/OS updates/upgrades. As I said, it's not just 89 looking for hosts, but 89 answering other hosts as well. Any and all ARP packets are counted. I'd blame Windows -- win7 tends to talk to everything it can find. – Ricky Beam Mar 9 '15 at 3:17
  • There have been many changes int the network/systems since 8 months ago. I am suspecting that server team pushed program or an update to workstations that caused this. There is similar issue here blogs.technet.com/b/nettracer/archive/2010/08/05/… – user4608 Mar 9 '15 at 12:26
2

user4608 said:

I was aware that rate limit is 20. What value would you recommend?

There is no one good answer to this question; you need to know your environment pretty well to know where to set the threshold.

Arguably, if you're still seeing err-disables at 50 DAI inspections per second, I'd keep bumping it in increments of 25 until you find something that works. You might consider using this as a workaround until you find a value that works for all ports in your environment...

errdisable recovery interval 30
errdisable recovery cause arp-inspection

This doesn't fix the problem, but it keeps your help-desk from getting a lot of calls from users who get err-disabled while you figure out what the right DAI thresholds are.

If you just don't care about ARP rate-limits you could configure your ports like this...

sw1(config-if)#ip arp inspection limit none

However, I would rather deal with a little bit of pain so ports with unusual arp rates get flagged. There is plenty of misbehavior that happens with ARP, so I'd rather have a chance of knowing that people are doing something strange.

You can just grep through your syslogs for SW_DAI-4-PACKET_RATE_EXCEEDED until you find values that work for you.

Bonus material (dhcp snooping):

BTW, strictly speaking, this isn't what you asked about, but if you haven't already, I'd set up an alert in your syslogs when you get %DHCP_SNOOPING-5-DHCP_SNOOPING_UNTRUSTED_PORT messages (ref Ethan Banks' blog). That could indicate someone is:

  • Trying to operate their own DHCP server on your network (naturally this isn't good, but it's not always malicious)
  • Trying to spoof DHCP OFFER packets (that's bad, for instance they could assign themselves as the client's gateway in preparation for a MITM attack)
  • Thank you! That is exactly what I've done till I find the cause of too many arps. Thank you for DHCP snooping bonus too. I will add alerts from syslog. – user4608 Mar 9 '15 at 12:29
1

In our network with more than 10000 devices (many of them user-administrated Windows devices) we experience this problem since the rise of Windows Vista and Windows 7.

The problem is mostly caused by Windows Link Layer Topology Discovery (LLTD), which enables the search of windows file and printer shares in the local network.

It is important to know, that Link Layer Topology Discovery only causes problems in big LANs or VLANs with many LLTD-enabled devices. There, you can usually reliably trigger error-disabled state via connecting your LLTD-enabled device and starting Windows printer search.

The solution is

  • to raise the arp inspection limit to the count of tolerated LLTD devices and/or
  • to consequently disable Link Layer Topology Discovery on most devices.

Link Layer Topology Discovery can be disabled via one of the following steps:

  • Choose Start → Control Panel → Network and Sharing Center → Manage Network Connections → right-click on your Ethernet Connection → Properties → deactivate both Link Layer Topology Discovery Mapper and Responder service.
  • Uninstall mapper and responder drivers via "netcfg.exe -u ms_lltdio" and netcfg.exe -u ms_rspndr". This removes the drivers for all network devices.
  • Group Policy for LLTD Mapper and LLTD Responder.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.