5

During a recent penetration test it was found that we didn't have dai set quite as tight as possible to fully stop things. I'm running Cisco 3750x if it matters.

The interfaces are configured with ip arp inspection rate limit 200. There is oftentimes that many ARP requests, so bumping down the limit is a problematic. Since that limit wasn't being exceeded the interface is not being blocked, even with malicious traffic.

I've already enabled the validation ip arp inspection validate ip dst-mac src-mac and set up the errdisable recovery to a longer time period.

What can I do here to tighten things up? Maybe be sure to shut down an interface if a specific ARP broadcast (such as changing the MAC address for the default gateway) occurs?

Edit for request, hopefully I didn't clean up too much.

CiscoStack1#sh ru
...
switch 1 provision ws-c3750x-48
switch 2 provision ws-c3750x-48
switch 3 provision ws-c3750x-48p
switch 4 provision ws-c3750x-48p
ip routing
...
!
ip dhcp snooping vlan 20-22
ip dhcp snooping
...
ip arp inspection vlan 20-22
ip arp inspection validate src-mac dst-mac ip
ip arp inspection filter static20 vlan  20

errdisable recovery cause arp-inspection

! All the ports are configured like this one, or marked as trusted.
interface GigabitEthernet1/0/14
 switchport access vlan 20
 switchport mode access
 ip arp inspection limit rate 200
 snmp trap mac-notification change added
 snmp trap mac-notification change removed
!
interface Vlan20
 ip address 192.168.20.1 255.255.255.0
 ip directed-broadcast 50
!
interface Vlan21
 ip address 192.168.21.1 255.255.255.0
 ip directed-broadcast 50
!
interface Vlan22
 ip address 192.168.22.1 255.255.255.0
 ip directed-broadcast 50
!
no ip http server
no ip http secure-server
logging 192.168.100.6

arp access-list static20
 permit ip host 192.168.20.242 mac host c4d6.5530.7ff1
 permit ip host 192.168.20.244 mac host 0000.b4d5.2929
 permit ip host 192.168.20.241 mac host 4c11.bfaf.5bdd
 permit ip host 192.168.20.252 mac host 085b.0e7e.0a54
 permit ip host 192.168.20.252 mac host 085b.0e7e.a54b
arp access-list staticmacs
 permit ip any mac any log

mac address-table notification change history-size 500
mac address-table notification change
mac address-table notification mac-move

And sh ip source binding, so I know I have snooping enabled.

#sh ip source binding
MacAddress          IpAddress        Lease(sec)  Type           VLAN  Interface
------------------  ---------------  ----------  -------------  ----  --------------------
00:08:5D:6E:D3:4C   192.168.22.114   592866      dhcp-snooping   22    GigabitEthernet3/0/32
D4:BE:D9:9C:53:8C   192.168.22.15    525782      dhcp-snooping   22    GigabitEthernet4/0/25
00:08:5D:6E:D1:83   192.168.22.105   547010      dhcp-snooping   22    GigabitEthernet4/0/39
64:00:6A:83:37:1A   192.168.22.6     590621      dhcp-snooping   22    GigabitEthernet4/0/15
8C:EC:4B:65:A4:EE   192.168.22.62    581158      dhcp-snooping   22    GigabitEthernet4/0/16
D4:BE:D9:9D:B3:96   192.168.20.7     587056      dhcp-snooping   20    GigabitEthernet2/0/20
EC:F4:BB:1A:4C:FD   192.168.20.88    501164      dhcp-snooping   20    GigabitEthernet1/0/3
00:08:5D:6E:D1:43   192.168.22.176   593789      dhcp-snooping   22    GigabitEthernet1/0/11
34:E6:D7:54:4F:E4   192.168.22.158   580190      dhcp-snooping   22    GigabitEthernet1/0/22
8C:EC:4B:14:17:C9   192.168.20.83    585122      dhcp-snooping   20    GigabitEthernet1/0/3

Total number of bindings: 205

EDIT - just checked back, the directed broadcast has a blank access list. I believe it was used previously regarding pxe booting. At some point we converted to using the bootfile and next-server in the dhcp scope.

Edit - the wake on lan functionality in SCCM is now fully turned off and the directed-broadcast entries removed.

13
  • Are you using the ARP access-list staticmacs anywhere? I see it created but it isn't referenced in what you provide. – YLearn Aug 1 '19 at 0:45
  • @YLearn I did have one orphaned list, I think that's it. – Tim Brigham Aug 1 '19 at 1:33
  • 1
    In a /24 you can have at most 254 hosts. A single host would only need to ARP for 253 other devices (or respond to them), so a host would need to either ARP for ~80% of all hosts on the subnet in under a second or have ~80% of other hosts ARP for them in under a second. I have never known end user devices that legitimately needed to send 200+ ARP packets per second. You should be able to use DHCP and/or ip helper to allow for PXE booting rather than the generalized ip directed broadcast which is discouraged for a number of reasons. – YLearn Aug 1 '19 at 22:42
  • 1
    @TimBrigham directed broadcast in a PXE environment isn't for PXE booting as such. It is usually is needed for Wake-on-LAN. PXE works fine with the right DHCP options and TFTP to download the initial boot file. But someone/something has to wake up the clients in the first place; it's here that Wake-on-LAN (WoL) kicks in. As long as you don't run the dreaded "wake up proxy" (docs.microsoft.com/en-us/sccm/core/clients/deploy/plan/…), which does unspeakable things on a network, you need WoL, directed broadcast (with ACL!) and a server sending the magic packet. – Marc 'netztier' Luethi Aug 2 '19 at 6:04
  • 1
    .. speaking of which... you would not actually be running the "SCCM wake-up proxy", would you? Wake-up proxy's (randomly elected) wake-up proxy manager computers will even go as far as to impersonate MAC addresses of other (sleeping) computers of the subnet, faking ARP responses on their behalf just to keep the router's ARP caches alive and to "tune" the switches' CAM tables to direct traffic to themselves. This might be the source of your somewhat high ARP traffic volumes. "Wake-up proxy" may work on simple networks, but it has no place in a properly maintained and secured network. – Marc 'netztier' Luethi Aug 2 '19 at 6:18
2

In the end I made a number of changes to address this issue, in large part thanks to the comments here. I'm working down the broadcast volume so I can set the rate limiting to a reasonable level.

Answering back just in case it's useful to anyone else down the road. I'd say there is about a 99% reduction. It's down to only requests from 192.168.20.1 and requests from admin workstations.

  • Removed the old, no longer needed directed-broadcast settings
  • Verified the sccm wake-up proxy was disabled
  • Shut off any sccm wake on lan functionality
  • Disable "delivery optimization" for windows update - this was a really chatty one
  • Disabled Google Chrome's casting, via the EnableMediaRouter policy
  • Disabled Windows SSDP service

One additional point of note. . If using windows firewall with ipsec enabled it looks like refresh occuring on main mode generates a periodic spike of ARP broadcasts.

My theory is the following.

  • IPSEC negotiation will establish a session with any applicable computer, including those on the same subnet
  • There are the windows 100+ devices on the same subnet. This is nearly identical to the number of arp requests we get to set off the DAI threshold event.
  • Certain broadcast traffic results in an ipsec main mode session between all windows PCs on the same subnet
  • IPSEC sessions periodically time out and need to be renegotiated.. If these all happen at the same time you get a spike of ARP requests, since there they would rarely still be in the cache.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.