Random DHCP clients do not accept Cisco Catalyst DHCP server's offer

Question

I have a central Cisco Catalyst 3750-X that acts as the DHCP server for all VLANs that need DHCP. Three VLANs need DHCP: 81, 100, and 111. I have had no issues with DHCP on VLANs 81 and 100. However, on VLAN 111, random devices do not accept the DHCP offer from the Catalyst.

Some devices work just fine, others do not. I've not yet found any similarities between the devices that don't work. I've seen this problem with Windows, Mac OS, and Android. I've not seen this happen to Linux machines or iPhones, but that might be because we don't have as many of those connecting to this VLAN.

I've run Wireshark with a display filter of "bootp" on devices that work and don't work, and the output looks the same. If necessary I can upload some pcaps somewhere. Admittedly, I am not practiced at reading packet captures, so the answer could be here.

To troubleshoot further, I found a laptop that I know has this issue. I've tried connecting to this VLAN from this laptop both wired and wirelessly, so this issue also doesn't seem to be specific to a network card. I've configured a switch port on the Catalyst as an access port on VLAN 111 and connected this laptop to it and experienced the same issue. This further proves that the issue doesn't come from one of the other switches or access points on the network.

Using "debug ip dhcp server events" on the Catalyst, I get the same output over and over again. It appears the client is constantly requesting an IP address, and despite the Catalyst assigning it one, the client never acknowledges this. Here's a small snippet of this event log:

Sep 23 16:54:39.023: DHCPD: Sending notification of DISCOVER:
Sep 23 16:54:39.023:   DHCPD: htype 1 chaddr ecf4.bb9b.c814
Sep 23 16:54:39.023:   DHCPD: interface = Vlan111
Sep 23 16:54:39.023:   DHCPD: class id 4d53465420352e30
Sep 23 16:54:39.023:   DHCPD: out_vlan_id 0
Sep 23 16:54:39.023: DHCPD: Sending notification of DISCOVER:
Sep 23 16:54:39.023:   DHCPD: htype 1 chaddr ecf4.bb9b.c814
Sep 23 16:54:39.023:   DHCPD: interface = Vlan111
Sep 23 16:54:39.023:   DHCPD: class id 4d53465420352e30
Sep 23 16:54:39.023:   DHCPD: out_vlan_id 0
Sep 23 16:54:39.023: DHCPD: DHCPOFFER notify setup address 10.10.111.161 mask 255.255.255.0
Sep 23 16:54:39.023: DHCPD: Sending notification of ASSIGNMENT:
Sep 23 16:54:39.023:  DHCPD: address 10.10.111.161 mask 255.255.255.0
Sep 23 16:54:39.023:   DHCPD: htype 1 chaddr ecf4.bb9b.c814
Sep 23 16:54:39.023:   DHCPD: lease time remaining (secs) = 86400
Sep 23 16:54:39.023:   DHCPD: interface = Vlan111
Sep 23 16:54:39.023:   DHCPD: out_vlan_id 0
Sep 23 16:54:40.047: DHCPD: Sending notification of DISCOVER:
Sep 23 16:54:40.047:   DHCPD: htype 1 chaddr ecf4.bb9b.c814
Sep 23 16:54:40.047:   DHCPD: interface = Vlan111
Sep 23 16:54:40.047:   DHCPD: class id 4d53465420352e30
Sep 23 16:54:40.047:   DHCPD: out_vlan_id 0
Sep 23 16:54:40.047: DHCPD: Sending notification of DISCOVER:
Sep 23 16:54:40.047:   DHCPD: htype 1 chaddr ecf4.bb9b.c814
Sep 23 16:54:40.047:   DHCPD: interface = Vlan111
Sep 23 16:54:40.047:   DHCPD: class id 4d53465420352e30
Sep 23 16:54:40.047:   DHCPD: out_vlan_id 0
Sep 23 16:54:40.047: DHCPD: DHCPOFFER notify setup address 10.10.111.161 mask 255.255.255.0
Sep 23 16:54:40.055: DHCPD: Sending notification of ASSIGNMENT:
Sep 23 16:54:40.055:  DHCPD: address 10.10.111.161 mask 255.255.255.0
Sep 23 16:54:40.055:   DHCPD: htype 1 chaddr ecf4.bb9b.c814
Sep 23 16:54:40.055:   DHCPD: lease time remaining (secs) = 86400
Sep 23 16:54:40.055:   DHCPD: interface = Vlan111
Sep 23 16:54:40.055:   DHCPD: out_vlan_id 0

These messages rapidly repeat themselves. I had to disconnect from the switch in order to copy this text. The MAC address referenced is the test laptop's MAC address.

Here are the relevant portions of my Catalyst switch config. I've redacted some of the sensitive parts with X's.

DHCP config:

ip routing
ip dhcp excluded-address 10.10.100.1 10.10.100.99
ip dhcp excluded-address 10.10.100.200 10.10.100.254
ip dhcp excluded-address 10.0.81.1 10.0.81.99
ip dhcp excluded-address 10.0.81.200 10.0.81.254
ip dhcp excluded-address 10.10.111.1 10.10.111.99
!
ip dhcp pool office100
   network 10.10.100.0 255.255.255.0
   bootfile gpxelinux.0
   next-server XXXXXX
   default-router 10.10.100.1 
   domain-name XXXXXX
   dns-server 10.10.100.7 
   option 66 ascii XXXXXX
   option 67 ascii gpxelinux.0
!
ip dhcp pool Ubiquiti
   network 10.0.81.0 255.255.255.0
   default-router 10.0.81.1 
   dns-server 8.8.8.8 
   domain-name XXXXXX
!         
ip dhcp pool Guest
   network 10.10.111.0 255.255.255.0
   default-router 10.10.111.1 255.255.255.0 
   dns-server 8.8.8.8 8.8.4.4 

VLAN config (only relevant VLANs):

interface Vlan81
 description Ubiquiti
 ip address 10.0.81.1 255.255.255.0
!
interface Vlan100
 description Office
 ip address 10.10.100.1 255.255.255.0
!
interface Vlan111
 description Guest
 ip address 10.10.111.1 255.255.255.0
!

Feel free to request more information. I would rather not paste the whole config, but I can put it on Pastebin if I have to.

I'm at a loss. The more I investigate, the more everything looks like it's working as it should. If this was happening on one device, I would blame it on something weird going on with that device, but this is happening on many. I appreciate any guidance on where to look next.

Answer 1

Brad,

As discussed in the comment, the issue is at default-router 10.10.111.1 255.255.255.0 on ip dhcp pool Guest. Once you changed it to default-router 10.10.111.1, the issue is gone.