2

I'm trying to sort a DHCP relay problem in a VLXAN/EVPN environment (Cisco N9K, NXOS 9.3.2).

We seem to be able to get DHCPDISCOVERs through to the server and DHCPOFFERS back from there. DHCPREQUESTs are being forwarded towards the server, too, but no DHCPACKs are ever seen coming back from the server.

QUESTION A: Is this DHCP relay behaving correctly? It's doing different things to DHCPDISCOVERs than it does to DHCPREQUESTs (see last section of post).

QUESTION B: Could the DHCP relay's behaviour be caused by the "Next Server" information given by the server in its DHCPOFFER? [1]

The server maintainer quotes RFC 5107 as the reason, for a problem description similar to the symptoms I can observe: In a DHCPREQUEST, after being relayed, Option 54 (Server Identifier) is different from the Server ID Override value, (Option 82, Suboption 11), and thus discarded by the server.

See http://lists.thekelleys.org.uk/pipermail/dnsmasq-discuss/2018q1/011899.html where the maintainer quotes RFC 5107, highligthed by me

That's old code, and pre-dates detailed git logs, but it's fairly clear that the test is implementing this part of RFC 5107

When servicing a DHCPREQUEST message, the DHCP server would normally look at the Server Identifier option for verification that the address specified there is one of the addresses associated with the DHCP server, silently ignoring the DHCPREQUEST if it does not match a configured DHCP server interface address. If the DHCPREQUEST message contains a Server Identifier Override suboption, however, comparison should be made between the address in this suboption and the Server Identifier option. If both the Server Identifier Override suboption and the Server Identifier option specify the same address, then the server should accept the DHCPREQUEST message for processing, regardless of whether or not the Server Identifier option matches a DHCP server interface.

So the problem is that the client, on renewal, is sending the server-ID as 10.36.20.19, which is tripping the test specified above.

This is a VLXAN/EVPN setup with distributed anycast gateways, so we had to resort to some trickery with a node-specific loopback address on the leaf as source address for DHCP relay packets.

This is documented in: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/vxlan/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x_appendix_010111.html#id_103141

So we configured:

ip dhcp relay
ip dhcp relay information option
ip dhcp relay information option vpn

[...]

interface loopback32
  description DHCP Relay source address for VLAN3632
  vrf member MYVRF
  ip address 10.32.15.39/32

[...]

interface Vlan3632
  description SVI for VLAN 3632
  vrf member MYVRF
  no ip redirects
  ip address 10.32.96.1/20
  fabric forwarding mode anycast-gateway
  ip dhcp relay address 10.32.33.40 
  ip dhcp relay source-interface loopback32

The whole BGP/VXLAN part may be assumed working. Also, loopback32 is reachable from the DHCP server (we are getting DHCPOFFERS back, after all), so I'm leaving out the VXLAN/EVPN config bits.

What I'm observing is the following behaviour (I'll omit debug output from the client, else this post would go beyond 32k).

Step 1: DHCPDISCOVER on relay

DHCP Relay receives the client's broadcast: Nothing wild to be expected here:

Internet Protocol, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255 (255.255.255.255)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 339
    Identification: 0x6cdb (27867)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x4dbf [correct]
        [Good: True]
        [Bad : False]
    Source: 0.0.0.0 (0.0.0.0)
    Destination: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: bootpc (68), Dst Port: bootps (67)
    Source port: bootpc (68)
    Destination port: bootps (67)
    Length: 319
    Checksum: 0x431d [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 0.0.0.0 (0.0.0.0)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Discover
        Option: (53) DHCP Message Type
        Length: 1
        Value: 01
    Option: (t=57,l=2) Maximum DHCP Message Size = 1200
        Option: (57) Maximum DHCP Message Size
        Length: 2
        Value: 04B0
    Option: (t=61,l=25) Client identifier
        Option: (61) Client identifier
        Length: 25
        Value: 00636973636F2D373037392E623366662E353134322D4769...
    Option: (t=12,l=13) Host Name = "ExpDHCPClient"
        Option: (12) Host Name
        Length: 13
        Value: 45787044484350436C69656E74
    Option: (t=55,l=9) Parameter Request List
        Option: (55) Parameter Request List
        Length: 9
        Value: 01060F2C790321962B
        1 = Subnet Mask
        6 = Domain Name Server
        15 = Domain Name
        44 = NetBIOS over TCP/IP Name Server
        121 = Classless Static Route
        3 = Router
        33 = Static Route
        150 = TFTP server address
        43 = Vendor-Specific Information
    Option: (t=60,l=8) Vendor class identifier = "ciscopnp"
        Option: (60) Vendor class identifier
        Length: 8
        Value: 636973636F706E70
    End Option

Step 2 relayed DHCPDISCOVER

Then this happens:

  • The DHCP relay VLXANifies the packet, and sends it off to the remote egress VTEP, after rewrting some of the content (outer packet headers not shown)
  • adds the configured dhcp relay source-interface as giaddr (Relay agent IP address
  • Adds/sets Option 82 Suboption 5 (Link selection) to 10.32.96.0
  • Adds/sets Option 82 Suboption 11 (Server ID Override) to 10.32.96.1
  • Adds/sets Option 82 Suboption 151 (Virtual Subnet Selection) to something I haven't been able to parse (I believe it is quite irrelevant to the current problem.)

(the follwing section's been giving me headaches to get the preformat tags around...)

Internet Protocol, Src: 10.32.15.39 (10.32.15.39), Dst: 10.32.33.40 (10.32.33.40)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 389
    Identification: 0xdb6c (56172)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x9a6c [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.15.39 (10.32.15.39)
    Destination: 10.32.33.40 (10.32.33.40)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootps (67)
    Source port: bootps (67)
    Destination port: bootps (67)
    Length: 369
    Checksum: 0xf666 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 1
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 10.32.15.39 (10.32.15.39)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Discover
        Option: (53) DHCP Message Type
        Length: 1
        Value: 01
    Option: (t=57,l=2) Maximum DHCP Message Size = 1200
        Option: (57) Maximum DHCP Message Size
        Length: 2
        Value: 04B0
    Option: (t=61,l=25) Client identifier
        Option: (61) Client identifier
        Length: 25
        Value: 00636973636F2D373037392E623366662E353134322D4769...
    Option: (t=12,l=13) Host Name = "ExpDHCPClient"
        Option: (12) Host Name
        Length: 13
        Value: 45787044484350436C69656E74
    Option: (t=55,l=9) Parameter Request List
        Option: (55) Parameter Request List
        Length: 9
        Value: 01060F2C790321962B
        1 = Subnet Mask
        6 = Domain Name Server
        15 = Domain Name
        44 = NetBIOS over TCP/IP Name Server
        121 = Classless Static Route
        3 = Router
        33 = Static Route
        150 = TFTP server address
        43 = Vendor-Specific Information
    Option: (t=60,l=8) Vendor class identifier = "ciscopnp"
        Option: (60) Vendor class identifier
        Length: 8
        Value: 636973636F706E70
    Option: (t=82,l=48) Agent Information Option
        Option: (82) Agent Information Option
        Length: 48
        Value: 010409010E30020628AC9E65F84F9714004F5330312D5245...
        Agent Circuit ID: 09010E30
        Agent Remote ID: 28AC9E65F84F
        DHCPv4 Virtual Subnet Selection: 004F5330312D52454730312D444556312D5A3030
        Server Identifier Override: 0A206001
        Link selection: 10.32.96.0
    End Option

Step 3 DHCPOFFER from server

Then, we get a DHCPOFFER back from the server, with one particularity:

  • unicasted to from server (10.32.33.40) to our loopback32 (10.32.15.39)
  • Option 54 (Server Identifier) set to 10.32.96.1
  • Relay Agent IP address set to 10.32.15.39
  • Next Server IP set to 10.32.33.40 (actual DHCP server's address, same as being relayed-to)
  • address offerred is 10.32.96.104
  • Option 82 info is reflected back to the relay

(nonsensical Line here to make preformat work...)

Internet Protocol, Src: 10.32.33.40 (10.32.33.40), Dst: 10.32.15.39 (10.32.15.39)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 395
    Identification: 0x55ba (21946)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 61
    Protocol: UDP (0x11)
    Header checksum: 0xe159 [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.33.40 (10.32.33.40)
    Destination: 10.32.15.39 (10.32.15.39)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootps (67)
    Source port: bootps (67)
    Destination port: bootps (67)
    Length: 375
    Checksum: 0xa7bb [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Reply (2)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 1
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 10.32.96.104 (10.32.96.104)
    Next server IP address: 10.32.33.40 (10.32.33.40)
    Relay agent IP address: 10.32.15.39 (10.32.15.39)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name: /undionly.kpxe
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Offer
        Option: (53) DHCP Message Type
        Length: 1
        Value: 02
    Option: (t=54,l=4) DHCP Server Identifier = 10.32.96.1
        Option: (54) DHCP Server Identifier
        Length: 4
        Value: 0A206001
    Option: (t=51,l=4) IP Address Lease Time = 1 hour
        Option: (51) IP Address Lease Time
        Length: 4
        Value: 00000E10
    Option: (t=118,l=4) Subnet Selection Option = 10.32.96.0
        Option: (118) Subnet Selection Option
        Length: 4
        Value: 0A206000
    Option: (t=58,l=4) Renewal Time Value = 30 minutes
        Option: (58) Renewal Time Value
        Length: 4
        Value: 00000708
    Option: (t=59,l=4) Rebinding Time Value = 52 minutes, 30 seconds
        Option: (59) Rebinding Time Value
        Length: 4
        Value: 00000C4E
    Option: (t=1,l=4) Subnet Mask = 255.255.240.0
        Option: (1) Subnet Mask
        Length: 4
        Value: FFFFF000
    Option: (t=28,l=4) Broadcast Address = 10.32.111.255
        Option: (28) Broadcast Address
        Length: 4
        Value: 0A206FFF
    Option: (t=15,l=13) Domain Name = "myvrf.internal"
        Option: (15) Domain Name
        Length: 13
        Value: 6465762E692E657763732E6368
    Option: (t=6,l=8) Domain Name Server
        Option: (6) Domain Name Server
        Length: 8
        Value: 0A08034A0A0803AE
        IP Address: 10.8.3.74
        IP Address: 10.8.3.174
    Option: (t=3,l=4) Router = 10.32.96.1
        Option: (3) Router
        Length: 4
        Value: 0A206001
    Option: (t=82,l=48) Agent Information Option
        Option: (82) Agent Information Option
        Length: 48
        Value: 010409010E30020628AC9E65F84F9714004F5330312D5245...
        Agent Circuit ID: 09010E30
        Agent Remote ID: 28AC9E65F84F
        DHCPv4 Virtual Subnet Selection: 004F5330312D52454730312D444556312D5A3030
        Server Identifier Override: 0A206001
        Link selection: 10.32.96.0
    End Option

Step 4 DHCPOFFER to client

The DHCPOFFER is being forwarded to the client

  • Option 82 and Suboptions removed
  • broadcasted from 10.32.96.1 to all-routes broadcast
  • Option 54 (Server Identifier) is being kept as 10.32.96.1
  • Next Server is kept as 10.32.33.40

(the follwing section's been giving me headaches to get the preformat tags around...)

Internet Protocol, Src: 10.32.96.1 (10.32.96.1), Dst: 255.255.255.255 (255.255.255.255)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0xc0 (DSCP 0x30: Class Selector 6; ECN: 0x00)
        1100 00.. = Differentiated Services Codepoint: Class Selector 6 (0x30)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 345
    Identification: 0xba55 (47701)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x955d [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.96.1 (10.32.96.1)
    Destination: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootpc (68)
    Source port: bootps (67)
    Destination port: bootpc (68)
    Length: 325
    Checksum: 0x202d [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Reply (2)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 2
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 10.32.96.104 (10.32.96.104)
    Next server IP address: 10.32.33.40 (10.32.33.40)
    Relay agent IP address: 10.32.96.1 (10.32.96.1)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name: /undionly.kpxe
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Offer
        Option: (53) DHCP Message Type
        Length: 1
        Value: 02
    Option: (t=54,l=4) DHCP Server Identifier = 10.32.96.1
        Option: (54) DHCP Server Identifier
        Length: 4
        Value: 0A206001
    Option: (t=51,l=4) IP Address Lease Time = 1 hour
        Option: (51) IP Address Lease Time
        Length: 4
        Value: 00000E10
    Option: (t=118,l=4) Subnet Selection Option = 10.32.96.0
        Option: (118) Subnet Selection Option
        Length: 4
        Value: 0A206000
    Option: (t=58,l=4) Renewal Time Value = 30 minutes
        Option: (58) Renewal Time Value
        Length: 4
        Value: 00000708
    Option: (t=59,l=4) Rebinding Time Value = 52 minutes, 30 seconds
        Option: (59) Rebinding Time Value
        Length: 4
        Value: 00000C4E
    Option: (t=1,l=4) Subnet Mask = 255.255.240.0
        Option: (1) Subnet Mask
        Length: 4
        Value: FFFFF000
    Option: (t=28,l=4) Broadcast Address = 10.32.111.255
        Option: (28) Broadcast Address
        Length: 4
        Value: 0A206FFF
    Option: (t=15,l=13) Domain Name = "myvrf.internal"
        Option: (15) Domain Name
        Length: 13
        Value: 6465762E692E657763732E6368
    Option: (t=6,l=8) Domain Name Server
        Option: (6) Domain Name Server
        Length: 8
        Value: 0A08034A0A0803AE
        IP Address: 10.8.3.74
        IP Address: 10.8.3.174
    Option: (t=3,l=4) Router = 10.32.96.1
        Option: (3) Router
        Length: 4
        Value: 0A206001
    End Option

Step 5: DHCPREQUEST on relay

Then the client generates the DHCPREQUEST which hits the relay looking like this:

  • Broadcast from 0.0.0.0 to 255.255.255.255
  • Client IP Address is all zeros
  • Next Server IP address is all zeros
  • Relay Agent IP address is all zeros
  • Option 54 (Server ID) is set to 10.32.96.1
  • And there is a requested IP of 10.32.96.104

(why are there some sections always refusing to accept the pre-format-tags?)

Internet Protocol, Src: 0.0.0.0 (0.0.0.0), Dst: 255.255.255.255 (255.255.255.255)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 351
    Identification: 0x6cdc (27868)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x4db2 [correct]
        [Good: True]
        [Bad : False]
    Source: 0.0.0.0 (0.0.0.0)
    Destination: 255.255.255.255 (255.255.255.255)
User Datagram Protocol, Src Port: bootpc (68), Dst Port: bootps (67)
    Source port: bootpc (68)
    Destination port: bootps (67)
    Length: 331
    Checksum: 0x0453 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 0
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 0.0.0.0 (0.0.0.0)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Request
        Option: (53) DHCP Message Type
        Length: 1
        Value: 03
    Option: (t=57,l=2) Maximum DHCP Message Size = 1200
        Option: (57) Maximum DHCP Message Size
        Length: 2
        Value: 04B0
    Option: (t=61,l=25) Client identifier
        Option: (61) Client identifier
        Length: 25
        Value: 00636973636F2D373037392E623366662E353134322D4769...
    Option: (t=54,l=4) DHCP Server Identifier = 10.32.96.1
        Option: (54) DHCP Server Identifier
        Length: 4
        Value: 0A206001
    Option: (t=50,l=4) Requested IP Address = 10.32.96.104
        Option: (50) Requested IP Address
        Length: 4
        Value: 0A206068
    Option: (t=12,l=13) Host Name = "ExpDHCPClient"
        Option: (12) Host Name
        Length: 13
        Value: 45787044484350436C69656E74
    Option: (t=55,l=9) Parameter Request List
        Option: (55) Parameter Request List
        Length: 9
        Value: 01060F2C790321962B
        1 = Subnet Mask
        6 = Domain Name Server
        15 = Domain Name
        44 = NetBIOS over TCP/IP Name Server
        121 = Classless Static Route
        3 = Router
        33 = Static Route
        150 = TFTP server address
        43 = Vendor-Specific Information
    Option: (t=60,l=8) Vendor class identifier = "ciscopnp"
        Option: (60) Vendor class identifier
        Length: 8
        Value: 636973636F706E70
    End Option

Step 6 the relayed DHCPREQUEST

And now things are going beyond me:

  • The relay VXLANifies the request and sends it to the remote egress VTEP (outer headers not shown)
  • The Next Server addess is all zeros
  • the relay agent ip address is set to 10.32.15.39
  • the requested IPO address is kept at 10.32.96.104
  • Option 54 is changed to 10.32.33.40
  • Option 82 and Suboptions are added again.

(aargh.. these preformat-tags are nerve-wrecking) 

Internet Protocol, Src: 10.32.15.39 (10.32.15.39), Dst: 10.32.33.40 (10.32.33.40)
    Version: 4
    Header length: 20 bytes
    Differentiated Services Field: 0x00 (DSCP 0x00: Default; ECN: 0x00)
        0000 00.. = Differentiated Services Codepoint: Default (0x00)
        .... ..0. = ECN-Capable Transport (ECT): 0
        .... ...0 = ECN-CE: 0
    Total Length: 401
    Identification: 0xdc6c (56428)
    Flags: 0x00
        0.. = Reserved bit: Not Set
        .0. = Don't fragment: Not Set
        ..0 = More fragments: Not Set
    Fragment offset: 0
    Time to live: 255
    Protocol: UDP (0x11)
    Header checksum: 0x9960 [correct]
        [Good: True]
        [Bad : False]
    Source: 10.32.15.39 (10.32.15.39)
    Destination: 10.32.33.40 (10.32.33.40)
User Datagram Protocol, Src Port: bootps (67), Dst Port: bootps (67)
    Source port: bootps (67)
    Destination port: bootps (67)
    Length: 381
    Checksum: 0xf675 [validation disabled]
        [Good Checksum: False]
        [Bad Checksum: False]
Bootstrap Protocol
    Message type: Boot Request (1)
    Hardware type: Ethernet
    Hardware address length: 6
    Hops: 1
    Transaction ID: 0x00002384
    Seconds elapsed: 0
    Bootp flags: 0x8000 (Broadcast)
        1... .... .... .... = Broadcast flag: Broadcast
        .000 0000 0000 0000 = Reserved flags: 0x0000
    Client IP address: 0.0.0.0 (0.0.0.0)
    Your (client) IP address: 0.0.0.0 (0.0.0.0)
    Next server IP address: 0.0.0.0 (0.0.0.0)
    Relay agent IP address: 10.32.15.39 (10.32.15.39)
    Client MAC address: 70:79:b3:ff:51:42 (70:79:b3:ff:51:42)
    Client hardware address padding: 00000000000000000000
    Server host name not given
    Boot file name not given
    Magic cookie: (OK)
    Option: (t=53,l=1) DHCP Message Type = DHCP Request
        Option: (53) DHCP Message Type
        Length: 1
        Value: 03
    Option: (t=57,l=2) Maximum DHCP Message Size = 1200
        Option: (57) Maximum DHCP Message Size
        Length: 2
        Value: 04B0
    Option: (t=61,l=25) Client identifier
        Option: (61) Client identifier
        Length: 25
        Value: 00636973636F2D373037392E623366662E353134322D4769...
    Option: (t=54,l=4) DHCP Server Identifier = 10.32.33.40
        Option: (54) DHCP Server Identifier
        Length: 4
        Value: 0A202128
    Option: (t=50,l=4) Requested IP Address = 10.32.96.104
        Option: (50) Requested IP Address
        Length: 4
        Value: 0A206068
    Option: (t=12,l=13) Host Name = "ExpDHCPClient"
        Option: (12) Host Name
        Length: 13
        Value: 45787044484350436C69656E74
    Option: (t=55,l=9) Parameter Request List
        Option: (55) Parameter Request List
        Length: 9
        Value: 01060F2C790321962B
        1 = Subnet Mask
        6 = Domain Name Server
        15 = Domain Name
        44 = NetBIOS over TCP/IP Name Server
        121 = Classless Static Route
        3 = Router
        33 = Static Route
        150 = TFTP server address
        43 = Vendor-Specific Information
    Option: (t=60,l=8) Vendor class identifier = "ciscopnp"
        Option: (60) Vendor class identifier
        Length: 8
        Value: 636973636F706E70
    Option: (t=82,l=48) Agent Information Option
        Option: (82) Agent Information Option
        Length: 48
        Value: 010409010E30020628AC9E65F84F9714004F5330312D5245...
        Agent Circuit ID: 09010E30
        Agent Remote ID: 28AC9E65F84F
        DHCPv4 Virtual Subnet Selection: 004F5330312D52454730312D444556312D5A3030
        Server Identifier Override: 0A206001
        Link selection: 10.32.96.0
    End Option

... and the DHCPREQUEST doesn't even show up in dnsmasq.log. Because of that, we didn't even start (yet) to tcpdump/capture at the server end.

Thanks for your suggestions, ideas, thoughts and pointers...

[1] The given server also happens to be used as HTTP based (not tftp) PXE boot server for the DHCP clients. Yet, I struggle to find anything in its config file clearly showing why an where it would set "next server" - but then again, that part of the problem is server configuration related and must be answered outside of network engineering.

Improve this question
1
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can post and accept your own answer.
    – Ron Maupin ♦
    Dec 16, 2020 at 23:06

2 Answers 2

Reset to default
2

Just wanted to share some info on a similar issue our customer is facing. They have migrated from VSS to NXOS VXLAN fabric and the PXE boot that uses BOOTP from the SCCM server stopped working for their VDI environment. Seems like option 54 override is the cause.

What we noticed is that NXOS does not support BOOTP relay as per their Security Guide. https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/security/configuration/guide/b-cisco-nexus-9000-nx-os-security-configuration-guide-93x/b-cisco-nexus-9000-nx-os-security-configuration-guide-93x_chapter_01110.html#id_72567

These two things might be correlated.

Another related issue with SCCM that needs to forward UDP port 4011 is that ip forward-protocol on NXOS is quite limited and cannot forward any UDP port, but just a particular one. The TAC enhancement is filled in since 2015 but its status is still new.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuu60491

HTH Alexei.

Improve this answer
1

Partial Answer to "Question B"

It's perfectly normal to have the DHCP Server's own IP address in the "Next Server" field in a DHCPOFFER, and this is entirely unrelated to the use of the (same) server as http or tftp boot server.

If that causes the relay to misbehave, well then it would be the relay's problem.

Partial Answer to Question A:

The relay might be operating correctly. With a Windows 2016 Server's DHCP service (I do know that we're touching the borders of offt-topic-ness now), we got this to work today. A final solution this can't be, but at least it works.

Interestingly, the Windows DHCP server at first returned DHCP NAKs after receiving the relayed DHCPREQUEST (whereas dnsmasq dropped the packets silently).

Then we set up a dummy DHCP Scope (as in: all addresses excluded) for an IP range covering all possible DHCP relay's source addresses, and we started getting DHCPACKs.

This explained e.g. here: https://social.technet.microsoft.com/Forums/en-US/20e50652-5a19-4dee-a6af-4c09f3fcfd1b/windows-dhcp-server-replying-with-dhcp-nak-rfc3527-link-selection-suboption?forum=winserveripamdhcpdns which quotes from here:

https://docs.microsoft.com/en-us/windows-server/networking/technologies/dhcp/dhcp-subnet-options

All relay agent IP addresses (GIADDR) must be part of an active DHCP scope IP address range. Any GIADDR outside of the DHCP scope IP address ranges is considered a rogue relay and Windows DHCP Server will not acknowledge DHCP client requests from those relay agents.

A special scope can be created to "authorize" relay agents. Create a scope with the GIADDR (or multiple if the GIADDR's are sequential IP addresses), exclude the GIADDR address(es) from distribution, and then activate the scope. This will authorize the relay agents while preventing the GIADDR addresses from being assigned.

We tried to replicate this "dummy scope" solution on dnsmasq, but so far, we failed.

If at all, we might be taking this discussion/investigation over to other SE site(s), now, as it seems to have become a server side issue.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.