So recently we received notification from one of our remote branches that certain PCs were unable to connect to the internet. We determined that DHCP was the cause. The basic L3 path between the remote branch and the dhcp server is

DHCP Server (VM) - R1 - R2 - ASA - VPN Tunnel - ASA - R3 - Remote Client

We recently had a UPS failure that caused R1 and R2 to reboot. Not sure if that could be relevant.

Anyway, I can ping clients on the remote subnet from our DHCP server fine. I can also tracert to the default gateway for that remote subnet.

R3 is set as a DHCP relay to DHCP Server.

When our client sends out a DHCP DISCOVER it's able to reach our DHCP server. Our DHCP Server then sounds out a DHCP OFFER which is met instantly with:

Network Capture

That screenshot is taken from a packet capture on the DHCP Server. Isn't that time between messages incredibly short for a TTL expiration? Also, this seems to be the only kind of traffic that's unable to pass through.

The ICMP comes from our gateway interface to the ASA.

I'm baffled by this.

Let me know what else I can provide to help.

closed as too broad by Ron Maupin Dec 6 '18 at 15:10

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 2
    Looks like a routing issue. Perhaps when the routers rebooted, there was an unsaved configuration that was lost. In order for us to help you we will need a simple diagram of your network and the configurations of the routers and ASA. – Ron Trunk Dec 5 '18 at 19:06
  • Looks like routing loop to me too. The DHCP OFFER should be unicast to R3: can you ping both sides of R3 from DHCP server? – jonathanjo Dec 5 '18 at 19:23
  • TTL is number of hops, so not really related to wall clock time. My question would be what's going on at R3 -- that's the thing relaying (unicast) to the server, and the server should be sending a reply directly (unicast) back. – Ricky Beam Dec 5 '18 at 20:48
  • @RickyBeam I believe the questioner is referring to the 28 ms between DHCP OFFER and ICMP TTL EXPIRED, which means the 255-hops are approx 0.1 ms each. (or however much TTL the OFFER is sent with) – jonathanjo Dec 5 '18 at 21:09
  • 28ms before TTL expired doesn't sound surprising to me. In fact it indicates to me that the routing loop is on the server side of the ASA (assuming the VPN tunnel has much higher latency than the rest of the interconnects). Also TTL of desktop OS's are often set to 128 or 64, so it takes even less time to expire the TTL. – hopla Dec 5 '18 at 21:30

This looks like a routing loop from the DHCP server to the client network R3. You need to try whether you can ping the same path or use tracert/traceroute to find the looping hop.

In case ping and tracert don't show the loop, there must be a policy route causing the loop. Linux traceroute is able to simulate a DHCP offer (-p 68); there are tools for Windows for this as well.

edit: of course, the offer is directed back to the relay agent, not the client directly

  • Thanks for this. I know that the route normally works because as I said in OP that I can tracert as expected from the DHCP server to the inside of the remote router R3. I think a policy loop is a good route to explore and I'm looking for a tool now to tracert a simulated dhcp offer for windows. – Nick Sea Dec 5 '18 at 20:31
  • I believe the DHCP offer is sent to the R3's DHCP side address (left in your diagram). I'd just check the wireshark packet trace for this specifically, and check ping of that address, rather than the client LAN addresses (right) you mention in the question. – jonathanjo Dec 5 '18 at 21:17
  • The Destination IP of the Offer is the right side of R3. – Nick Sea Dec 5 '18 at 21:24
  • @Nick Sea: do you have any policy based forwarding configured on the routers? If not, then I don't see any reason why DHCP would be treated differently. I'm just as baffled as you, but these baffling problems often lead to the best learning moments :) – hopla Dec 5 '18 at 21:35
  • 3
    @NickSea No, it can't, unless you have some policy routing configured on it.. You really need to provide us with the router and ASA configurations. Otherwise, we're just guessing in the dark. A listing of the routing table from the routers would also be helpful. – Ron Trunk Dec 5 '18 at 21:59

Not the answer you're looking for? Browse other questions tagged or ask your own question.