Cisco Catalyst 9410 - Using Default Gateway versus ARPing for locally connected host

Question

I may be missing something extremely simple here, but im at loss trying to figure out what this Nexus switch is doing.

Here is the topology:

DISCLAIMER! I had made a mistake on the diagram by putting the wrong mask on the firewall, Ron pointed this out and i have updated the diagram.

Note: I'm expecting people to ask why this route exists:

172.16.46.0/23 [1/0] via 172.16.40.1

It's a static route that's completely redundant to the default gateway, yet if i remove this it seems to sever all communication between the 2 VLANs.

---

1) Successful - PC (Orange) tries to ping PC (Green), the flow seems to look like:

[Square brackets indicate possible internal routing]

Echo Request: 
172.16.46.100 -> 172.16.40.1 -> [172.16.40.254 -> 172.16.48.254(?)] -> 172.16.48.100

Echo Reply: 
172.16.48.100 -> [172.16.48.254 -> 172.16.40.254(?)] -> 172.16.40.1 -> 172.16.46.100

Im looking to find out what decision process is taking place on the Nexus(SW1) when going between internal VLAN IP's (if thats whats happening).

---

2) Failed - PC (Orange) tries to ping SW1 interface VLAN2040 (172.16.40.254), denied by firewall (it did not see the echo request):

Echo Request: 172.16.46.100 -> 172.16.40.254 

Echo Reply: 172.16.40.254 -> 172.16.40.1 *Denied*

For this im trying to understand why the Nexus (SW1) is choosing to route to the default gateway instead of ARPing for the locally connected subnet (172.16.40.0/21).

This is my ping results from the PC:

Pinging 172.16.40.254 with 32 bytes of data:
Request timed out.
Request timed out.

This capture shows the firewalls perspective, only seeing the reply:

172.16.40.254 > 172.16.46.100: icmp: echo reply
172.16.40.254 > 172.16.46.100: icmp: echo reply

If I set up a capture type of 'asp-drop', i can confirm the drop reason:

172.16.40.254 > 172.16.46.100: icmp: echo reply
Drop-reason: (inspect-icmp-seq-num-not-matched) ICMP Inspect seq num not matched

---

It's also worth noting that pinging will work in the reverse order, where SW1 can still ping the PC (asymmetrically) even though the PC cannot successfully ping SW1(denied by firewall).

This asymmetry causes ICMP connections to time out on the ASA using default timers (2 seconds):

16:22:57 Built inbound ICMP connection for faddr 172.16.40.254/13 gaddr 172.16.46.100/0 laddr 172.16.46.100/0 type 8 code 0
16:22:59 Teardown ICMP connection for faddr 172.16.40.254/13 gaddr 172.16.46.100/0 laddr 172.16.46.100/0 type 8 code 0

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02

Answer 1

172.16.46.0/23 [1/0] via 172.16.40.1

It's a static route that's completely redundant to the default gateway, yet if i remove this it seems to sever all communication between the 2 VLANs.

No, it's not. 172.16.46.0/23 is a subnet of 172.16.40.0/21. Without that routing entry, SW1 would try to send the packet locally within VLAN2040 - and not use its default gateway. With that entry, it passes the packet to FW1 to sort out. You haven't provided enough detail to tell what's going on there but apparently it does work.

You need to sort out the overlapping subnets (=renumber). That is a bad design generally, and sooner or later lands on your feet. I'd strongly advise to use a single standard network mask/prefix length throughout.

Answer 2

I see two things wrong:

1. Your firewall is misconfigured. The inside interface has the wrong mask. VLAN 2040 is /21, but you've configured it as a /24.

2. The orange PC has the wrong gateway. According to your diagram, the SVI for VLAN 2040 is 172.16.40.254, not 172.16.40.1.