Cisco 5510 between-interface traffic

Question

Let me try rewriting the problem description and see if it helps...

My Cisco 5510 is currently running with the out-of-the-box default access rules

I'm running the ASA's DHCP server to assign addresses to devices within each downstream port's subnet, and using its DHCP client to obtain the outside-facing address from upstream. The security level and address ranges assigned to each port are:

management: 100, 172.17.17.0/255.255.255.0
inside:     90, 172.17.1.0/255.255.255.0
public:     50, 172.17.3.0/255.255.255.0
DMZ:        25, 172.17.2.0/255.255.255.0
outside:    0 (Set from upstream DHCP)

The good news: Those default rules are correctly letting all my subnets reach the public internet, presumably because outside is indeed a "less secure network" than they are. Ping, HTTP, ssh, you name it.

The bad news: You would think that, if these basic access control rules let me reach Outside, they would also let me connect from (for example) the machine 172.17.1.101 attached to "inside" (port 1) to 172.17.3.20 attached to "public" (port 3). That would make sense; partitioning by subnet security level must be the single most common configuration for these firewalls. But for whatever reason, they don't. Packet tracing suggests that the request is going out correctly, but the ASA's protocol recognition (which is supposed to automatically open a path for the response) is not letting the connection be completed.

I've been flailing around trying alternative ACL settings without progress. It's as if the ASA isn't correctly implementing "any less secure network" for anything but the zero-security network. Unfortunately putting in explicit rules for the individual ports hasn't yet seemed to improve things. I've made this work before, but apparently I've forgotten some bit of necessary configuration.

Can anyone help me figure out what my stupid error is?

---

MODEL_FAMILY_NAME ASA
ASDM_VERSION 7.6(1)
PLATFORM_BASE_VERSION 9.1
PLATFORM_VERSION 9.1(7)16
ASDM_BASE_VERSION 7.6xp

---

[redacted] indicates information deleted for security reasons.

: Saved
: 
: Serial Number: JMX1546X0SJ
: Hardware:   ASA5510, 1024 MB RAM, CPU Pentium 4 Celeron 1600 MHz
: Written by enable_15 at 20:41:09.377 EDT Sat Aug 12 2023
!
ASA Version 9.1(7)16 
!
hostname casadelgato-firewall
domain-name kubyc.solutions
enable password [redacted] encrypted
multicast-routing
names
dns-guard
!
interface Ethernet0/0
 description External network.
 nameif outside
 security-level 0
 ip address dhcp setroute 
 ipv6 enable
!
interface Ethernet0/1
 description Kubyc.Solutions development subnet
 nameif inside
 security-level 80
 ip address 172.17.1.1 255.255.255.0 
 ipv6 enable
 ipv6 nd managed-config-flag
 ipv6 nd other-config-flag
!
interface Ethernet0/2
 description Servers visible from outside or inside, but only able to initiate contact with outside.
 nameif DMZ
 security-level 20
 ip address 172.17.2.1 255.255.255.0 
 ipv6 enable
!
interface Ethernet0/3
 description Home Automation, printers, guests
 nameif public
 security-level 60
 ip address 172.17.3.1 255.255.255.0 
 ipv6 enable
!
interface Management0/0
 description Privilaged subnet; ASA console access, fallback to reach ASA if "inside" is messed up.
 nameif management
 security-level 99
 ip address 172.17.17.1 255.255.255.0 
 ipv6 enable
 ipv6 nd managed-config-flag
 ipv6 nd other-config-flag
!
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns domain-lookup outside
dns domain-lookup inside
dns domain-lookup DMZ
dns domain-lookup public
dns domain-lookup management
dns server-group DefaultDNS
 name-server 9.9.9.9
 name-server 8.8.8.8
 name-server 8.8.8.4
 name-server 208.67.220.220
 name-server 208.67.222.222
 domain-name kubyc.solutions
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object-group protocol TCPUDP
 protocol-object udp
 protocol-object tcp
object-group protocol TCPUDPIP
 description Alternative to TCPUDP. Overkill?
 protocol-object ip
 group-object TCPUDP
object-group icmp-type ping
 description ICMP echo and echo-reply
 icmp-object echo
 icmp-object echo-reply
object-group protocol DM_INLINE_PROTOCOL_1
 protocol-object udp
 protocol-object tcp
access-list global-access extended permit icmp any any 
pager lines 24
logging enable
logging asdm warnings
mtu outside 1500
mtu inside 1500
mtu DMZ 1500
mtu public 1500
mtu management 1500
ip verify reverse-path interface outside
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp inside 192.170.1.58 d807.b6d7.04ab 
arp timeout 60
no arp permit-nonconnected
nat (any,outside) source dynamic any interface dns
ipv6 enforce-eui64 inside
ipv6 enforce-eui64 management
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
user-identity default-domain LOCAL
http server enable
http 172.17.1.0 255.255.255.0 inside
http 172.17.17.0 255.255.255.0 management
no snmp-server location
no snmp-server contact
sysopt noproxyarp outside
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpoint _SmartCallHome_ServerCA
 no validation-usage
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_0
 enrollment self
 fqdn none
 subject-name CN=192.170.17.1,CN=casadelgato-firewall
 keypair ASDM_LAUNCHER
 crl configure
crypto ca trustpool policy
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_0
 certificate 30123e
    [redacted]
  quit
ssh stricthostkeycheck
ssh 172.17.17.0 255.255.255.0 management
ssh timeout 5
ssh key-exchange group dh-group14-sha1
console timeout 0
management-access management
dhcp-client client-id interface outside
dhcpd auto_config outside
dhcpd update dns both 
!
dhcpd address 172.17.1.100-172.17.1.254 inside
dhcpd update dns both interface inside
dhcpd enable inside
!
dhcpd address 172.17.2.100-172.17.2.254 DMZ
dhcpd update dns both interface DMZ
dhcpd enable DMZ
!
dhcpd address 172.17.3.50-172.17.3.254 public
dhcpd update dns both interface public
dhcpd enable public
!
dhcpd address 172.17.17.100-172.17.17.254 management
dhcpd update dns both interface management
dhcpd enable management
!
threat-detection basic-threat
threat-detection scanning-threat
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
ntp server 216.239.35.0 source outside
ntp server 204.11.201.10 source outside
ntp server 163.237.218.19 source outside
ntp server 173.0.48.220 source outside
ntp server 108.61.73.244 source outside
ssl trust-point ASDM_Launcher_Access_TrustPoint_0 management vpnlb-ip
ssl trust-point ASDM_Launcher_Access_TrustPoint_0 management
quota management-session 10
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ip-options 
  inspect icmp 
  inspect icmp error 
  inspect http 
!
service-policy global_policy global
prompt hostname context 
no call-home reporting anonymous
hpm topN enable
Cryptochecksum:[redacted]
: end

---

UPDATE: I tried using ASDM to add an incoming rule to outside saying

any any icmp permit

That does improve tracert to the outside network. But outside isn't my problem; I can already HTTP etc. to anything in the outside world. What I can't do is ssh or ping between my own networks, eg inside to public. (Or tracert, but tracert is not my immediate priority!)

Why does "any/any less secure network/ip" work from anything to outside, but not from anything to anything on another local port?

---

UPDATE 2: Tried using ASDM's diagnostics.

Packet Tracer: In TCP mode, from inside interface address 172.17.1.100 to 172.17.3.20 port 22: It says the packet is allowed and routed from inside to public. A flow is created. So it claims it's attempting to create my ssh connection.

OK, lemme confirm 3.20 is in fact live... It is. And it can connect to the outside world. So, yes, the problem is entirely between my local subnets/interfaces.

---

UPDATE 3: Part of what's confusing me is that the default ACLs, "any, any less secure, permit", define "any less secure" as "all of the following less secure networks", such as "public:any,DMZ:any,outside:any". But my connectivity to outside is working, and connectivity to public isn't.

Yet if I replace that with "any, outside, permit", I lose the connectivity to outside.

Assuming that the ACL isn't lying to me, what could affect connectivity other than the ACL? Network address translation, given that all these subnets are using a 255.255.255.0 mask? Proxy ARP? I'm really running out of ideas to try.

"It happened once. Therefore it must be possible."

---

UPDATE 4: I have just reproduced the problem with Linux on both ends. (antiX as the client on the more-secure network, Raspbian as the server, if anyone cares.) With the NAT workaround, traffic goes through as expected. Remove network address translation, and neither ping nor ssh succeed. I think that eliminates Windows' internal firewall as a contributing factor.

There may still be some systematic issue in the network parameters being used, perhaps in the ones being pushed out from the Cisco via DHCP.

I have no explanation for why Ricky has been unable to reproduce it when loading my configuration into his 5540, unless some bit of configuration in that box which wasn't overwritten is the key to resolving the problem. He has sent me a copy of that combined config, but loading that into my 5510 (minus a few things I didn't want to overwrite) produced the same failure I've been seeing. So either something is more different between the 5510 and 5540 than we've accounted for (I think we're on the same firmware level...?), or I'm just haunted.

Next step may have to be to sic wireshark and/or the ASA's internal monitoring on the problem and look at exactly what packets are actually moving through the two LANs.

I'm still betting that if/when we find it we're going to kick ourselves for not having spotted the obvious error.

Answer 1

Higher security interfaces can always talk to lower security interfaces, unless an ACL blocks it.

The most likely issue is the lack of any active rules for access-list outside_access_in If you look at this ACL in ASDM, the implicit "deny any any" will be shown. Without any "permits", that ACL should be blocking everything. And it will...

E:\>tracert -d networkengineering.stackexchange.com

Tracing route to networkengineering.stackexchange.com [151.101.193.69]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  172.17.1.1
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4     *        *        *     Request timed out.
  5     *        *        *     Request timed out.
  6     *        *        *     Request timed out.
  7     *        *        *     Request timed out.
  8     *        *        *     Request timed out.
  9     *        *        *     Request timed out.
 10     *        *        *     Request timed out.
 11    28 ms    26 ms    26 ms  151.101.193.69

Trace complete.

add access-list outside_access_in extended permit icmp any4 any4 and traceroute will magically start working.

E:\>tracert -d networkengineering.stackexchange.com

Tracing route to networkengineering.stackexchange.com [151.101.193.69]
over a maximum of 30 hops:

  1    <1 ms    <1 ms    <1 ms  172.17.1.1
  2    21 ms     9 ms     7 ms  192.168.55.1
  3     1 ms    <1 ms    <1 ms  [...]
  4    12 ms    14 ms    12 ms  [ISP]
  5    21 ms    16 ms    15 ms  174.111.105.16
  6    19 ms    13 ms    17 ms  24.25.41.106
  7    15 ms    14 ms    11 ms  24.93.64.184
  8    28 ms    32 ms    29 ms  107.14.18.106
  9    28 ms    29 ms    29 ms  209.18.43.59
 10    43 ms    27 ms    33 ms  140.248.127.82
 11    28 ms    25 ms    27 ms  151.101.193.69

Trace complete.

I also added:

policy-map global_policy
 class class-default
  set connection decrement-ttl
!
icmp unreachable rate-limit 5 burst-size 10
icmp permit any outside
icmp permit any inside
icmp permit any public
icmp permit any DMZ
icmp permit any management

---

I've loaded your provided configuration as-is (password added) and have zero problems. Traceroute needs some work, but traffic does flow across the fw as it should.

[ System image file is "disk0:/asa917-32-k8.bin" ]

Gateway of last resort is 192.168.55.1 to network 0.0.0.0

C    172.17.17.0 255.255.255.0 is directly connected, management
C    172.17.1.0 255.255.255.0 is directly connected, inside
C    192.168.55.0 255.255.255.0 is directly connected, outside
d*   0.0.0.0 0.0.0.0 [1/0] via 192.168.55.1, outside

Current IP Addresses:
Interface                Name                   IP address      Subnet mask     Method 
GigabitEthernet0/0       outside                192.168.55.65   255.255.255.0   DHCP  
GigabitEthernet0/1       inside                 172.17.1.1      255.255.255.0   CONFIG
GigabitEthernet0/2       DMZ                    172.17.2.1      255.255.255.0   CONFIG
GigabitEthernet0/3       public                 172.17.3.1      255.255.255.0   CONFIG
Management0/0            management             172.17.17.1     255.255.255.0   CONFIG

(from the fw directly to my laptop :: drop)

kubyc-firewall# packet-tracer input management icmp 172.17.17.1 8 0 44 172.17.1.100

Phase: 1
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   172.17.1.0      255.255.255.0   inside

Phase: 3
Type: ACCESS-LIST
Subtype: 
Result: DROP
Config:
Implicit Rule
Additional Information:
              
Result:
input-interface: management
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

(from a machine inside the MGMT network :: allow)

kubyc-firewall# packet-tracer input management icmp 172.17.17.10 8 0 44 172.17.1.100

Phase: 1
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   172.17.1.0      255.255.255.0   inside

Phase: 2
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW 
Config:
Additional Information:

Phase: 4
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:

Phase: 6      
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: IP-OPTIONS
Subtype: 
Result: ALLOW 
Config:
Additional Information:

Phase: 10
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: FLOW-CREATION
Subtype: 
Result: ALLOW
Config:
Additional Information:
New flow created with id 585, packet dispatched to next module

Result:
input-interface: management
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: allow

(ssh from my laptop to a machine on the "outside" :: allow)

kubyc-firewall# packet-tracer input inside tcp 172.17.1.100 8888 10.0.0.3 22 

Phase: 1
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: input
Result: ALLOW
Config:
Additional Information:
in   0.0.0.0         0.0.0.0         outside

Phase: 3
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:

Phase: 4
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (any,outside) source dynamic any interface dns
Additional Information:
Dynamic translate 172.17.1.100/8888 to 192.168.55.65/8888

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (any,outside) source dynamic any interface dns
Additional Information:

Phase: 8
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: FLOW-CREATION
Subtype: 
Result: ALLOW
Config:
Additional Information:
New flow created with id 636, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Answer 2

I've taken it to https://community.cisco.com/t5/network-security/nat-control-deprecated-any-equivalent-moved-here/m-p/4907971 -- though once I've confirmed a solution I'll probably retitle that.

---

LATE UPDATE: We seem to have found it. Somehow, the PCs on the "public" and "DMZ" interfaces are winding up with a subnet mask of 255.255.0.0, the /16 mask. Yes, that's typically what a 172... address is used for, though that wasn't my intent this time and isn't what's specified in the interface parameters.

I don't know where that 0 in the third octet is coming from. But this means those PCs aren't recognizing that packets outside their /24 range need to be sent though the gateway to reach the other interfaces and the machines attached thereunto, and hence communications is being dropped on the floor.

Possible Fixes:

1. Resolve why we're getting a subnet mask other than I intended, and fix that.
2. Go back to 192.168 address range, where /24 is the assumption, and hope the subnet masks suddenly become /24 as well. I don't have any good reason not to, I'm just annoyed at being quite so tightly locked in my local addressing.
3. Separate the interface address ranges in the second octet rather than the third, so /16 is correct. Since the public range is 172.16 to 172.31, I'd need to encode my current "address indicates interface number", moving to something like 172.20, 172.21, 172.22, 172.23, and something vaguely appropriate for the Management interface. This might be a least-change solution if (1) is not possible.
4. Oh, almost forgot: Use NAT.