Cisco ASA not accepting external traffic without NAT configuration

Question

First of all, I am quite newie in the cisco world, so this could be some easy stuff that I'm missing.

My first problem is trying to understand why a Cisco ASA5585 does not show traffic while doing a capture. The capture is quite straightforward

capture casa interface OUTSIDE match ip 9.18.28.24 255.255.255.255 7.1.4.17 255.255.255.255

Being 9.18.28.24 my ip and 7.1.4.17 one ip routed to the cisco asa. Reading this ASA troubleshooting guide (pag.62) it says that capture is done before anything else, so I do not understand why it does not shows the packets.

I have configured some NAT and its working, so I started to play with NAT configs for this ip (7.1.4.17) and I found that after the next config it is show in the network capture.

object network bgpopen
 host 10.10.10.10
 nat (INSIDE,OUTSIDE) static 7.1.4.17 service tcp 100 100

IP 10.10.10.10 is nothing in my network, I just added it to generate a valid config. Same with TCP port 100.

Once this config is running I can see a packet coming from my IP but nothing returning. If I ask to the packet-tracer it drops the packet because:

Drop-reason: (nat-no-xlate-to-pat-pool) Connection to PAT address without pre-existing xlate

And finally, what astonish me the most, if I remove that network object:

no object network bgpopen

I can finally send requests to the 7.1.4.17 (there is a web server listening).

After a while that "hole" is closed and I'm back at the beggining, without seeing the packets in the capture.

Thanks!

Edit: added the full running configuration:

! 
! Cisco Adaptive Security Appliance Software Version 9.12(4)40 
! SSP Operating System Version 2.6(1.254)
! Device Manager Version 6.4(5)
! 
! Compiled on Wed 06-Apr-22 04:36 GMT by builders
! System image file is "disk0:/asa9-12-4-40-smp-k8.bin"
! Config file at boot was "startup-config"
! 
! 
! Hardware:   ASA5585-SSP-40, 11849 MB RAM, CPU Xeon 5500 series 2133 MHz, 2 CPUs (16 cores)
! Internal ATA Compact Flash, 2048MB
! BIOS Flash M25P32 @ 0x0, 4096KB
! 
! Encryption hardware device : Cisco ASA-5585 on-board accelerator (revision 0x1)
!                              Boot microcode        : CNPx-MC-BOOT-2.00
!                              SSL/IKE microcode     : CNPx-MC-SSL-SB-PLUS-0005
!                              IPSec microcode       : CNPx-MC-IPSEC-MAIN-0026
!                              Number of accelerators: 3
! 
! Programmable device : CPLD revision 0x8
! 
!  0: Int: Internal-Data0/0    : address is 0000.0001.0001, irq 5
!  2: Int: Internal-Data0/1    : address is 0000.0001.0002, irq 10
!  3: Int: Internal-Data0/2    : address is 0000.0001.0004, irq 10
!  4: Int: Internal-Data0/3    : address is 0000.0001.0003, irq 5
!  5: Ext: Management0/0       : address is c464.1366.e330, irq 10
!  6: Ext: Management0/1       : address is c464.1366.e331, irq 5
! 18: Ext: GigabitEthernet0/0  : address is c464.1366.e332, irq 255
! 19: Ext: GigabitEthernet0/1  : address is c464.1366.e333, irq 255
! 20: Ext: GigabitEthernet0/2  : address is c464.1366.e334, irq 255
! 21: Ext: GigabitEthernet0/3  : address is c464.1366.e335, irq 255
! 22: Ext: GigabitEthernet0/4  : address is c464.1366.e336, irq 255
! 23: Ext: GigabitEthernet0/5  : address is c464.1366.e337, irq 255
! 24: Ext: TenGigabitEthernet0/6: address is c464.1366.e338, irq 255
! 25: Ext: TenGigabitEthernet0/7: address is c464.1366.e339, irq 255
! 26: Ext: TenGigabitEthernet0/8: address is c464.1366.e33a, irq 255
! 27: Ext: TenGigabitEthernet0/9: address is c464.1366.e33b, irq 255
! 28: Int: Internal-Data0/4    : address is 0000.0100.001d, irq 255
! 29: Int: Internal-Data0/5    : address is 0000.0100.001e, irq 255
! 30: Int: Internal-Data0/6    : address is 0000.0100.001f, irq 255
! 31: Int: Internal-Data0/7    : address is 0000.0100.0020, irq 255
! 32: Int: Not used            : irq 255
! 33: Int: Not used            : irq 255
! 34: Int: Not used            : irq 255
! 35: Int: Not used            : irq 255
! 36: Int: Not used            : irq 255
! 37: Int: Not used            : irq 255
! 
! Licensed features for this platform:
! Maximum Physical Interfaces       : Unlimited      perpetual
! Maximum VLANs                     : 1024           perpetual
! Inside Hosts                      : Unlimited      perpetual
! Failover                          : Active/Active  perpetual
! Encryption-DES                    : Enabled        perpetual
! Encryption-3DES-AES               : Enabled        perpetual
! Security Contexts                 : 2              perpetual
! Carrier                           : Disabled       perpetual
! AnyConnect Premium Peers          : 2              perpetual
! AnyConnect Essentials             : Disabled       perpetual
! Other VPN Peers                   : 10000          perpetual
! Total VPN Peers                   : 10000          perpetual
! AnyConnect for Mobile             : Disabled       perpetual
! AnyConnect for Cisco VPN Phone    : Disabled       perpetual
! Advanced Endpoint Assessment      : Disabled       perpetual
! Shared License                    : Disabled       perpetual
! Total TLS Proxy Sessions          : 2              perpetual
! Botnet Traffic Filter             : Disabled       perpetual
! 10GE I/O                          : Enabled        perpetual
! Cluster                           : Disabled       perpetual
! 
! This platform has an ASA5585-SSP-40 VPN Premium license.
! 
! Serial Number: JAF1553ALFJ
! Running Permanent Activation Key: 0x9b13f96c 0xe8114e7d 0x0191a9ac 0xca3084dc 0x0d3ef080 
! Configuration register is 0x41
! 
! Image type          : Release
! Key version         : A
! 
! Name: "module 0", DESCR: "ASA 5585-X Security Services Processor-40 w 6GE,4 SFP+"
! PID: ASA5585-SSP-40    , VID: V04     , SN: JAF1553ALFJ
! 
! Name: "Chassis", DESCR: "ASA 5585-X"
! PID: ASA5585           , VID: V02     , SN: JMX1615704L
! 
! Name: "TenGigabitEthernet0/6", DESCR: "10G Based-ZR"
! PID: SFP-H10GB-CU3M    , VID: V03 , SN: CSC210.2.30016 
! 
! Name: "TenGigabitEthernet0/7", DESCR: "10G Based-ZR"
! PID: SFP-H10GB-CU3M    , VID: V03 , SN: CSC210.2.30022 
! 
! Name: "TenGigabitEthernet0/8", DESCR: "10G Based-ZR"
! PID: SFP-H10GB-CU3M    , VID: V03 , SN: CSC210.2.30038 
! 
! Name: "TenGigabitEthernet0/9", DESCR: "10G Based-ZR"
! PID: SFP-H10GB-CU3M    , VID: V03 , SN: CSC210.2.30080 
! 
! Name: "power supply 0", DESCR: "ASA 5585-X AC Power Supply"
! PID: ASA5585-PWR-AC    , VID: V03     , SN: POG160900HT
! 
! Name: "power supply 1", DESCR: "ASA 5585-X AC Power Supply"
! PID: ASA5585-PWR-AC    , VID: V03     , SN: POG160900K1
! 
!
ASA Version 9.12(4)40 
!
hostname ROUTER
domain-name example.com
enable password $sha512$ pbkdf2
service-module 0 keepalive-timeout 4
service-module 0 keepalive-counter 6
service-module 1 keepalive-timeout 4
service-module 1 keepalive-counter 6
names
no mac-address auto

!
interface GigabitEthernet0/0
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet0/1
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet0/2
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet0/3
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet0/4
 shutdown
 no nameif
 no security-level
 no ip address
!
interface GigabitEthernet0/5
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Management0/0
 no management-only
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Management0/1
 no management-only
 shutdown
 no nameif
 no security-level
 no ip address
!
interface TenGigabitEthernet0/6
 description SW-AGG_Te0/1 OUTSIDE
 nameif OUTSIDE
 security-level 0
 ip address 7.1.4.133 255.255.255.240 
!
interface TenGigabitEthernet0/7
 nameif INSIDE
 security-level 100
 ip address 10.2.3.1 255.255.255.0 
!
interface TenGigabitEthernet0/8
 shutdown
 no nameif
 no security-level
 no ip address
!
interface TenGigabitEthernet0/9
 shutdown
 no nameif
 no security-level
 no ip address
!
ftp mode passive
dns server-group DefaultDNS
 domain-name example.com
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network 10.2.3-nat
 subnet 10.2.3.0 255.255.255.0
object network k8s-ingress-public-external-ip
 host 7.1.4.134
object network k8s-ingress-public
 host 10.2.3.138
object network k8s-ingress-public-https
 host 10.2.3.138
object network metallb-7.1.4.135
 host 7.1.4.135
object network 10.2.3
 range 10.2.3.0 10.2.3.255
object network k8s-ingress-pomerium
object network host-10.5.0.52
 host 10.5.0.52
object network ip-7.1.4.135
 host 7.1.4.135
object network host-10.5.0.52-https
object network host-10.5.0.52-80
 host 10.5.0.52
object network host-10.5.0.52-443
 host 10.5.0.52
access-list outside_acl_k8s_ingress_public extended permit tcp any object k8s-ingress-public 
access-list outside_acl_k8s_ingress_public extended permit tcp any host 7.1.4.135 
access-list outside_acl_k8s_ingress_public extended permit tcp any any 
access-list outside_acl_k8s_ingress_public extended permit icmp any any 
access-list outside-acl-metallb-7.1.4.135 extended permit tcp any object metallb-67.211.34.135 
access-list outside_acl_any_to_any extended permit tcp any any 
access-list pruebas extended permit tcp host 10.2.3.17 host 10.5.0.50 eq www 
access-list pruebas extended permit tcp any host 10.5.0.50 
access-list pruebas extended permit tcp any host 7.1.4.135 
access-list pruebas extended permit tcp any any 
access-list pruebas extended permit udp any any 
access-list pruebas extended permit icmp any any 
access-list metallb-public-out extended permit tcp host 7.1.4.135 any 
access-list metallb-public-out extended permit tcp any any 
pager lines 24
mtu INSIDE 1500
mtu OUTSIDE 1500
no failover
no failover wait-disable
no monitor-interface service-module 
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
arp rate-limit 32768
!
object network 10.2.3-nat
 nat (INSIDE,OUTSIDE) dynamic interface
object network k8s-ingress-public
 nat (INSIDE,OUTSIDE) static k8s-ingress-public-external-ip service tcp 32080 www 
object network k8s-ingress-public-https
 nat (INSIDE,OUTSIDE) static k8s-ingress-public-external-ip service tcp 32443 https 
object network host-10.5.0.52
 nat (INSIDE,OUTSIDE) static ip-7.1.4.135 service tcp www www 
object network host-10.5.0.52-80
 nat (INSIDE,OUTSIDE) static ip-7.1.4.135 service tcp www www 
object network host-10.5.0.52-443
 nat (INSIDE,OUTSIDE) static ip-7.1.4.135 service tcp https https 
access-group pruebas in interface INSIDE
access-group metallb-public-out out interface INSIDE
access-group outside_acl_k8s_ingress_public in interface OUTSIDE
access-group pruebas out interface OUTSIDE
router bgp 65100
 bgp log-neighbor-changes
 bgp router-id 10.2.3.1
 address-family ipv4 unicast
  neighbor 10.2.3.138 remote-as 65200
  neighbor 10.2.3.138 activate
  neighbor 10.2.3.140 remote-as 65200
  neighbor 10.2.3.140 activate
  no auto-summary
  no synchronization
 exit-address-family
!
route OUTSIDE 0.0.0.0 0.0.0.0 7.1.4.129 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 sctp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
timeout conn-holddown 0:00:15
timeout igp stale-route 0:01:10
user-identity default-domain LOCAL
aaa authentication ssh console LOCAL 
aaa authentication http console LOCAL 
aaa authorization exec authentication-server auto-enable
aaa authentication login-history
snmp-server host INSIDE 10.2.3.16 community XXX version 2c
snmp-server host-group INSIDE 10.2.3 poll community XXX version 2c
no snmp-server location
no snmp-server contact
snmp-server community XXX
crypto ipsec security-association pmtu-aging infinite
crypto ca trustpool policy
telnet timeout 5
ssh stricthostkeycheck
ssh pubkey-chain
  server 10.2.3.16
    key-hash sha256 b3:a0:32:33:af:34:07:24:4c:84:0e:a8:1d:5d:f2:ac:69:a8:39:b6:45:b8
ssh timeout 5
ssh version 2
ssh key-exchange group dh-group14-sha256
ssh 10.2.3.0 255.255.255.0 INSIDE
console timeout 0
!
tls-proxy maximum-session 1000
!
threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
dynamic-access-policy-record DfltAccessPolicy
username sshadmin password $sha512$5 pbkdf2 privilege 15
username sshadmin attributes
 ssh authentication publickey 8b:27:70:4d:78:81:27:61:95:ae:ea:0f:d3:96:74:50:c5:4a:68:ce:ba:ec hashed
username oxidized password $sha512$5 pbkdf2 privilege 15
!
!
prompt hostname context 
no call-home reporting anonymous
call-home
 profile CiscoTAC-1
  no active
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:10fb9fee94f8e54eff2d

Edit 2: After adding and removing the object network bgpopen the connections works for 20 minutes.

Edit 3: I have found why adding and removing the NAT make it works. The key is the ARP traffic. The cisco asa is not answering to ARP traffic for IP 7.1.4.17 until the NAT is configured for that IP. Once NAT is removed, the upstream route knows how to send traffic to 7.1.4.17 because it has cached the ARP response, so it works. Cisco doc explaning the behaviour of ASA responding to IPs configured in NAT.

Answer 1

After doing some tests with GNS3 and talking with the owner of the upstream router I have clarified all the problems.

My first problem, not being able to capture traffic, was because I was filtering too much. I was missing the ARP requests coming from the upstream router not being answered by my Cisco ASA.

Then I added NAT for the public ip (7.1.4.17). Doing that instructs the ASA to respond for ARP requests to that IP (cisco doc).

One way to make it works is to configure a static route in the upstream router, like:

ip route add 7.1.4.17/32 via 7.1.4.133

But after talking with the network engineer in charge of the upstream router he convinced me to just define NAT to simplify configuration in their end and delegate all the routing decisions to our end.

P.S.: virtualizing was key to understand what was happening, being able to capture all traffic in different parts of the network.