1

We just replaced an ASA5510 with a new ASA5516x. Everything worked smoothly except for traffic between our network and traffic on the customer end of a site-to-site VPN tunnel.

Old system: ASA5510 version 9.1(7)15. New system: ASA5516x version 9.9(1).

Traffic originating from our network to the customer network through the tunnel is fine.

Traffic originating from the customer network that should flow outside of the tunnel (it's destination is a public web server in our network), does not work.

After performing packet captures on both sides of the ASA before the update and after the update reveals the difference:

Before update (working case), this is the partial output of a traced capture. This is the first packet entering the ASA from outside, leaving on the inside, the web servers response entering the ASA and then leaving back to the outside. It heads off to our ISP's router (209.xxx.142.25)

1: 09:26:13.592605       129.xxx.235.132.53828 > 209.xxx.142.28.443: S 3332805073:3332805073(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK> 
2: 09:26:13.592986       129.xxx.235.132.53828 > 192.168.2.28.443: S 3832318623:3832318623(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK> 
..
5: 09:26:13.593429       192.168.2.28.443 > 129.xxx.235.132.53828: S 3052302764:3052302764(0) ack 3832318624 win 14600 <mss 1460,nop,nop,sackOK,nop,wscale 7> 


Phase: 4
Type: ROUTE-LOOKUP
Subtype: output and adjacency
Result: ALLOW
Config:
Additional Information:
found next-hop 209.xxx.142.25 using egress ifc outside
adjacency Active
next-hop mac address 68ef.bd4e.7eff hits 1012402

6: 09:26:13.593505       209.xxx.142.28.443 > 129.xxx.235.132.53828: S 4066137499:4066137499(0) ack 3332805074 win 14600 <mss 1300,nop,nop,sackOK,nop,wscale 7> 

Here you can see in the phase 4 the next hop for the exiting packet is 209.xxx.142.25 (our ISP's gateway).

Here is the same capture after the update.

1: 14:52:12.701349       129.xxx.235.132.61129 > 209.xxx.142.28.443: S 775761873:775761873(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK> 
2: 14:52:12.701639       129.xxx.235.132.61129 > 192.168.2.28.443: S 776220941:776220941(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK> 
3: 14:52:12.701791       192.168.2.28.443 > 129.xxx.235.132.61129: S 533424725:533424725(0) ack 776220942 win 14600 <mss 1460,nop,nop,sackOK,nop,wscale 7> 

Phase: 4
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 129.xxx.235.132 using egress ifc  outside

In this case, it's decided that the next hop is 129.xxx.235.132 (the final destination and not our ISP's router). This is the heart of the problem.

That lead us to check the output of "show route" on the ASA for routes related to this destination:

Before (working):

Gateway of last resort is 209.xxx.142.25 to network 0.0.0.0

S    129.xxx.235.132 255.255.255.255 [1/0] via 209.xxx.142.25, outside

After (failing):

Gateway of last resort is 209.xxx.142.25 to network 0.0.0.0

V        129.xx.235.132 255.255.255.255 
           connected by VPN (advertised), outside

Clearly something has changed between 9.1 and 9.9 ASA version in regards to how routes for traffic for a VPN are handled. How can I adjust my config to accommodate this?

Sanitized Config:

ASA Version 9.1(7)15 
!
terminal width 180
hostname 5516xa
domain-name our.company.com
enable password ***** encrypted
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
passwd ***** encrypted
names
ip local pool vpnpool1 192.168.2.51-192.168.2.90 mask 255.255.255.0
!
interface Ethernet0/0
 nameif inside
 security-level 100
 ip address 192.168.2.98 255.255.255.0 
!
interface Ethernet0/1
 no nameif
 no security-level
 no ip address
!
interface Ethernet0/2
 nameif outside
 security-level 0
 ip address 209.xxx.142.26 255.255.255.248 
!
interface Ethernet0/3
 description LAN/STATE Failover Interface
!
interface Management0/0
 management-only
 nameif management
 security-level 0
 ip address 192.168.6.6 255.255.255.0 standby 192.168.6.7 
!
boot system disk0:/asa917-15-k8.bin
ftp mode passive
clock timezone CST -6
clock summer-time CDT recurring
dns server-group DefaultDNS
 domain-name our.company.com
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network ourcompany-inside-isp
 subnet 192.168.2.0 255.255.255.0
object service ssh
 service tcp source eq ssh 
object network webschedule-server
 host 192.168.2.28
 description Web server.
object network webschedule-server-outside
 host 209.xxx.142.28
 description The public (outside) address of the web schedule server.
object service http
 service tcp source eq www 
object service https
 service tcp source eq https 
object network falcon
 host 192.168.2.14
 description Falcon 
object network nat-ourcompany-at-customer
 host 192.168.5.2
 description our addresses as they appear at customer thru tunnel.
object network obj_any
 subnet 0.0.0.0 0.0.0.0
object network RA-DHCP-Pool
 range 192.168.2.51 192.168.2.90
 description RemoteAccess DHCP Pool
object network customerprd1
 host 129.xxx.235.135
 description Customer Server (prd1)
object network customerprd2
 host 129.xxx.235.134
 description Customer Server (prd2)
object network customertst
 host 129.xxx.235.132
 description Customer Server (tst)
object-group service traceroute udp
 description traceroute udp ports
 port-object range 33434 33534
object-group service allowed_outbound_services tcp
 port-object eq www
 port-object eq https
 port-object eq ssh
 port-object eq 465
 port-object eq 587
 port-object eq smtp
object-group network group-remote-customer
 network-object object customerprd1
 network-object object customerprd2
 network-object object customertst
object-group network group-inhouse-customer
 network-object object nat-ourcompany-at-customer
object-group network DHCP_VPN_Users
 description DHCP_VPN_Users
 network-object object RA-DHCP-Pool
 network-object object webschedule-server
 network-object object falcon

access-list group1_splitTunnelAcl standard permit 192.168.2.0 255.255.255.0 
access-list group1_splitTunnelAcl standard permit host 129.xxx.235.132 
access-list group1_splitTunnelAcl standard permit host 129.xxx.235.134 
access-list group1_splitTunnelAcl standard permit host 129.xxx.235.135 

access-list inside_access_in extended permit udp 192.168.2.0 255.255.255.0 any4 eq domain 
access-list inside_access_in extended permit icmp any4 any4 
access-list inside_access_in extended permit udp any4 any4 object-group traceroute 
access-list inside_access_in extended permit ip 192.168.2.0 255.255.255.0 any4 

access-list outside_210_cryptomap extended permit ip object nat-ourcompany-at-customer object-group group-remote-customer 

access-list outside_access_in extended permit icmp any4 any4 
access-list outside_access_in extended permit udp any4 any4 object-group traceroute 
access-list outside_access_in extended permit tcp any4 object webschedule-server-outside eq https inactive 
access-list outside_access_in extended permit tcp any4 object webschedule-server-outside eq www inactive 
access-list outside_access_in extended permit tcp any4 object webschedule-server eq www 
access-list outside_access_in extended permit tcp any4 object webschedule-server eq https 
access-list outside_access_in extended permit udp any4 192.168.2.0 255.255.255.0 eq ntp 

access-list RA-ACL extended permit ip any4 any4 

pager lines 50
mtu inside 1500
mtu outside 1500
mtu management 1500
failover
failover lan unit primary
failover lan interface failoverlink Ethernet0/3
failover replication http
failover link failoverlink Ethernet0/3
failover interface ip failoverlink 10.1.10.1 255.255.255.0 standby 10.1.10.2
no monitor-interface management
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-762.bin
asdm history enable
arp timeout 14400
no arp permit-nonconnected
nat (inside,any) source static any any destination static ourcompany-inside-isp ourcompany-inside-isp no-proxy-arp route-lookup
nat (inside,outside) source static webschedule-server webschedule-server-outside service http http
nat (inside,outside) source static webschedule-server interface service http http
nat (inside,outside) source static webschedule-server webschedule-server-outside service https https
nat (inside,outside) source static webschedule-server interface service https https
nat (inside,inside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer
nat (inside,outside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer
nat (outside,outside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer
!
object network ourcompany-inside-isp
 nat (outside,outside) dynamic interface
object network obj_any
 nat (inside,outside) dynamic interface
access-group inside_access_in in interface inside
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 209.xxx.142.25 1 
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
ldap attribute-map ldapmap2
  map-name  memberOf Group-Policy
  map-value memberOf CN=FullVPN,CN=Users,DC=,DC=us,DC=com group2
dynamic-access-policy-record DfltAccessPolicy
aaa-server Radius protocol radius
aaa-server Radius (inside) host 192.168.2.9
 key *********************
aaa-server Radius (inside) host 10.0.1.128
 key *********************
user-identity default-domain LOCAL
aaa authentication ssh console Radius LOCAL
aaa authentication http console Radius LOCAL
aaa authentication serial console Radius LOCAL
http server enable 4443
http 192.168.4.0 255.255.255.0 management
http 192.168.2.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
no snmp-server enable
sysopt connection tcpmss 1300
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set strong esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set nah esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ts-memorial esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set transform-amzn esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES256-SHA1_TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES256-SHA1_TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES128-SHA1_TRANS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES128-SHA1_TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES256-SHA1 esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport
crypto ipsec security-association pmtu-aging infinite
crypto dynamic-map outside_dyn_map 20 set ikev1 transform-set ESP-AES256-SHA1_TRANS ESP-AES128-SHA1_TRANS ESP-AES256-SHA1
crypto dynamic-map outside_dyn_map 30 set pfs 
crypto map outside_map 210 match address outside_210_cryptomap
crypto map outside_map 210 set peer 129.xxx.230.17 
crypto map outside_map 210 set ikev1 transform-set strong
crypto map outside_map 210 set reverse-route
crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map
crypto map outside_map interface outside
crypto ca trustpool policy
crypto isakmp disconnect-notify
crypto ikev1 enable outside
crypto ikev1 ipsec-over-tcp port 10000 
crypto ikev1 policy 20
 authentication pre-share
 encryption 3des
 hash md5
 group 2
 lifetime 86400
crypto ikev1 policy 40
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 28800
crypto ikev1 policy 60
 authentication pre-share
 encryption aes-256
 hash sha
 group 5
 lifetime 28800
crypto ikev1 policy 70
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 28800
crypto ikev1 policy 100
 authentication rsa-sig
 encryption aes-256
 hash sha
 group 5
 lifetime 86400
telnet timeout 5
ssh scopy enable
ssh stricthostkeycheck
ssh 192.168.2.0 255.255.255.0 inside
ssh 192.168.0.0 255.255.0.0 management
ssh timeout 5
ssh version 2
ssh cipher encryption custom "aes128-ctr"
ssh cipher integrity custom "hmac-sha1"
ssh key-exchange group dh-group1-sha1
console timeout 5
management-access inside
no vpn-addr-assign dhcp
vpn-addr-assign local reuse-delay 1
dhcprelay timeout 60
threat-detection basic-threat
threat-detection statistics host
threat-detection statistics port
threat-detection statistics protocol
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ntp server 216.239.35.4 source outside prefer
tftp-server inside 192.168.2.17 5516ax
ssl trust-point _wildcard inside
ssl trust-point _wildcard outside
webvpn
 port 4443
 enable outside
 dtls port 4443
 anyconnect image disk0:/anyconnect-win-4.3.02039-k9.pkg 1
 anyconnect image disk0:/anyconnect-macosx-i386-4.3.02039-k9.pkg 2
 anyconnect profiles Default disk0:/default.xml
 anyconnect enable
 port-forward test1 4001 10.23.10.109 4001 
 tunnel-group-list enable
 cache
  disable
group-policy DfltGrpPolicy attributes
 vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless
group-policy group2 internal
group-policy group2 attributes
 wins-server none
 dns-server value 192.168.2.17 192.168.2.14
 dhcp-network-scope 192.168.2.98
 vpn-idle-timeout none
 vpn-session-timeout none
 vpn-filter value RA-ACL
 vpn-tunnel-protocol ikev1 ikev2 l2tp-ipsec ssl-client 
 password-storage disable
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value group1_splitTunnelAcl
 default-domain value our.company.com
 intercept-dhcp 255.255.255.255 enable
 webvpn
  html-content-filter none
  anyconnect keep-installer installed
  anyconnect ssl rekey time 30
  anyconnect ssl rekey method ssl
  anyconnect dpd-interval client none
  anyconnect dpd-interval gateway none
  anyconnect profiles value Default type user
  anyconnect ask none default anyconnect
tunnel-group DefaultL2LGroup ipsec-attributes
 ikev1 pre-shared-key **********
tunnel-group DefaultRAGroup general-attributes
 address-pool (inside) vpnpool1
 address-pool vpnpool1
 authentication-server-group Radius
 default-group-policy group2
tunnel-group DefaultRAGroup ipsec-attributes
 ikev1 pre-shared-key ***********
 isakmp keepalive disable
tunnel-group DefaultRAGroup ppp-attributes
 authentication ms-chap-v2
tunnel-group DefaultWEBVPNGroup general-attributes
 authentication-server-group Radius
 default-group-policy group2
 dhcp-server 192.168.2.98
tunnel-group group2 type remote-access
tunnel-group group2 general-attributes
 address-pool vpnpool1
 authentication-server-group Radius LOCAL
 default-group-policy group2
 password-management
tunnel-group group2 webvpn-attributes
 group-alias Group2 disable
 group-alias group2 enable
tunnel-group group2 ipsec-attributes
 ikev1 pre-shared-key ******
tunnel-group 129.xxx.230.17 type ipsec-l2l
tunnel-group 129.xxx.230.17 ipsec-attributes
 ikev1 pre-shared-key ***********
 isakmp keepalive disable
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 4096
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ip-options 
 class class-default
  user-statistics accounting
!
service-policy global_policy global
smtp-server 192.168.2.17
prompt hostname context 
no call-home reporting anonymous
hpm topN enable
Cryptochecksum:7d5b78b68915ad22e68c090a72abb86a
: end
  • 1
    Please post your (sanitized) config. We can't just guess what might be wrong. – Ron Trunk May 4 '18 at 13:40
  • Edited post to include config. – Chris Holt May 4 '18 at 16:54
  • I managed to set this up in the lab and discovered the answer myself: "no crypto map outside_map 210 set reverse-route". – Chris Holt May 9 '18 at 13:43
  • 2
    Please post your solution as an answer and accept it, so this question won't keep popping up. – Teun Vink May 9 '18 at 13:46
  • 1
    Great! Please consider answering your own question so that others with the same problem can learn from your experience. – Ron Trunk May 9 '18 at 13:47
1

I managed to set this up in the lab and discovered the answer myself:

"no crypto map outside_map 210 set reverse-route"

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.