3

I have two offices (Victoria at IP 1.2.3.4 and Toronto at IP 5.6.7.8) each with pfSense running Strongswan, and each with an IKEv2 IPSec tunnel back to a Cisco ASA 5512 at IP 9.8.7.6. I recently updated software on the ASA from 9.4.2(11) to 9.4.3(4) without issue. Both tunnels came back up and worked fine for 1 day and 17 hours, but (without any configuration changes on either side) the Victoria tunnel has now stopped passing traffic.

The tunnel is established without a problem, but show ipsec sa tells me no traffic is passing. Restarting the tunnel does not make a difference.

ASA1# show ipsec sa peer 1.2.3.4
peer address: 1.2.3.4
    Crypto map tag: OUTSIDE_map, seq num: 1, local addr: 9.8.7.6

      access-list OUTSIDE_cryptomap_2 extended permit ip 192.168.242.0 255.255.255.0 192.168.244.0 255.255.255.0 
      local ident (addr/mask/prot/port): (192.168.242.0/255.255.255.0/0/0)
      remote ident (addr/mask/prot/port): (192.168.244.0/255.255.255.0/0/0)
      current_peer: 1.2.3.4


      #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
      #pkts decaps: 1428, #pkts decrypt: 1428, #pkts verify: 1428
      #pkts compressed: 0, #pkts decompressed: 0
      #pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
      #pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
      #PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
      #TFC rcvd: 0, #TFC sent: 0
      #Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
      #send errors: 0, #recv errors: 0

      local crypto endpt.: 9.8.7.6/500, remote crypto endpt.: 1.2.3.4/500
      path mtu 1500, ipsec overhead 55(36), media mtu 1500
      PMTU time remaining (sec): 0, DF policy: copy-df
      ICMP error validation: disabled, TFC packets: disabled
      current outbound spi: CB3A6309
      current inbound spi : 5E3D8A13

    inbound esp sas:
      spi: 0x5E3D8A13 (1581091347)
         transform: esp-aes-gcm-256 esp-null-hmac no compression 
         in use settings ={L2L, Tunnel, PFS Group 14, IKEv2, }
         slot: 0, conn_id: 167936, crypto-map: OUTSIDE_map
         sa timing: remaining key lifetime (sec): 2676
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap: 
          0xFFFFFFFF 0xFFFFFFFF
    outbound esp sas:
      spi: 0xCB3A6309 (3409601289)
         transform: esp-aes-gcm-256 esp-null-hmac no compression 
         in use settings ={L2L, Tunnel, PFS Group 14, IKEv2, }
         slot: 0, conn_id: 167936, crypto-map: OUTSIDE_map
         sa timing: remaining key lifetime (sec): 2676
         IV size: 8 bytes
         replay detection support: Y
         Anti replay bitmap: 
          0x00000000 0x00000001

A packet capture looking for dropped packets gives me this, but doesn't tell me what rule is being applied.

939: 20:11:44.438591 0023.ab3f.8255 24e9.b315.cddf 0x0800 Length: 89
      192.168.244.114.51353 > 192.168.242.200.53:  [no cksum] udp 47 [tos 0x10]  (ttl 63, id 8826) Drop-reason: (acl-drop) Flow is denied by configured rule

Here is some of the relevant configuration; as far as I can see both offices are identical. If anyone has any suggestions as to what could be happening here, I'd appreciate it!

ASA1# show running-config crypto map
crypto map OUTSIDE_map 1 match address OUTSIDE_cryptomap_2
crypto map OUTSIDE_map 1 set pfs group14
crypto map OUTSIDE_map 1 set peer 1.2.3.4
crypto map OUTSIDE_map 1 set ikev2 ipsec-proposal AESGCM
crypto map OUTSIDE_map 1 set security-association lifetime seconds 3600
crypto map OUTSIDE_map 1 set nat-t-disable
crypto map OUTSIDE_map 2 match address OUTSIDE_cryptomap_3
crypto map OUTSIDE_map 2 set pfs group14
crypto map OUTSIDE_map 2 set peer 5.6.7.8
crypto map OUTSIDE_map 2 set ikev2 ipsec-proposal AESGCM
crypto map OUTSIDE_map 2 set security-association lifetime seconds 3600
crypto map OUTSIDE_map 2 set nat-t-disable
crypto map OUTSIDE_map interface OUTSIDE

ASA1# show running-config access-list OUTSIDE_cryptomap_2
access-list OUTSIDE_cryptomap_2 extended permit ip object NOC-network object Victoria-network

ASA1# show running-config access-list OUTSIDE_cryptomap_3
access-list OUTSIDE_cryptomap_3 extended permit ip object NOC-network object Toronto-network

ASA1# show running-config nat
nat (INSIDE,OUTSIDE) source static NOC-network NOC-network destination static Victoria-network Victoria-network no-proxy-arp route-lookup
nat (INSIDE,OUTSIDE) source static NOC-network NOC-network destination static Toronto-network Toronto-network no-proxy-arp route-lookup

ASA1# show running-config tunnel-group 
tunnel-group 1.2.3.4 type ipsec-l2l
tunnel-group 1.2.3.4 general-attributes
 default-group-policy GroupPolicy_Victoria
tunnel-group 1.2.3.4 ipsec-attributes
 ikev1 pre-shared-key *****
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****
tunnel-group 5.6.7.8 type ipsec-l2l
tunnel-group 5.6.7.8 general-attributes
 default-group-policy GroupPolicy_Toronto
tunnel-group 5.6.7.8 ipsec-attributes
 ikev1 pre-shared-key *****
 isakmp keepalive threshold 15 retry 2
 ikev2 remote-authentication pre-shared-key *****
 ikev2 local-authentication pre-shared-key *****

ASA1# show running-config crypto ikev2
crypto ikev2 policy 2
 encryption aes-gcm-256
 integrity null
 group 21 24
 prf sha512
 lifetime seconds 28800
crypto ikev2 policy 3
 encryption aes-256
 integrity sha512
 group 21 24
 prf sha512
 lifetime seconds 28800
crypto ikev2 enable OUTSIDE

ASA1# show running-config crypto ipsec
crypto ipsec ikev2 ipsec-proposal AES256-SHA512
 protocol esp encryption aes-256
 protocol esp integrity sha-512
crypto ipsec ikev2 ipsec-proposal AESGCM
 protocol esp encryption aes-gcm-256
 protocol esp integrity sha-512
crypto ipsec ikev2 sa-strength-enforcement
crypto ipsec security-association pmtu-aging infinite

#ASA1 show running-config all sysopt
no sysopt traffic detailed-statistics
no sysopt connection timewait
sysopt connection tcpmss 1380
sysopt connection tcpmss minimum 0
sysopt connection permit-vpn
sysopt connection reclassify-vpn
no sysopt connection preserve-vpn-flows
no sysopt radius ignore-secret
no sysopt noproxyarp OUTSIDE
no sysopt noproxyarp INSIDE
no sysopt noproxyarp DMZ1
no sysopt noproxyarp management
  • #pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0 is very suspicious. Looks like the ASA isn't encrypting any traffic to send to the remote end. That could happen because of NAT or because hosts in 192.168.242.0/24 don't know to go to the ASA to reach 192.168.244.0/24. If you traceroute towards towards 192.168.244.0 from a host behind the ASA on 192.168.242.0 do you reach the ASA? – Mark Jul 16 '16 at 17:47
  • 2
    You claim nothing has changed, but in 99% of the cases I've ever encountered, somebody changed something; these sort of things don't just up-and-break. The first thing I would check is that traffic for the VPN is actually getting from end to end -- tcpdump from an "internet" end, post-crypto. Then look at traffic entering/exiting the tunnel. Etc. – Ricky Beam Jul 16 '16 at 21:17
  • @RickyBeam I am the only one in this company that touches these. I was using the internet one minute and then it was dead the next. As I said, I'd upgraded the software on the ASA; uptime was 41 hours when the tunnel dropped. Traffic is getting through the tunnel, as evidenced by the error logs when I try to ping the ASA on its inside interface. – miken32 Jul 16 '16 at 22:06
  • ICMP is always denied unless explicitly permitted. This is separate from access lists. – Ron Trunk Jul 16 '16 at 22:55
  • Actually looks like the VPN config is correct but the access list of the outside interface is denying the traffic. Double check whether the ACL permits the traffic or if sysopt connection permit-vpn is enabled. Please do not enable it right away unless you fully understand what that command does. – waza-ari Jul 17 '16 at 19:13
3

Your scenario looks very similar to something you'll see if you have a stale SPI on your ASA that is "trumping" the valid active SPI. The bad news is, if this is the case...you can only correct this by rebooting the ASA itself.

Something you can try is running a packet tracer to simulate traffic and check that you see a "VPN Encrypt Phase" in the output. If you do, but the traffic fails to encrypt then it could be match against a stale SPI that isn't active.

You can also use show asp table classify crypto and look for matches on your encryption domain. You do not want to see multiple matches with the same cs_id. See this example: (notice how they networks are exactly the same but they have different cs_id and only one has hitscounts?)


in  id=0x7fff370d6450, priority=70, domain=ipsec-tunnel-flow, deny=false
    hits=17302, user_data=0x8e0d6a4, **cs_id=0x7fff36c15af0**, reverse, flags=0x0, protocol=0
    src ip/id=10.202.140.0, mask=255.255.255.0, port=0, tag=0
    dst ip/id=10.202.126.0, mask=255.255.255.0, port=0, tag=0, dscp=0x0
    input_ifc=outside, output_ifc=any

in  id=0x7fff3d48dda0, priority=70, domain=ipsec-tunnel-flow, deny=false
    hits=0, user_data=0xaaf9b0c, **cs_id=0x7fff38d9d080**, reverse, flags=0x0, protocol=0
    src ip/id=10.202.140.0, mask=255.255.255.0, port=0, tag=0
    dst ip/id=10.202.126.0, mask=255.255.255.0, port=0, tag=0, dscp=0x0
    input_ifc=outside, output_ifc=any

The above example is something I've taken directly from troubleshooting exactly the same problem you are describing in which case a reboot corrected the issue. I also had an issue where the same firewall stopped passing traffic on an active tunnel with encaps and decaps for only a few hosts. Bouncing the tunnel corrected the problem.

These issues only started after upgrading their VPNs to IKEv2. I suspect if you re-key you're tunnels to often on IKEv2 it seems to be very likely you're have issues with SPIs. You may want to try increasing the lifetime of your phase two tunnels since you have it set to 3600 right now (My customer was hitting the default data lifetime on the Phase 2 tunnels every 4 minutes and rekeying, so I set the data lifetime to unlimited)

  • I do see a lot of duplicates; though I still see those after a reboot. For domain ipsec-tunnel-flow it looks like each tunnel has three entries, all with the same cs_id, and only one with hits; two have the correct src and dst addresses, the others are zeroes. – miken32 Jul 22 '16 at 0:19
  • You said "You do not want to see multiple matches with the same cs_id" but your example, which you said was from a broken ASA, there are different cs_id. If you can clarify a bit and perhaps speak a bit to what these numbers mean it would be great. Thanks for taking the time to answer. – miken32 Jul 22 '16 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.