2

I have a site-2-site vpn tunnel between a Pix 506e 6.3(3) and an ASA 5510 running 9.0(3)6. I can control both sides of the config.

This tunnel worked fine until we did a firmware update on the ASA which was originally running 8.4(2). I have 3 tunnels which terminate at our ASA with peer Ips that are on device pix 506e models having issues and I cant figure out why. I will focus on one tunnel in particular in hopes that someone can help me fix it and I can try to apply the fix to the other two acting up.

The symptoms are as follows:

Tunnel will come up with Phase 1 and Phase 2. Everything will work fine for a variable amount of time then the tunnel will drop. I see this over and over again in the logs of the ASA

Tunnel Manager has failed to establish an L2L SA. All configured IKE versions failed to establish the tunnel

If I go into the ASA and I remove the crypto map and then re-add it. The tunnel comes back up and remains active for a variable amount of time once again. And when I say variable I mean it can stay up and working for as long as a half a day or as little as 15 min .

During the outage if I do a show isakmp sa on the pix I get the following

pix# show isakmp sa

Total     : 6
Embryonic : 0
        dst               src        state     pending     created
   2.2.2.2   1.1.1.1    QM_IDLE         0         115
   2.2.2.2   1.1.1.1    QM_IDLE         0         254
   2.2.2.2   1.1.1.1    QM_IDLE         0         123
   2.2.2.2   1.1.1.1    QM_IDLE         0         108
   2.2.2.2   1.1.1.1    QM_IDLE         0         224
   2.2.2.2   1.1.1.1    QM_IDLE         0         129

On the ASA doing the same cmd will get me

IKE Peer: 2.2.2.2
    Type    : L2L             Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG2

If I want to bring the tunnel back up right away, I can remove the crypto map from the ASA and then re-add it and everything will work again for a bit. What should I be checking?

Is there some kind of difference I should now be aware of between the isakmp config on pix and ikev1 on ASA? It was all working before when the ASA was on 8.4(2) and this is ONLY happening to my tunnels that are terminating on the PIX 506 e devices running 6.3(3) . Thats a clue I know, I just don't understand what I should be looking at to figure out how to fix it.

The PRODUCTION-NET in this example config would be 192.168.11.0/24

ASA 5520

object-group network POGO
 network-object host 172.232.x.x

access-list POGO extended permit ip object PRODUCTION-NET object-group POGO 

nat (inside,outside) source static PRODUCTION-NET PRODUCTION-NET destination static POGO POGO

crypto map vpn-tunnel 320 match address POGO
crypto map vpn-tunnel 320 set peer 2.2.2.2
crypto map vpn-tunnel 320 set ikev1 transform-set ESP-AES-128-MD5

tunnel-group 2.2.2.2 type ipsec-l2l
tunnel-group 2.2.2.2 general-attributes
 default-group-policy VPN-TUNNEL-POLICY
tunnel-group 2.2.2.2 ipsec-attributes
 ikev1 pre-shared-key cisco123
 isakmp keepalive threshold 30 retry 10

Pix 506e

object-group network POGO_VPN 
  network-object host 172.232.x.x

access-list DSW permit ip object-group POGO_VPN 192.168.11.0 255.255.255.0

crypto ipsec transform-set AES128 esp-aes esp-md5-hmac 
crypto map pcmap 320 ipsec-isakmp
crypto map pcmap 320 match address POGO
crypto map pcmap 320 set peer 1.1.1.1
crypto map pcmap 320 set transform-set AES128
crypto map pcmap client configuration address initiate
crypto map pcmap client configuration address respond
crypto map pcmap interface outside
isakmp enable outside
isakmp key ******** address 1.1.1.1 netmask 255.255.255.255 no-xauth no-config-mode 
isakmp identity address
isakmp keepalive 30 10
isakmp nat-traversal 20
isakmp policy 1 authentication pre-share
isakmp policy 1 encryption aes
isakmp policy 1 hash md5
isakmp policy 1 group 2
isakmp policy 1 lifetime 86400

on the debug crypto isakmp I see this going on while the outage occurs on the ASA side. On the remote pix side it does not show any logs just

ASA isakmp Debug J

un 18 12:31:23 [IKEv1]IP = 2.2.2.2, IKE Initiator: New Phase 1, Intf inside, IKE Peer 2.2.2.2  local Proxy Address 192.168.11.0, remote Proxy Address 172.233.240.226,  Crypto map (vpn-tunnel)
    Jun 18 12:31:23 [IKEv1 DEBUG]IP = 2.2.2.2, constructing ISAKMP SA payload
    Jun 18 12:31:23 [IKEv1 DEBUG]IP = 2.2.2.2, constructing NAT-Traversal VID ver 02 payload
    Jun 18 12:31:23 [IKEv1 DEBUG]IP = 2.2.2.2, constructing NAT-Traversal VID ver 03 payload
    Jun 18 12:31:23 [IKEv1 DEBUG]IP = 2.2.2.2, constructing NAT-Traversal VID ver RFC payload
    Jun 18 12:31:23 [IKEv1 DEBUG]IP = 2.2.2.2, constructing Fragmentation VID + extended capabilities payload
    Jun 18 12:31:23 [IKEv1]IP = 2.2.2.2, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 672
    Jun 18 12:31:30 [IKEv1]IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
    Jun 18 12:31:30 [IKEv1]IP = 2.2.2.2, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
    Jun 18 12:31:31 [IKEv1]IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 672
    Jun 18 12:31:39 [IKEv1]IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 672
    Jun 18 12:31:47 [IKEv1]IP = 2.2.2.2, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 672
    Jun 18 12:31:55 [IKEv1 DEBUG]IP = 2.2.2.2, IKE MM Initiator FSM error history (struct &0x74d097c8)  <state>, <event>:  MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY
    Jun 18 12:31:55 [IKEv1 DEBUG]IP = 2.2.2.2, IKE SA MM:e7fa4a37 terminating:  flags 0x01000022, refcnt 0, tuncnt 0
    Jun 18 12:31:55 [IKEv1 DEBUG]IP = 2.2.2.2, sending delete/delete with reason message

debug isakmp on the PIX

PEER_REAPER_TIMERshow isakmp sa         
Total     : 2
Embryonic : 0
        dst               src        state     pending     created
   2.2.2.2   1.1.1.1    QM_IDLE         0           6
   2.2.2.2   1.1.1.1    QM_IDLE         0         117
dswpix(config)# 
PEER_REAPER_TIMER
PEER_REAPER_TIMER
REAPER_TIMER
ISADB: reaper checking SA 0x10232bc, conn_id = 0
ISADB: reaper checking SA 0xfe9a44, conn_id = 0
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
REAPER_TIMER
ISADB: reaper checking SA 0x10232bc, conn_id = 0
ISADB: reaper checking SA 0xfe9a44, conn_id = 0
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
PEER_REAPER_TIMER
REAPER_TIMER
ISADB: reaper checking SA 0x10232bc, conn_id = 0
ISADB: reaper checking SA 0xfe9a44, conn_id = 0debug crypto ipsec 200 
  • if 8.4 was working, why move to 9.0? (and not upgrade the pix to 635, or 704) What do the logs say when this happens? – Ricky Beam Jun 10 '14 at 18:39
  • The firmware update to the ASA was forced by an internal security audit, and the pix is at the max ram it can have which is under what is needed for a firmware update to it. I have not seen an error on the pix log but the error above about ikev is show on the asa. If I go to the pix and do a clear isakmp sa then it comes back. – ATX_Admin Jun 10 '14 at 20:06
  • Go ask your auditors why it doesn't work as their the fools demanding a major upgrade (not even to a current release, at that.) Why didn't they demand replacing the long EOL 506e's with their seriously exploitable 6.3(3) firmware? (yes, it's a peav) I would have to say you've run into a bug in 6.3(3) or an interop issue between it and 9.0(3). – Ricky Beam Jun 10 '14 at 20:56
  • While I tend to agree its a pain in the ass I still believe there is a solution to be found without having to ship new hardware. Thanks for the look and the comments. – ATX_Admin Jun 11 '14 at 14:56
  • Not new hardware, just load the last released pix image (6.3.5, or 6.3.5-145) on the 506's. 633 is a decade+ old, 635 is nearing a decade old. I'd setup some debugs and traffic captures to look at when it does fail -- yes, that may be a lot of traffic. – Ricky Beam Jun 11 '14 at 22:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.