1

I have 2 cisco ASA 5512 units (security plus license) in an Active / Standby HA Failover cluster (I've tested both stateful and stateless failover unsuccessfully).

Both units are connected (over a switching path) to a VRRP - enabled gateway, set up by our hosting center provider where the units are installed.

Only one unit can be active (and retain the active state) in a stable way at a given time - meaning that as soon as I attempt to manually trigger failover from the currently active unit to the standby unit, the standby unit (gaining the ACTIVE role) will not be able to route traffic and will fall back to the standby role automatically within minutes - the standby unit is not able to retain stably the active state (even if L3 connectivity should be available, according to the hosting center statement).

This scenario happens with Stateful Active / standby failover as well as with stateless Active / standby failover.

What is also interesting is that the 2 units are configured with 2 public IPs, namely X.X.X.1 and X.X.X.2.

At time t0 (before manual failover is triggered) -> both Public IPs are reachable and PRIMARY Unit is Active

At time t1 -> manual failover is triggered and X.X.X.1 is no longer reachable (assigned to the SECONDARY unit now ACTIVE) -> X.X.X.2 is assigned to the PRIMARY unit and is 100 % reachable

at time t2 -> cluster realizes that cannot route traffic via the SECONDARY unit and falls back to the former scenario (PRIMARY unit is ACTIVE and SECONDARY unit is STANDBY) -> X.X.X.2 is no longer reachable; X.X.X.1 is again reachable;

at time t2 + approx. 30 minutes -> X.X.X.2 becomes again reachable.

The 2 units outside interfaces are connected to the same segment with VRRP settings. According to the hosting site, there are no particular L2 settings in place concerning L2 ports connectivity upstream.

It looks like the cluster is able to route traffic (in a stable way, that is with the unit staying permanently in "Active" status) only when connected to one of the 2 VRRP enabled ports - the behavior described above happens no matter which ASA is connected to THAT port, and always follows the same pattern.

Hosting center has checked the VRRP settings (as well as L2 / switching path) and wasn't able to find any significant issues that might justify this behavior.

In the case of Stateful failover, I've also observed several "Route Session" errors in the LU updates exchanged between the 2 ASAs:

on the PRIMARY ASA:

ASA-fw# show failover<br/>
Failover On<br/>
Failover unit Primary<br/>
Failover LAN Interface: failover GigabitEthernet0/5 (up)<br/>
Unit Poll frequency 3 seconds, holdtime 10 seconds<br/>
Interface Poll frequency 3 seconds, holdtime 15 seconds<br/>
Interface Policy 1<br/>
Monitored Interfaces 3 of 114 maximum<br/>
failover replication http<br/>
Version: Ours 9.1(1), Mate 9.1(1)<br/>
Last Failover at: 09:14:07 UTC Jan 25 2016<br/>
        This host: Primary - Active<br/>
                Active time: 415235 (sec)<br/>
                slot 0: ASA5512 hw/sw rev (1.0/9.1(1)) status (Up Sys)<br/>
                  Interface outside (X.X.X.195): Normal (Monitored)<br/>
                  Interface inside (192.168.1.1): Normal (Monitored)<br/>
                  Interface management (192.168.99.1): Normal (Monitored)<br/>
        Other host: Secondary - Standby Ready<br/>
                Active time: 3090024 (sec)<br/>
                slot 0: ASA5512 hw/sw rev (1.0/9.1(1)) status (Up Sys)<br/>
                  Interface outside (X.X.X.200): Normal (Monitored)<br/>
                  Interface inside (192.168.1.3): Normal (Monitored)<br/>
                  Interface management (192.168.99.2): Normal (Monitored)<br/>
<br/>
Stateful Failover Logical Update Statistics<br/>
        Link : failover GigabitEthernet0/5 (up)<br/>
        Stateful Obj    xmit       xerr       rcv        rerr<br/>
        General         1518905    0          56441      9<br/>
        sys cmd         54791      0          54790      0<br/>
        up time         0          0          0          0<br/>
        RPC services    0          0          0          0<br/>
        TCP conn        1126177    0          156        0<br/>
        UDP conn        140986     0          889        0<br/>
        ARP tbl         190094     0          606        0<br/>
        Xlate_Timeout   0          0          0          0<br/>
        IPv6 ND tbl     0          0          0          0<br/>
        VPN IKEv1 SA    4          0          0          0<br/>
        VPN IKEv1 P2    8          0          0          0<br/>
        VPN IKEv2 SA    0          0          0          0<br/>
        VPN IKEv2 P2    0          0          0          0<br/>
        VPN CTCP upd    0          0          0          0<br/>
        VPN SDI upd     0          0          0          0<br/>
        VPN DHCP upd    0          0          0          0<br/>
        SIP Session     0          0          0          0<br/>
        Route Session   6842       0          0          9<br/>
        User-Identity   3          0          0          0<br/>
        CTS SGTNAME     0          0          0          0<br/>
        CTS PAC         0          0          0          0<br/>
        TrustSec-SXP    0          0          0          0<br/>
        IPv6 Route      0          0          0          0<br/>
<br/>
        Logical Update Queue Information<br/>
                        Cur     Max     Total<br/>
        Recv Q:         0       12      135522<br/>
        Xmit Q:         0       31      1717200<br/>
<br/>
ASA-fw#<br/>

on the SECONDARY ASA:

ASA-fw# show failover<br/>
Failover On<br/>
Failover unit Secondary<br/>
Failover LAN Interface: failover GigabitEthernet0/5 (up)<br/>
Unit Poll frequency 3 seconds, holdtime 10 seconds<br/>
Interface Poll frequency 3 seconds, holdtime 15 seconds<br/>
Interface Policy 1<br/>
Monitored Interfaces 3 of 114 maximum<br/>
failover replication http<br/>
Version: Ours 9.1(1), Mate 9.1(1)<br/>
Last Failover at: 09:14:07 UTC Jan 25 2016<br/>
        This host: Secondary - Standby Ready<br/>
                Active time: 3090024 (sec)<br/>
                slot 0: ASA5512 hw/sw rev (1.0/9.1(1)) status (Up Sys)<br/>
                  Interface outside (X.X.X.200): Normal (Monitored)<br/>
                  Interface inside (192.168.1.3): Normal (Monitored)<br/>
                  Interface management (192.168.99.2): Normal (Monitored)<br/>
        Other host: Primary - Active<br/>
                Active time: 415193 (sec)<br/>
                slot 0: ASA5512 hw/sw rev (1.0/9.1(1)) status (Up Sys)<br/>
                  Interface outside (X.X.X.195): Normal (Monitored)<br/>
                  Interface inside (192.168.1.1): Normal (Monitored)<br/>
                  Interface management (192.168.99.1): Normal (Monitored)<br/>

Stateful Failover Logical Update Statistics<br/>
        Link : failover GigabitEthernet0/5 (up)<br/>
        Stateful Obj    xmit       xerr       rcv        rerr<br/>
        General         408964     0          1527527    6852<br/>
        sys cmd         69155      0          69153      0<br/>
        up time         0          0          0          0<br/>
        RPC services    0          0          0          0<br/>
        TCP conn        258657     0          1127204    0<br/>
        UDP conn        34851      0          140988     0<br/>
        ARP tbl         44206      0          190171     0<br/>
        Xlate_Timeout   0          0          0          0<br/>
        IPv6 ND tbl     0          0          0          0<br/>
        VPN IKEv1 SA    30         0          4          0<br/>
        VPN IKEv1 P2    8          0          4          0<br/>
        VPN IKEv2 SA    0          0          0          0<br/>
        VPN IKEv2 P2    0          0          0          0<br/>
        VPN CTCP upd    0          0          0          0<br/>
        VPN SDI upd     0          0          0          0<br/>
        VPN DHCP upd    0          0          0          0<br/>
        SIP Session     0          0          0          0<br/>
        Route Session   2010       0          0          6852<br/>
        User-Identity   47         0          3          0<br/>
        CTS SGTNAME     0          0          0          0<br/>
        CTS PAC         0          0          0          0<br/>
        TrustSec-SXP    0          0          0          0<br/>
        IPv6 Route      0          0          0          0<br/>
<br/>
        Logical Update Queue Information<br/>
                        Cur     Max     Total<br/>
        Recv Q:         0       20      1811827<br/>
        Xmit Q:         0       30      466771<br/>
<br/>
ASA-fw#<br/>

I was wondering if there are some debugs that might assist me in troubleshooting this issue and possibly help in identifying Hosting center related misconfig - issues.

Also, I wanted to ask details on the "Route Session" entry and the impact of several xerr/rerr in this field.

Lastly, could this be a SW bug ? I read on cisco's website of several bugs for ASA clustering under 9.1(1) release:

http://www.cisco.com/c/en/us/td/docs/security/asa/asa91/release/notes/asarn91.html

  • What are these ASAs connected to? The 30 minute delay strongly suggests you are having ARP issues. That is, the upstream/downstream devices are getting confused as to which device is active. – Ron Trunk Jan 25 '16 at 12:52
  • Hi Ron, thank you very much for formatting my question as well as for your initial assessment. Unfortunately I don't have knowledge of the L2 switches / devices in between, but definitely it sound like an ARP / protection mechanism - I believe that in my case we are talking about the switches sitting between the VRRP-enabled devices and the ASAs. I also saw ARP requests/queries coming in over the outside interface of the outside unit connected to the "problematic L2 panel port" during the 30 min "outage" ... the ASA is replying but the other side keeps inquiring .... – Ottootto Jan 25 '16 at 13:18
  • Based on what you're saying, it sounds like your problem is not with the ASAs, but with whatever switches are in the middle. – Ron Trunk Jan 25 '16 at 13:22
  • Thank you. It turned out to be Port Security enabled on ISP access switches, which was blocking frames (following failover) for approx. 30 minutes. Thanks – Ottootto Feb 2 '16 at 14:36
  • Great. You should answer your own question so that other people will have the benefit of your experience. – Ron Trunk Feb 2 '16 at 14:40
2

It turned out to be Port Security enabled on ISP access switches, which was blocking frames (following failover) for approx. 30 minutes. Thanks everyone and Ron Trunk especially

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.