4

I have a very strange issue with Cisco ASA 5505 started about week ago: ASA like a router with outside port to ISP, next port to DMZ and another one to inside network (contains about 50 hosts thru several Cisco SMB switches).

Many, but not all hosts (SIP phones, laptops, desktops) from inside network suddenly stop accessing internet and even ping to ASA doesn't pass thru. Some other hosts, like servers connected thru the same switches always works. DMZ hosts are also works without issues. It could work for 8 hours or happens every 15 mins with no visible understanding. Once the connectivtity for a hosts lost it is possible to restore it with ASA reboot or just wait for 15-20-30 minutes (randomly) before it will self-restore.

Today I have just started digging and found what "clear arp" immediately resolves the issue, but with my knowledge I can't identify the root and how to fix it. Configuration work almost for an year and nothing were changed recently. ISP checked from their side and reported the issue at our side so coundn't help somehow.

Upgraded ASA's software - no changes.

Any help will be appreciated, depersonalized running config is below.

Thanks!

Result of the command: "sh ru"

: Hardware:   ASA5505, 512 MB RAM, CPU Geode 500 MHz
:
ASA Version 9.2(4) 
!
hostname ciscoasa
domain-name XXXX
enable password XXXX encrypted
names
ip local pool vpn-pool 192.168.3.2-192.168.3.254 mask 255.255.255.0
!
interface Ethernet0/0
 switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
 switchport access vlan 12
!
interface Ethernet0/7
 switchport access vlan 12
!
interface Vlan1
 nameif inside
 security-level 100
 ip address 192.168.0.1 255.255.255.0 
!
interface Vlan2
 nameif outside
 security-level 0
 ip address ISP_PROVIDED_IP 255.255.255.192 
!
interface Vlan12
 no forward interface Vlan1
 nameif DMZ
 security-level 50
 ip address 172.16.0.1 255.255.255.0 
!
boot system disk0:/asa924-k8.bin
ftp mode passive
clock timezone MSK/MSD 3
clock summer-time MSK/MDD recurring last Sun Mar 2:00 last Sun Oct 3:00
dns domain-lookup inside
dns domain-lookup outside
dns server-group DefaultDNS
 name-server DNS1-ISP
 name-server DNS2-ISP
 name-server DNS3-INTERNAL_HOSTS
 name-server 8.8.8.8
 domain-name XXXXX
same-security-traffic permit intra-interface
object network obj_any
 subnet 0.0.0.0 0.0.0.0
object network NETWORK_OBJ_192.168.2.0_24
 subnet 192.168.2.0 255.255.255.0
object network 192.168.3.0
 subnet 192.168.3.0 255.255.255.0
object network 192.168.0.1
 subnet 192.168.0.0 255.255.255.0
object network ins
 subnet 192.168.1.0 255.255.255.0
object network 192.168.1.1
 host 192.168.1.1
object network asdc
 host 192.168.0.50
 description asdc
object network TGP500-01
 host 192.168.0.32
object network TGP500-02
 host 192.168.0.25
object network TGP500-03
 host 192.168.0.30
object network TGP500-04
 host 192.168.0.10
object network TGP500-05
 host 192.168.0.11
object network DMZ_outside
 subnet 0.0.0.0 0.0.0.0
object network DMZ
 subnet 172.16.0.0 255.255.255.0
object-group network SIP_PROVIDER
 description SIP_PROVIDER
 network-object SIP_ISP_NETW 255.255.255.0
 network-object host SIP_ISP_HOST
 network-object host SIP_ISP_HOST
 network-object host SIP_ISP_HOST
object-group service SIP-stun tcp-udp
 description SIP-stun
 port-object range 3478 3479
object-group service SIP60000 udp
 description SIP60000
 port-object eq 60000
object-group service sip-SIP udp
 description sip-SIP
 port-object range 1024 65535
object-group service DM_INLINE_UDP_1 udp
 group-object SIP-stun
 group-object SIP60000
 group-object sip-SIP
object-group protocol TCPUDP
 protocol-object udp
 protocol-object tcp
object-group network SIPphones
 description SIP phones
 network-object object TGP500-01
 network-object object TGP500-02
 network-object object TGP500-03
 network-object object TGP500-04
 network-object object TGP500-05
object-group service DM_INLINE_UDP_2 udp
 group-object SIP60000
 group-object sip-SIP
object-group service DM_INLINE_TCP_1 tcp
 port-object eq 3389
 port-object eq 7766
access-list main standard permit 192.168.1.0 255.255.255.0 
access-list main standard permit 192.168.2.0 255.255.255.0 
access-list main standard permit 192.168.3.0 255.255.255.0 
access-list main standard permit 192.168.0.0 255.255.255.0
access-list inside_access_in extended permit icmp any any 
access-list inside_access_in extended permit ip 192.168.0.0 255.255.255.0 192.168.0.0 255.255.255.0 
access-list inside_access_in extended permit ip any any 
access-list global_access extended permit icmp any any 
access-list outside_access_in remark SIPNAME SIP 5060
access-list outside_access_in extended permit object-group TCPUDP object-group SIPName object-group SIPphones eq sip 
access-list outside_access_in remark SIPNAME SIP
access-list outside_access_in extended permit udp object-group SIP object-group SIPphones object-group DM_INLINE_UDP_1 
access-list outside_access_in extended deny object-group TCPUDP any object-group SIPphones eq sip 
access-list outside_access_in extended deny udp any object-group SIPphones object-group DM_INLINE_UDP_2 
access-list outside_access_in extended permit tcp any object asdc object-group DM_INLINE_TCP_1 
access-list outside_access_in extended deny ip any object asdc inactive 
access-list outside_access_in extended permit ip any any 
access-list outside_access_in extended permit ip any object DMZ 
access-list AnyConnect_Client_Local_Print extended deny ip any4 any4 
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq lpd 
access-list AnyConnect_Client_Local_Print remark IPP: Internet Printing Protocol
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq 631 
access-list AnyConnect_Client_Local_Print remark Windows' printing port
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq 9100 
access-list AnyConnect_Client_Local_Print remark mDNS: multicast DNS protocol
access-list AnyConnect_Client_Local_Print extended permit udp any4 host 224.0.0.251 eq 5353 
access-list AnyConnect_Client_Local_Print remark LLMNR: Link Local Multicast Name Resolution protocol
access-list AnyConnect_Client_Local_Print extended permit udp any4 host 224.0.0.252 eq 5355 
access-list AnyConnect_Client_Local_Print remark TCP/NetBIOS protocol
access-list AnyConnect_Client_Local_Print extended permit tcp any4 any4 eq 137 
access-list AnyConnect_Client_Local_Print extended permit udp any4 any4 eq netbios-ns 
access-list DMZ_access_in extended permit ip any any 
access-list DMZ_mpc extended permit ip any 172.16.0.0 255.255.255.0 
access-list outside_mpc extended permit ip object-group SIPphones object-group SIP 
access-list outside_mpc_1 extended permit ip any 172.16.0.0 255.255.255.0 
pager lines 24
logging enable
logging asdm debugging
mtu inside 1500
mtu outside 1500
mtu DMZ 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-751.bin
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
!
object network asdc
 nat (inside,outside) static interface no-proxy-arp service tcp 3389 7766 
!
nat (inside,outside) after-auto source dynamic any interface
nat (outside,inside) after-auto source static any any no-proxy-arp
nat (DMZ,outside) after-auto source dynamic any interface
access-group inside_access_in in interface inside
access-group outside_access_in in interface outside
access-group DMZ_access_in in interface DMZ
access-group global_access global
route outside 0.0.0.0 0.0.0.0 ISP_GW_ADDR 1
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 0:15:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
no user-identity enable
user-identity default-domain LOCAL
aaa authentication enable console LOCAL 
aaa authentication ssh console LOCAL 
http server enable
http 0.0.0.0 0.0.0.0 outside
http 192.168.2.0 255.255.255.0 inside
http 192.168.0.0 255.255.255.0 inside
http 192.168.3.0 255.255.255.0 outside
http 192.168.1.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
crypto ipsec ikev2 ipsec-proposal DES
 protocol esp encryption des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal 3DES
 protocol esp encryption 3des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES
 protocol esp encryption aes
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES192
 protocol esp encryption aes-192
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES256
 protocol esp encryption aes-256
 protocol esp integrity sha-1 md5
crypto ipsec security-association pmtu-aging infinite
crypto dynamic-map SYSTEM_DEFAULT_CRYPTO_MAP 65535 set ikev2 ipsec-proposal AES256 AES192 AES 3DES DES
crypto map inside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map inside_map interface inside
crypto map outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map outside_map interface outside
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_0
 enrollment self
 subject-name CN=192.168.1.2,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_1
 enrollment self
 subject-name CN=192.168.1.2,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_2
 enrollment self
 subject-name CN=192.168.1.1,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_3
 enrollment self
 subject-name CN=192.168.2.1,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_4
 enrollment self
 subject-name CN=192.168.1.1,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_5
 enrollment self
 subject-name CN=192.168.0.1,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_6
 enrollment self
 subject-name CN=192.168.0.1,CN=ciscoasa
 crl configure
crypto ca trustpoint ASDM_Launcher_Access_TrustPoint_7
 enrollment self
 subject-name CN=192.168.0.1,CN=ciscoasa
 crl configure
crypto ca trustpool policy
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_0
 certificate 747d9154
 XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_1
 certificate 465e9554
   XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_2
 certificate 475e9554
    XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_3
 certificate 485e9554
   XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_4
 certificate 495e9554
   XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_5
 certificate 4a5e9554
    XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_6
 certificate 4b5e9554
    XXX
  quit
crypto ca certificate chain ASDM_Launcher_Access_TrustPoint_7
 certificate 9a7b9554
    XXX
  quit
crypto ikev2 policy 1
 encryption aes-256
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 10
 encryption aes-192
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 20
 encryption aes
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 30
 encryption 3des
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 40
 encryption des
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 enable inside client-services port 443
crypto ikev2 enable outside
crypto ikev2 remote-access trustpoint ASDM_Launcher_Access_TrustPoint_0
telnet timeout 5
ssh stricthostkeycheck
ssh 192.168.0.0 255.255.255.0 inside
ssh timeout 5
ssh version 2
ssh key-exchange group dh-group14-sha1
console timeout 0

dhcpd dns 192.168.0.1 interface inside
!
dhcpd address 172.16.0.5-172.16.0.50 DMZ
dhcpd dns 8.8.8.8 interface DMZ
dhcpd lease 604800 interface DMZ
dhcpd enable DMZ
!
priority-queue outside
threat-detection basic-threat
threat-detection statistics
threat-detection statistics tcp-intercept rate-interval 30 burst-rate 400 average-rate 200
ntp server 129.6.15.29 source outside
ssl trust-point ASDM_Launcher_Access_TrustPoint_2 outside
ssl trust-point ASDM_Launcher_Access_TrustPoint_7 inside vpnlb-ip
ssl trust-point ASDM_Launcher_Access_TrustPoint_7 inside
ssl trust-point ASDM_Launcher_Access_TrustPoint_2 outside vpnlb-ip
webvpn
 enable inside
 enable outside
 anyconnect-essentials
 anyconnect image disk0:/anyconnect-win-4.0.00048-k9.pkg 1
 anyconnect enable
 tunnel-group-list enable
group-policy DfltGrpPolicy attributes
 dns-server value 192.168.0.1
 split-tunnel-network-list value main
 webvpn
  anyconnect ssl compression deflate
group-policy GroupPolicy_main internal
group-policy GroupPolicy_main attributes
 wins-server none
 dns-server value 192.168.0.1
 vpn-tunnel-protocol ikev2 ssl-client 
 split-tunnel-policy tunnelspecified
 split-tunnel-network-list value main
 default-domain value XXX
 webvpn
  anyconnect profiles value main_client_profile type user
tunnel-group main type remote-access
tunnel-group main general-attributes
 address-pool vpn-pool
 authentication-server-group (inside) LOCAL
 authentication-server-group (outside) LOCAL
 authorization-server-group LOCAL
 default-group-policy GroupPolicy_main
tunnel-group main webvpn-attributes
 group-alias main enable
!
class-map outside-class1
 match access-list outside_mpc_1
class-map DMZ-class
 match access-list DMZ_mpc
class-map inspection_default
 match default-inspection-traffic
class-map outside-class
 match access-list outside_mpc
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ip-options 
 class class-default
  user-statistics accounting
policy-map outside-policy
 class outside-class
  priority
 class outside-class1
  police input 2000000 1500
policy-map DMZ-policy
 class DMZ-class
  police output 2000000 1500
!
service-policy global_policy global
service-policy outside-policy interface outside
service-policy DMZ-policy interface DMZ
prompt hostname context 
no call-home reporting anonymous
hpm topN enable
Cryptochecksum:XX
: end

sh ver

Result of the command: "sh ver"

Cisco Adaptive Security Appliance Software Version 9.2(4) 
Device Manager Version 7.5(1)

Compiled on Tue 14-Jul-15 22:19 by builders
System image file is "disk0:/asa924-k8.bin"
Config file at boot was "startup-config"

ciscoasa up 5 hours 38 mins

Hardware:   ASA5505, 512 MB RAM, CPU Geode 500 MHz,
Internal ATA Compact Flash, 128MB
BIOS Flash M50FW016 @ 0xfff00000, 2048KB

Encryption hardware device : Cisco ASA-5505 on-board accelerator (revision 0x0)
                             Boot microcode        : CN1000-MC-BOOT-2.00 
                             SSL/IKE microcode     : CNLite-MC-SSLm-PLUS-2.06
                             IPSec microcode       : CNlite-MC-IPSECm-MAIN-2.09
                             Number of accelerators: 1

 0: Int: Internal-Data0/0    : address is 442b.034a.96fa, irq 11
 1: Ext: Ethernet0/0         : address is 442b.034a.96f2, irq 255
 2: Ext: Ethernet0/1         : address is 442b.034a.96f3, irq 255
 3: Ext: Ethernet0/2         : address is 442b.034a.96f4, irq 255
 4: Ext: Ethernet0/3         : address is 442b.034a.96f5, irq 255
 5: Ext: Ethernet0/4         : address is 442b.034a.96f6, irq 255
 6: Ext: Ethernet0/5         : address is 442b.034a.96f7, irq 255
 7: Ext: Ethernet0/6         : address is 442b.034a.96f8, irq 255
 8: Ext: Ethernet0/7         : address is 442b.034a.96f9, irq 255
 9: Int: Internal-Data0/1    : address is 0000.0003.0002, irq 255
10: Int: Not used            : irq 255
11: Int: Not used            : irq 255

Licensed features for this platform:
Maximum Physical Interfaces       : 8              perpetual
VLANs                             : 20             DMZ Unrestricted
Dual ISPs                         : Enabled        perpetual
VLAN Trunk Ports                  : 8              perpetual
Inside Hosts                      : Unlimited      perpetual
Failover                          : Active/Standby perpetual
Encryption-DES                    : Enabled        perpetual
Encryption-3DES-AES               : Enabled        perpetual
AnyConnect Premium Peers          : 25             perpetual
AnyConnect Essentials             : 25             perpetual
Other VPN Peers                   : 25             perpetual
Total VPN Peers                   : 25             perpetual
Shared License                    : Disabled       perpetual
AnyConnect for Mobile             : Enabled        perpetual
AnyConnect for Cisco VPN Phone    : Enabled        perpetual
Advanced Endpoint Assessment      : Enabled        perpetual
UC Phone Proxy Sessions           : 2              perpetual
Total UC Proxy Sessions           : 2              perpetual
Botnet Traffic Filter             : Disabled       perpetual
Intercompany Media Engine         : Disabled       perpetual
Cluster                           : Disabled       perpetual

This platform has an ASA 5505 Security Plus license.

Configuration register is 0x1

Added: show logging once user's reported internet absense http://pastebin.com/3CqXGtuk

  • Can you post show version? – Chris Balmer Sep 5 '15 at 11:39
  • Sure, done it in the main post – Maxim Sep 5 '15 at 11:44
  • That rules out my initial thought about limited inside hosts. Can you add 'logging buffered debug' and 'logging buffer-size 52000' to your config? Then when it happens again, do a show logging and post the logs from a couple min before the problem started until you ran the command? – Chris Balmer Sep 5 '15 at 11:51
  • Done, waiting for the issue. I googled a lot and licensing is the first I checked... Thanks! – Maxim Sep 5 '15 at 12:11
  • 2
    The issue has been resolved! I have tried to analyze non-working hosts arp tables and found ASA's mac address change. Decreased ASA's log verbosity to Warning level, rebooted it and got a message during boot IP address collision detected between host 192.168.0.1 at f80f.4197.a18d and interface inside, So found host with the MAC noted. Despite it has different IP address and it is separate point to investigate, host's disconnection immediately resolves the issue. – Maxim Sep 6 '15 at 7:39
3

The issue has been resolved! I have tried to analyze non-working hosts ARP tables and found ASA's MAC address change. Decreased ASA's log verbosity to Warning level, rebooted it and got a message during boot

IP address collision detected between host 192.168.0.1 at f80f.4197.a18d and interface inside

So found host with the MAC noted. Despite it has different IP address and it is separate point to investigate, host's disconnection immediately resolves the issue.

  • You should accept your answer so that the question doesn't keep popping up forever, looking for an answer. – Ron Maupin Mar 19 '18 at 20:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.