I have a Cisco Aironet 1142 (AIR-LAP1142N-A-K9) running autonomous firmware (c1140-k9w7-mx.153-3.JBB1) which started frequently rebooting it self today due to watchdog timeouts. I've checked sh proc cpu history and it shows that the CPU is stuck at 100% usage since it last rebooted which could explain the watchdog timeouts. But if I check sh proc cpu none of the processes have any CPU usage for 5 seconds or the 5 minutes counters which is perplexing to me. The CPU will remain stuck at 100% with the Ethernet cable unplugged and a single client associated to the access point. How can I go about determining the root cause of this issue, and what exactly is causing this high CPU utilization?
sh ver
Cisco IOS Software, C1140 Software (C1140-K9W7-M), Version
15.3(3)JBB1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 23-Jul-15 10:50 by prod_rel_team
ROM: Bootstrap program is C1140 boot loader
BOOTLDR: C1140 Boot Loader (C1140-BOOT-M) Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)
ap1 uptime is 42 minutes
System returned to ROM by watchdog timer expired
System image file is "flash:/c1140-k9w7-mx.153-3.JBB1/c1140-k9w7-xx.153-3.JBB1"
Last reload reason:
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
[email protected].
cisco AIR-AP1142N-A-K9 (PowerPC405ex) processor (revision B0) with 98294K/32768K bytes of memory.
Processor board ID FTX1318S0HK
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from watchdog timer expired
1 Gigabit Ethernet interface
2 802.11 Radios
32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:22:90:91:57:C6
Part Number : 73-11451-05
PCA Assembly Number : 800-30554-02
PCA Revision Number : B0
PCB Serial Number : FOC13160ZW1
Top Assembly Part Number : 800-31273-01
Top Assembly Serial Number : FTX1318S0HK
Top Revision Number : A0
Product/Model Number : AIR-LAP1142N-A-K9
Configuration register is 0xF
sh proc cpu history
ap1 02:27:11 AM Monday Mar 1 1993 UTC
111111111111111111111111111111111111111111111111111111111111
000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
100 ************************************************************
90 ************************************************************
80 ************************************************************
70 ************************************************************
60 ************************************************************
50 ************************************************************
40 ************************************************************
30 ************************************************************
20 ************************************************************
10 ************************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
1111111111111111111111111111111111111111
0000000000000000000000000000000000000000
0000000000000000000000000000000000000000
100 ########################################
90 ########################################
80 ########################################
70 ########################################
60 ########################################
50 ########################################
40 ########################################
30 ########################################
20 ########################################
10 ########################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
100
90
80
70
60
50
40
30
20
10
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
sh proc cpu sorted 5sec
CPU utilization for five seconds: 100%/0%; one minute: 100%; five minutes: 100%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 3 28 107 0.00% 0.00% 0.00% 0 Chunk Manager
2 0 490 0 0.00% 0.00% 0.00% 0 Load Meter
3 1994 1716 1162 0.00% 0.00% 0.00% 0 Exec
4 0 1 0 0.00% 0.00% 0.00% 0 RO Notify Timers
5 1650 330 5000 0.00% 0.00% 0.00% 0 Check heaps
6 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager
7 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro
8 1 2 500 0.00% 0.00% 0.00% 0 Timers
9 0 58 0 0.00% 0.00% 0.00% 0 WATCH_AFS
10 0 1 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
11 0 1 0 0.00% 0.00% 0.00% 0
...
sh run
Current configuration : 5750 bytes
!
version 15.3
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service sequence-numbers
!
hostname ap1
!
!
logging rate-limit console 9
enable secret 5 CENSORED
!
no aaa new-model
no ip source-route
no ip cef
ip domain name CENSORED
ip name-server 10.1.5.1
ip name-server 10.1.5.2
!
!
!
!
dot11 syslog
dot11 vlan-name Guest_Wi-Fi vlan 3
dot11 vlan-name Intranet vlan 1
dot11 vlan-name Wi-Fi vlan 2
!
dot11 ssid CENSORED
vlan 3
authentication open
authentication key-management wpa version 2
wpa-psk ascii 7 CENSORED
information-element ssidl
!
dot11 ssid CENSORED
vlan 2
authentication open
authentication key-management wpa version 2
wpa-psk ascii 7 CENSORED
information-element ssidl
!
!
dot11 network-map
!
no ipv6 cef
!
crypto pki trustpoint TP-self-signed-2425444294
enrollment selfsigned
subject-name cn=CENSORED
revocation-check none
rsakeypair TP-self-signed-2425444294
!
!
crypto pki certificate chain TP-self-signed-2425444294
certificate self-signed 01
CENSORED
quit
username admin privilege 15 secret 5 CENSORED
!
!
bridge irb
!
!
!
interface Dot11Radio0
no ip address
!
encryption vlan 2 mode ciphers aes-ccm
!
encryption vlan 3 mode ciphers aes-ccm
!
ssid CENSORED
!
ssid CENSORED
!
antenna gain 0
speed basic-1.0 basic-2.0 basic-5.5 basic-11.0 basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0. m1. m2. m3. m4. m5. m6. m7. m8. m9. m10. m11. m12. m13. m14. m15.
station-role root
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 spanning-disabled
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
!
interface Dot11Radio0.2
description Wi-Fi
encapsulation dot1Q 2
bridge-group 2
bridge-group 2 subscriber-loop-control
bridge-group 2 spanning-disabled
bridge-group 2 block-unknown-source
no bridge-group 2 source-learning
no bridge-group 2 unicast-flooding
!
interface Dot11Radio0.3
description Guest_Wi-Fi
encapsulation dot1Q 3
bridge-group 3
bridge-group 3 subscriber-loop-control
bridge-group 3 spanning-disabled
bridge-group 3 block-unknown-source
no bridge-group 3 source-learning
no bridge-group 3 unicast-flooding
!
interface Dot11Radio1
no ip address
!
encryption vlan 2 mode ciphers aes-ccm
!
encryption vlan 3 mode ciphers aes-ccm
!
ssid CENSORED
!
ssid CENSORED
!
antenna gain 0
peakdetect
no dfs band block
speed basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0. m1. m2. m3. m4. m5. m6. m7. m8. m9. m10. m11. m12. m13. m14. m15.
channel width 40-above
channel 5180
station-role root
bridge-group 1
bridge-group 1 subscriber-loop-control
bridge-group 1 spanning-disabled
bridge-group 1 block-unknown-source
no bridge-group 1 source-learning
no bridge-group 1 unicast-flooding
!
interface Dot11Radio1.2
encapsulation dot1Q 2
bridge-group 2
bridge-group 2 subscriber-loop-control
bridge-group 2 spanning-disabled
bridge-group 2 block-unknown-source
no bridge-group 2 source-learning
no bridge-group 2 unicast-flooding
!
interface Dot11Radio1.3
encapsulation dot1Q 3
bridge-group 3
bridge-group 3 subscriber-loop-control
bridge-group 3 spanning-disabled
bridge-group 3 block-unknown-source
no bridge-group 3 source-learning
no bridge-group 3 unicast-flooding
!
interface Dot11Radio1.13
!
interface GigabitEthernet0
no ip address
duplex auto
speed auto
!
interface GigabitEthernet0.1
description Intranet
encapsulation dot1Q 1 native
bridge-group 1
no bridge-group 1 source-learning
!
interface GigabitEthernet0.2
description Wi-Fi
encapsulation dot1Q 2
bridge-group 2
no bridge-group 2 source-learning
!
interface GigabitEthernet0.3
description Guest_Wi-Fi
encapsulation dot1Q 3
bridge-group 3
no bridge-group 3 source-learning
!
interface BVI1
mac-address 0022.9091.57c6
ip address 10.1.0.3 255.255.0.0
!
ip default-gateway 10.1.0.1
ip forward-protocol nd
ip http server
ip http secure-server
ip http help-path http://www.cisco.com/warp/public/779/smbiz/prodconfig/help/eag
!
!
snmp-server community public RO
snmp-server chassis-id CENSORED
bridge 1 route ip
!
!
!
line con 0
line vty 0 4
login local
transport input ssh
line vty 5 15
login local
transport input ssh
!
end
sh log - After upgrading to JC firmware
Cisco IOS Software, C1140 Software (C1140-K9W7-M), Version 15.3(3)JC, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Fri 11-Dec-15 12:56 by prod_rel_team
000014: *Mar 1 01:51:00.078: %SNMP-5-COLDSTART: SNMP agent on host ap1 is undergoing a cold start
000015: *Mar 1 01:51:00.112: %SSH-5-ENABLED: SSH 1.99 has been enabled
000016: *Mar 1 01:51:00.115: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
000017: *Mar 1 01:51:00.115: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000018: *Mar 1 01:51:00.122: %CDP_PD-4-POWER_OK: Full power - AC_ADAPTOR inline power source
000019: *Mar 1 01:51:00.748: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up
000020: *Mar 1 01:51:01.115: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
000021: *Mar 1 01:51:01.115: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000022: *Mar 1 01:51:01.136: %DOT11-6-FREQ_SCAN: Interface Dot11Radio0, Scanning frequencies for 8 seconds
000023: *Mar 1 01:51:02.168: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000024: *Mar 1 01:51:03.167: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000025: *Mar 1 01:51:10.372: %DOT11-6-FREQ_USED: Interface Dot11Radio0, frequency 2442 selected
000026: *Mar 1 01:51:10.372: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
000027: *Mar 1 01:51:11.372: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
000028: *Mar 1 01:51:24.198: %DOT11-6-ASSOC: Interface Dot11Radio1, Station 60f8.1db9.a5ce Associated KEY_MGMT[WPAv2 PSK]
000029: *Mar 1 01:51:37.173:
Cmd 0x16 Timeout d1 After 12s
000030: *Mar 1 01:51:42.255: %EVT-4-WRN: Write of flash:/event.r1 done
000031: *Mar 1 01:51:43.059: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000032: *Mar 1 01:51:44.472: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000033: *Mar 1 01:51:45.310: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000034: *Mar 1 01:51:45.902: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000035: *Mar 1 01:51:46.311: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000036: *Mar 1 01:51:49.651: %DOT11-6-DISASSOC: Interface Dot11Radio1, Deauthenticating Station 60f8.1db9.a5ce Reason: Sending station has left the BSS
000037: *Mar 1 01:51:49.667: %DOT11-6-ASSOC: Interface Dot11Radio1, Station 60f8.1db9.a5ce Reassociated KEY_MGMT[WPAv2 PSK]
000038: *Mar 1 01:52:03.626: %DOT11-6-DISASSOC: Interface Dot11Radio1, Deauthenticating Station 60f8.1db9.a5ce Reason: Sending station has left the BSS
000039: *Mar 1 01:52:06.311:
Cmd 0x16 Timeout d1 After 12s
000040: *Mar 1 01:52:06.352: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000041: *Mar 1 01:52:07.766: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000042: *Mar 1 01:52:08.605: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000043: *Mar 1 01:52:09.605: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000044: *Mar 1 01:52:14.435: %DOT11-6-ASSOC: Interface Dot11Radio1, Station 60f8.1db9.a5ce Reassociated KEY_MGMT[WPAv2 PSK]
000045: *Mar 1 01:52:25.531: %DOT11-6-DISASSOC: Interface Dot11Radio1, Deauthenticating Station 60f8.1db9.a5ce Reason: Sending station has left the BSS
000046: *Mar 1 01:52:28.608:
Cmd 0x16 Timeout d1 After 12s
000047: *Mar 1 01:52:28.652: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000048: *Mar 1 01:52:30.067: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000049: *Mar 1 01:52:30.900: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000050: *Mar 1 01:52:31.040: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000051: *Mar 1 01:52:31.222: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000052: *Mar 1 01:52:31.388: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000053: *Mar 1 01:52:31.563: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000054: *Mar 1 01:52:31.903: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000055: *Mar 1 01:52:34.986: %DOT11-6-ASSOC: Interface Dot11Radio0, Station 60f8.1db9.a5ce Reassociated KEY_MGMT[WPAv2 PSK]
show proc cpu | ex _0.00%__0.00%__0.00%to filter out all the zeros) Every process showing ZERO is clearly a bug.proc cpuinfo as this 1142. However the device is still rebooting itself due to a watchdog timeout, and is having new issues. Either as soon as a client associates to the dot11radio1 interface the AP either reboots, or the AP resets the interface with the JC firmware. After many resets it finally decided to work. Do I likely have a broken radio interface, or is there anything else you can think of I should check? See log.