2

I have a Cisco Aironet 1142 (AIR-LAP1142N-A-K9) running autonomous firmware (c1140-k9w7-mx.153-3.JBB1) which started frequently rebooting it self today due to watchdog timeouts. I've checked sh proc cpu history and it shows that the CPU is stuck at 100% usage since it last rebooted which could explain the watchdog timeouts. But if I check sh proc cpu none of the processes have any CPU usage for 5 seconds or the 5 minutes counters which is perplexing to me. The CPU will remain stuck at 100% with the Ethernet cable unplugged and a single client associated to the access point. How can I go about determining the root cause of this issue, and what exactly is causing this high CPU utilization?

sh ver

Cisco IOS Software, C1140 Software (C1140-K9W7-M), Version 

15.3(3)JBB1, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 23-Jul-15 10:50 by prod_rel_team

ROM: Bootstrap program is C1140 boot loader
BOOTLDR: C1140 Boot Loader (C1140-BOOT-M) Version 12.4(18a)JA, RELEASE SOFTWARE (fc4)

ap1 uptime is 42 minutes
System returned to ROM by watchdog timer expired
System image file is "flash:/c1140-k9w7-mx.153-3.JBB1/c1140-k9w7-xx.153-3.JBB1"
Last reload reason: 



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco AIR-AP1142N-A-K9 (PowerPC405ex) processor (revision B0) with 98294K/32768K bytes of memory.
Processor board ID FTX1318S0HK
PowerPC405ex CPU at 586Mhz, revision number 0x147E
Last reset from watchdog timer expired
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address: 00:22:90:91:57:C6
Part Number                          : 73-11451-05
PCA Assembly Number                  : 800-30554-02
PCA Revision Number                  : B0
PCB Serial Number                    : FOC13160ZW1
Top Assembly Part Number             : 800-31273-01
Top Assembly Serial Number           : FTX1318S0HK
Top Revision Number                  : A0
Product/Model Number                 : AIR-LAP1142N-A-K9   



Configuration register is 0xF

sh proc cpu history

ap1   02:27:11 AM Monday Mar 1 1993 UTC



      111111111111111111111111111111111111111111111111111111111111
      000000000000000000000000000000000000000000000000000000000000
      000000000000000000000000000000000000000000000000000000000000
  100 ************************************************************
   90 ************************************************************
   80 ************************************************************
   70 ************************************************************
   60 ************************************************************
   50 ************************************************************
   40 ************************************************************
   30 ************************************************************
   20 ************************************************************
   10 ************************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)



      1111111111111111111111111111111111111111                    
      0000000000000000000000000000000000000000                    
      0000000000000000000000000000000000000000                    
  100 ########################################                    
   90 ########################################                    
   80 ########################################                    
   70 ########################################                    
   60 ########################################                    
   50 ########################################                    
   40 ########################################                    
   30 ########################################                    
   20 ########################################                    
   10 ########################################                    
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%






  100                                                                         
   90                                                                         
   80                                                                         
   70                                                                         
   60                                                                         
   50                                                                         
   40                                                                         
   30                                                                         
   20                                                                         
   10                                                                         
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0  
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%

sh proc cpu sorted 5sec

CPU utilization for five seconds: 100%/0%; one minute: 100%; five minutes: 100%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process 
   1           3          28        107  0.00%  0.00%  0.00%   0 Chunk Manager    
   2           0         490          0  0.00%  0.00%  0.00%   0 Load Meter       
   3        1994        1716       1162  0.00%  0.00%  0.00%   0 Exec             
   4           0           1          0  0.00%  0.00%  0.00%   0 RO Notify Timers 
   5        1650         330       5000  0.00%  0.00%  0.00%   0 Check heaps      
   6           0           1          0  0.00%  0.00%  0.00%   0 Pool Manager     
   7           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro 
   8           1           2        500  0.00%  0.00%  0.00%   0 Timers           
   9           0          58          0  0.00%  0.00%  0.00%   0 WATCH_AFS        
  10           0           1          0  0.00%  0.00%  0.00%   0 AAA_SERVER_DEADT 
  11           0           1          0  0.00%  0.00%  0.00%   0  
...

sh run

Current configuration : 5750 bytes
!
version 15.3
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
service sequence-numbers
!
hostname ap1
!
!
logging rate-limit console 9
enable secret 5 CENSORED
!
no aaa new-model
no ip source-route
no ip cef
ip domain name CENSORED
ip name-server 10.1.5.1
ip name-server 10.1.5.2
!
!
!
!
dot11 syslog
dot11 vlan-name Guest_Wi-Fi vlan 3
dot11 vlan-name Intranet vlan 1
dot11 vlan-name Wi-Fi vlan 2
!
dot11 ssid CENSORED
   vlan 3
   authentication open 
   authentication key-management wpa version 2
   wpa-psk ascii 7 CENSORED
   information-element ssidl
!
dot11 ssid CENSORED
   vlan 2
   authentication open 
   authentication key-management wpa version 2
   wpa-psk ascii 7 CENSORED
   information-element ssidl
!
!
dot11 network-map
!
no ipv6 cef
!
crypto pki trustpoint TP-self-signed-2425444294
 enrollment selfsigned
 subject-name cn=CENSORED
 revocation-check none
 rsakeypair TP-self-signed-2425444294
!
!
crypto pki certificate chain TP-self-signed-2425444294
 certificate self-signed 01
    CENSORED
    quit
username admin privilege 15 secret 5 CENSORED
!
!
bridge irb
!
!
!
interface Dot11Radio0
 no ip address
 !
 encryption vlan 2 mode ciphers aes-ccm 
 !
 encryption vlan 3 mode ciphers aes-ccm 
 !
 ssid CENSORED
 !
 ssid CENSORED
 !
 antenna gain 0
 speed  basic-1.0 basic-2.0 basic-5.5 basic-11.0 basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0. m1. m2. m3. m4. m5. m6. m7. m8. m9. m10. m11. m12. m13. m14. m15.
 station-role root
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 spanning-disabled
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
!
interface Dot11Radio0.2
 description Wi-Fi
 encapsulation dot1Q 2
 bridge-group 2
 bridge-group 2 subscriber-loop-control
 bridge-group 2 spanning-disabled
 bridge-group 2 block-unknown-source
 no bridge-group 2 source-learning
 no bridge-group 2 unicast-flooding
!
interface Dot11Radio0.3
 description Guest_Wi-Fi
 encapsulation dot1Q 3
 bridge-group 3
 bridge-group 3 subscriber-loop-control
 bridge-group 3 spanning-disabled
 bridge-group 3 block-unknown-source
 no bridge-group 3 source-learning
 no bridge-group 3 unicast-flooding
!
interface Dot11Radio1
 no ip address
 !
 encryption vlan 2 mode ciphers aes-ccm 
 !
 encryption vlan 3 mode ciphers aes-ccm 
 !
 ssid CENSORED
 !
 ssid CENSORED
 !
 antenna gain 0
 peakdetect
 no dfs band block
 speed  basic-6.0 basic-9.0 basic-12.0 basic-18.0 basic-24.0 basic-36.0 basic-48.0 basic-54.0 m0. m1. m2. m3. m4. m5. m6. m7. m8. m9. m10. m11. m12. m13. m14. m15.
 channel width 40-above
 channel 5180
 station-role root
 bridge-group 1
 bridge-group 1 subscriber-loop-control
 bridge-group 1 spanning-disabled
 bridge-group 1 block-unknown-source
 no bridge-group 1 source-learning
 no bridge-group 1 unicast-flooding
!
interface Dot11Radio1.2
 encapsulation dot1Q 2
 bridge-group 2
 bridge-group 2 subscriber-loop-control
 bridge-group 2 spanning-disabled
 bridge-group 2 block-unknown-source
 no bridge-group 2 source-learning
 no bridge-group 2 unicast-flooding
!
interface Dot11Radio1.3
 encapsulation dot1Q 3
 bridge-group 3
 bridge-group 3 subscriber-loop-control
 bridge-group 3 spanning-disabled
 bridge-group 3 block-unknown-source
 no bridge-group 3 source-learning
 no bridge-group 3 unicast-flooding
!
interface Dot11Radio1.13
!
interface GigabitEthernet0
 no ip address
 duplex auto
 speed auto
!
interface GigabitEthernet0.1
 description Intranet
 encapsulation dot1Q 1 native
 bridge-group 1
 no bridge-group 1 source-learning
!
interface GigabitEthernet0.2
 description Wi-Fi
 encapsulation dot1Q 2
 bridge-group 2
 no bridge-group 2 source-learning
!
interface GigabitEthernet0.3
 description Guest_Wi-Fi
 encapsulation dot1Q 3
 bridge-group 3
 no bridge-group 3 source-learning
!
interface BVI1
 mac-address 0022.9091.57c6
 ip address 10.1.0.3 255.255.0.0
!
ip default-gateway 10.1.0.1
ip forward-protocol nd
ip http server
ip http secure-server
ip http help-path http://www.cisco.com/warp/public/779/smbiz/prodconfig/help/eag
!
!
snmp-server community public RO
snmp-server chassis-id CENSORED
bridge 1 route ip
!
!
!
line con 0
line vty 0 4
 login local
 transport input ssh
line vty 5 15
 login local
 transport input ssh
!
end      

sh log - After upgrading to JC firmware

Cisco IOS Software, C1140 Software (C1140-K9W7-M), Version 15.3(3)JC, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Fri 11-Dec-15 12:56 by prod_rel_team
000014: *Mar  1 01:51:00.078: %SNMP-5-COLDSTART: SNMP agent on host ap1 is undergoing a cold start
000015: *Mar  1 01:51:00.112: %SSH-5-ENABLED: SSH 1.99 has been enabled
000016: *Mar  1 01:51:00.115: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
000017: *Mar  1 01:51:00.115: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000018: *Mar  1 01:51:00.122: %CDP_PD-4-POWER_OK: Full power - AC_ADAPTOR inline power source
000019: *Mar  1 01:51:00.748: %LINEPROTO-5-UPDOWN: Line protocol on Interface BVI1, changed state to up
000020: *Mar  1 01:51:01.115: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
000021: *Mar  1 01:51:01.115: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000022: *Mar  1 01:51:01.136: %DOT11-6-FREQ_SCAN: Interface Dot11Radio0, Scanning frequencies for 8 seconds
000023: *Mar  1 01:51:02.168: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000024: *Mar  1 01:51:03.167: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000025: *Mar  1 01:51:10.372: %DOT11-6-FREQ_USED: Interface Dot11Radio0, frequency 2442 selected
000026: *Mar  1 01:51:10.372: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
000027: *Mar  1 01:51:11.372: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
000028: *Mar  1 01:51:24.198: %DOT11-6-ASSOC: Interface Dot11Radio1, Station   60f8.1db9.a5ce Associated KEY_MGMT[WPAv2 PSK]
000029: *Mar  1 01:51:37.173: 
Cmd 0x16 Timeout d1 After 12s

000030: *Mar  1 01:51:42.255: %EVT-4-WRN: Write of flash:/event.r1 done
000031: *Mar  1 01:51:43.059: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000032: *Mar  1 01:51:44.472: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000033: *Mar  1 01:51:45.310: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000034: *Mar  1 01:51:45.902: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000035: *Mar  1 01:51:46.311: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000036: *Mar  1 01:51:49.651: %DOT11-6-DISASSOC: Interface Dot11Radio1, Deauthenticating Station 60f8.1db9.a5ce Reason: Sending station has left the BSS 
000037: *Mar  1 01:51:49.667: %DOT11-6-ASSOC: Interface Dot11Radio1, Station   60f8.1db9.a5ce Reassociated KEY_MGMT[WPAv2 PSK]
000038: *Mar  1 01:52:03.626: %DOT11-6-DISASSOC: Interface Dot11Radio1, Deauthenticating Station 60f8.1db9.a5ce Reason: Sending station has left the BSS 
000039: *Mar  1 01:52:06.311: 
Cmd 0x16 Timeout d1 After 12s

000040: *Mar  1 01:52:06.352: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000041: *Mar  1 01:52:07.766: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000042: *Mar  1 01:52:08.605: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000043: *Mar  1 01:52:09.605: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000044: *Mar  1 01:52:14.435: %DOT11-6-ASSOC: Interface Dot11Radio1, Station   60f8.1db9.a5ce Reassociated KEY_MGMT[WPAv2 PSK]
000045: *Mar  1 01:52:25.531: %DOT11-6-DISASSOC: Interface Dot11Radio1, Deauthenticating Station 60f8.1db9.a5ce Reason: Sending station has left the BSS 
000046: *Mar  1 01:52:28.608: 
Cmd 0x16 Timeout d1 After 12s

000047: *Mar  1 01:52:28.652: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset
000048: *Mar  1 01:52:30.067: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
000049: *Mar  1 01:52:30.900: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
000050: *Mar  1 01:52:31.040: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000051: *Mar  1 01:52:31.222: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000052: *Mar  1 01:52:31.388: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000053: *Mar  1 01:52:31.563: %DOT11-4-MAXRETRIES: Packet to client 60f8.1db9.a5ce reached max retries, removing the client
000054: *Mar  1 01:52:31.903: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
000055: *Mar  1 01:52:34.986: %DOT11-6-ASSOC: Interface Dot11Radio0, Station   60f8.1db9.a5ce Reassociated KEY_MGMT[WPAv2 PSK]
3
  • Try upgrading to JBB5/JBB6/JC??? (use show proc cpu | ex _0.00%__0.00%__0.00% to filter out all the zeros) Every process showing ZERO is clearly a bug.
    – Ricky
    May 7 '16 at 0:42
  • Upgrading to JC fixed the errenous CPU usage information, and I can confirm it is a bug. I have a pair of 1141s (same hw but a single radio) and they're displaying the same proc cpu info as this 1142. However the device is still rebooting itself due to a watchdog timeout, and is having new issues. Either as soon as a client associates to the dot11radio1 interface the AP either reboots, or the AP resets the interface with the JC firmware. After many resets it finally decided to work. Do I likely have a broken radio interface, or is there anything else you can think of I should check? See log.
    – Matt
    May 7 '16 at 1:47
  • It looks like it started working only after the client associated to the dot11radio0 interface, so i'm guessing I do have a faulty 5 Ghz radio. If you want to answer with the JC firmware upgrade, i'll accept it.
    – Matt
    May 7 '16 at 1:53
2

As Ricky Beam answered in the comments, this appears to be a bug with firmware c1140-k9w7-mx.153-3.JBB1 and is resolved in at least firmware c1140-k9w7-mx.153-3.JC. The other issue I was experiencing with watchdog timeouts appear to be caused by a hardware failure.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.