6

We recently had a sporadic reboot of a distribution switch for an unknown reason. No recent hardware/software changes have been made. Our log files don't indicate anything leading up to the incident.

The system is a 6509-E running software version 15.1(1)SY on both active and backup supervisor modules.

The system had a primary and backup supervisor card (SUP720) running in SSO. The system was actively functioning in Standby Hot, but the Backup RP never took over as the Active RP, which caused a complete system reboot.

The backup RP eventually booted back up and came back online with the most recent configuration, but the primary supervisor card slot won't boot any cards. We replaced the card with the same card model with no luck. The cards were all tested on a backup 6509 chassis, all of them booted without any issues; which eliminates the possibility of a faulty card.

When trying to boot the cards on this chassis, this is as far as we get:

System Bootstrap, Version 8.4(2) Release
Copyright (c) 1994-2005 by cisco Systems, Inc.
Cat6k-Sup720/SP processor with 524288 Kbytes of main memory

Autoboot executing command: "boot disk0:s72033-adventerprisek9-mz.151-1.SY.bin"
Loading image, please wait ...


Initializing ATA monitor library...

Self extracting the image... [OK]
Self decompressing the image : ########################################################################################]

Cisco IOS Software, s72033_sp Software (s72033_sp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:21 by prod_rel_team
Image text-base: 0x40101778, data-base: 0x427F0760


*Nov 19 17:31:23.407: %SYS-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 17:31:23.407: %PFREDUN-6-STANDBY: Initializing as STANDBY processor for this switch
*Nov 19 17:31:30.179: %SYS-SPSTBY-3-LOGGER_FLUSHING: System pausing to ensure console debugging output.

*Nov 19 17:31:23.607: %SYS-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debugging output.


Firmware compiled 11-Sep-12 12:45 by integ Build [26624]


*Nov 19 17:31:30.179: %OIR-SPSTBY-6-CONSOLE: Changing console ownership to route processor



System Bootstrap, Version 12.2(17r)S4, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2005 by cisco Systems, Inc.
Cat6k-Sup720/RP platform with 524288 Kbytes of main memory

Download Start
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Download Completed! Booting the image.
Self decompressing the image : ########################################################################################]

<...Output Truncated...>   

Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:13 by prod_rel_team
Image text-base: 0x4010F6D8, data-base: 0x44F3E6C0

It doesn't get any further. When we check the active RP, it shows the card is attempting to come online but never fully boots.

router#show redundancy states
       my state = 13 -ACTIVE
     peer state = in progress to standby cold
           Mode = Duplex
           Unit = Secondary
        Unit ID = 6

Redundancy Mode (Operational) = sso
Redundancy Mode (Configured)  = sso
Redundancy State              = sso
     Maintenance Mode = Disabled
 Communications = Up

   client count = 145
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 1
    keep_alive threshold = 19
           RF debug mask = 0x0

Any suggestions on how to fix this? Or is the chassis defective?

show module reaffirms that the card isn't recognized by the system.

router#show module
Mod Ports Card Type                              Model              Serial No.
--- ----- -------------------------------------- ------------------ -----------
  1    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAD10XXXXXX
  2    4  CEF720 4 port 10-Gigabit Ethernet      WS-X6704-10GE      SAD10XXXXXX
  3   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       SAL10XXXXXX
  4   48  CEF720 48 port 1000mb SFP              WS-X6748-SFP       SAL10XXXXXX
  5    0  Supervisor-Other                       Unknown            Unknown
  6    2  Supervisor Engine 720 (Active)         WS-SUP720-3B       SAL12XXXXXX
  7   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAD11XXXXXX
  8   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL10XXXXXX
  9   48  CEF720 48 port 10/100/1000mb Ethernet  WS-X6748-GE-TX     SAL10XXXXXX

Mod MAC addresses                       Hw    Fw           Sw           Status
--- ---------------------------------- ------ ------------ ------------ -------
  1  000a.b866.74da to 000a.b866.74dd   2.4   12.2(14r)S5  15.1(1)SY    Ok
  2  0018.b9a6.a888 to 0018.b9a6.a88b   2.4   12.2(14r)S5  15.1(1)SY    Ok
  3  000a.b8cf.01e8 to 000a.b8cf.0217   1.7   12.2(14r)S5  15.1(1)SY    Ok
  4  000a.b80f.8da0 to 000a.b80f.8dcf   1.7   12.2(14r)S5  15.1(1)SY    Ok
  5  0000.0000.0000 to 0000.0000.0000   0.0   Unknown      Unknown      Unknown
  6  000a.b86d.8df4 to 000a.b86d.8df7   5.7   8.5(2)       15.1(1)SY    Ok
  7  0007.0e0f.98ca to 0007.0e0f.98f9   2.6   12.2(14r)S5  15.1(1)SY    Ok
  8  0019.06a0.bb30 to 0019.06a0.bb5f   2.4   12.2(14r)S5  15.1(1)SY    Ok
  9  0019.2f2f.e280 to 0019.2f2f.e2af   2.4   12.2(14r)S5  15.1(1)SY    Ok

Mod  Sub-Module                  Model              Serial       Hw     Status
---- --------------------------- ------------------ ----------- ------- -------
  1  Centralized Forwarding Card WS-F6700-CFC       SAD10XXXXXX  3.0    Ok
  2  Centralized Forwarding Card WS-F6700-CFC       SAD10XXXXXX  3.0    Ok
  3  Distributed Forwarding Card WS-F6700-DFC3A     SAL10XXXXXX  2.5    Ok
  4  Distributed Forwarding Card WS-F6700-DFC3A     SAL10XXXXXX  2.5    Ok
  5  Policy Feature Card 3       WS-F6K-PFC3B       SAD09XXXXXX  2.3    Other
  6  Policy Feature Card 3       WS-F6K-PFC3B       SAL12XXXXXX  2.4    Ok
  6  MSFC3 Daughterboard         WS-SUP720          SAL12XXXXXX  3.2    Ok
  7  Centralized Forwarding Card WS-F6700-CFC       SAD11XXXXXX  3.1    Ok
  8  Centralized Forwarding Card WS-F6700-CFC       SAL10XXXXXX  2.0    Ok
  9  Centralized Forwarding Card WS-F6700-CFC       SAL10XXXXXX  2.0    Ok

Mod  Online Diag Status
---- -------------------
  1  Pass
  2  Pass
  3  Pass
  4  Pass
  5  Unknown
  6  Pass
  7  Pass
  8  Pass
  9  Pass

Running show redundancy gives a more detailed output of show redundancy states. Note that both active and backup are running the same software versions, 15.1(1)SY.

router#show redundancy
Redundant System Information :
------------------------------
       Available system uptime = 2 weeks, 4 days, 4 hours, 5 minutes
Switchovers system experienced = 0
              Standby failures = 508
        Last switchover reason = none

                 Hardware Mode = Duplex
    Configured Redundancy Mode = sso
     Operating Redundancy Mode = sso
              Maintenance Mode = Disabled
                Communications = Up

Current Processor Information :
-------------------------------
               Active Location = slot 6
        Current Software state = ACTIVE
       Uptime in current state = 2 weeks, 4 days, 4 hours, 3 minutes
                 Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:13 by prod_rel_team
                          BOOT = disk0:s72033-adventerprisek9-mz.151-1.SY.bin,1;
                   CONFIG_FILE =
                       BOOTLDR =
        Configuration register = 0x2102

Peer Processor Information :
----------------------------
              Standby Location = slot 5
        Current Software state = in progress to standby cold
       Uptime in current state = 3 minutes
                 Image Version = Cisco IOS Software, s72033_rp Software (s72033_rp-ADVENTERPRISEK9-M), Version 15.1(1)SY, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2012 by Cisco Systems, Inc.
Compiled Tue 09-Oct-12 15:13 by prod_rel_team
                          BOOT = disk0:s72033-adventerprisek9-mz.151-1.SY.bin,1;
                   CONFIG_FILE =
                       BOOTLDR =
        Configuration register = 0x2102
Improve this question
4
  • Just a few questions which might make it more likely to get some useful answers: 1. Are you able to login on the console on the standby sup? 2. what does 'show module' on the active sup say about the standby? 3. what do the logs on the active sup say about redundancy and the standby sup? 4. what does 'show redundancy' on the active sup say? 5. are you booting the standby sup with identical bootloader and IOS versions as the active one?
    – Teun Vink
    Commented Nov 24, 2013 at 18:17
  • @TeunVink I can't log in to the standby supervisor card when trying to boot it up. Above, where the boot information is shown, is as far as it gets. Both cards are on the same version, both were previously on this version. As previously stated, there haven't been any changes to the system; to include logical and physical.
    – Ryan Foley
    Commented Nov 25, 2013 at 18:42
  • How much time did u give it? It can take well over 30 minutes to get from standby cold to standby hot. And show logging and show tech will give more useful information. I don't see anything wrong from what u posted except patience. SUP720 is slow...
    – sdaffa23fdsf
    Commented Nov 28, 2013 at 9:05
  • @realdreams I gave this a couple days to start up, it just hasn't come back online. No supervisor card that I've ever worked with takes over 30 minutes to initialize and sync. When upgrading the images, it generally only takes around 10 minutes from zero to hot-standby. On our spare chassis, it came online in under 10 minutes; just takes a while to boot-up, the SSO sync only takes a few seconds.
    – Ryan Foley
    Commented Nov 28, 2013 at 9:50

2 Answers 2

Reset to default
2

Based on what you describe and tested I'd expect the slot on the chassis to be broken.

Improve this answer
2
  • 1
    That's been my working assumption as well. I think the OP needs to consider that as a solid possibility
    – Mike Pennington
    Commented Nov 28, 2013 at 11:17
  • @MikePennington This was my assumption as well. I've been routing a change request through our different organizations to replace our primary chassis with a backup we have in storage.
    – Ryan Foley
    Commented Nov 28, 2013 at 11:23
-1

Try swapping the Sup Cards and see which will come up. If the primary one works in the slot of the backup, then you will have to RMA the device.

Improve this answer
2
  • 1
    I already did that. From the OP: "We replaced the card with the same card model with no luck. The cards were all tested on a backup 6509 chassis, all of them booted without any issues; which eliminates the possibility of a faulty card." We also won't be able to RMA the device, this chassis is over 8 years old.
    – Ryan Foley
    Commented Nov 29, 2013 at 21:09
  • Well, in that case you will have to upgrade the device. It sounds like a faulty back panel .. Why don't you open a Cisco TAC? This can help you identify the fault better
    – sikas
    Commented Dec 1, 2013 at 5:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.