4507r Supervisor engines aren't operating redundantly

Question

The tl;dr : I have two supervisor engines in a 4507r but the standby unit doesn't appear to be ready to perform when the active module dies.

On Sunday I had a network problem that appeared to be ISP related. I could connect to the VPN but internet traffic was slow, around 500 Kbps. I eventually found it to be a problem on our LAN. No switch log files pointed to an obvious cause, so I rebooted switches, starting at the 4507R. It didn't come back up after a few minutes. When I checked on it, the "status" light was red, and the supervisor module below it wasn't active. Long story short, just reseating the active blade was enough to get it running again. I'd still like to ensure the standby unit can take over. It doesn't appear to be able to.

show version :

core#sh ver
...
IOS (tm) Catalyst 4000 L3 Switch Software (cat4000-IS-M), Version 12.1(12c)EW1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
...
Image text-base: 0x00000000, data-base: 0x00CA7368

ROM: 12.1(12r)EW
Dagobah Revision 63, Swamp Revision 24

core uptime is 5 days, 13 hours, 16 minutes
System returned to ROM by power-on
System restarted at 05:53:53 EST Sun Dec 16 2018
System image file is "bootflash:cat4000-is-mz.121-12c.EW1.bin"

cisco WS-C4507R (XPC8245) processor (revision 5) with 524288K bytes of memory.
Processor board ID FOX06381MAH
Last reset from PowerUp
76 Gigabit Ethernet/IEEE 802.3 interface(s)
403K bytes of non-volatile configuration memory.

Configuration register is 0x2102

sh mod shows a Standby Supervisor in slot 2, but there is no data reported in the second table showing MAC addresses and SW/FW versions.

core#sh mod

Mod  Ports Card Type                              Model             Serial No.
----+-----+--------------------------------------+-----------------+-----------
 1      2  1000BaseX (GBIC) Supervisor(active)    WS-X4515          JAB0704058G
 2         Standby Supervisor
 3     18  1000BaseX (GBIC)                       WS-X4418-GB       JAE065100XB
 4     48  10/100/1000BaseTX (RJ45)               WS-X4448-GB-RJ45  JAE10191Q4Q
 5         Seeprom Not Programmed
 7      6  1000BaseX (GBIC)                       WS-X4306-GB       JAE0838VVQK

 M MAC addresses                    Hw  Fw           Sw               Status
--+--------------------------------+---+------------+----------------+---------
 1 000b.fdd5.3ec0 to 000b.fdd5.3ec1 1.2 12.1(12r)EW  12.1(12c)EW1, EA Ok
 2 Unknown                              Unknown      Unknown          Other
 3 0009.7ceb.9624 to 0009.7ceb.9635 1.1                               Ok
 4 0017.957a.ca70 to 0017.957a.ca9f 1.2                               Ok
 5 Seeprom Not Programmed
 7 000f.2494.dddc to 000f.2494.dde1 2.2                               Ok

show redundancy states shows the standby module is disabled.

core#sh redundancy states
       my state = 13 -ACTIVE
     peer state = 1  -DISABLED
           Mode = Simplex
           Unit = Primary
        Unit ID = 1

Redundancy Mode (Operational) = RPR
Redundancy Mode (Configured)  = RPR
     Split Mode = Disabled
   Manual Swact = Disabled  Reason: Simplex mode
 Communications = Down      Reason: Simplex mode

   client count = 4
 client_notification_TMR = 30000 milliseconds
          keep_alive TMR = 9000 milliseconds
        keep_alive count = 0
    keep_alive threshold = 18
           RF debug mask = 0x0

but "startup-config" (and running) shows the commands pertinent to redundancy.

redundancy
 mode rpr
 main-cpu
  auto-sync standard

The only local filesystem is "bootflash". There are no CF cards installed on either module.

core#dir /all /rec
Directory of bootflash:/*

    1  -rw-     8150628   Apr 04 2002 08:42:15  cat4000-is-mz.121-12c.EW1.bin
    2  -rw-     6088068   Apr 04 2002 08:47:32  k2diags.2-0-1.bin
    3  -rw-       10648   Aug 03 2004 15:34:49  y
    4  -rw-       10648   Aug 03 2004 15:37:59  n
    5  -rw-       10648   Aug 03 2004 15:45:37  test

"startup-config" shows the switch is to boot from the "cat4000-is-mz.121-12c.EW1.bin" file.

boot system bootflash:cat4000-is-mz.121-12c.EW1.bin

Some of the Cisco documentation I've read doesn't say how to see what software image the standby module is booting from. If there's no "slaveslot", then is it reverting to the BIN file in bootflash?

During my troubleshooting, the 4507 booted to a "rommon" prompt when I moved the standby module to slot 1. I installed a spare supervisor (WS-X4515) into slot 1 and it had the config of our other 4507, which wasn't usable on this unit.

I'd like to push the startup-config to the slave module - or otherwise ensure the config is already present - and test performing a redundancy force-switchover to verify the module is capable of taking over.

Does anyone see any steps I've overlooked?

EDIT: added sh power output

core>sh power
Power                                             Fan     Inline
Supply  Model No          Type       Status       Sensor  Status
------  ----------------  ---------  -----------  ------  ------
PS1     PWR-C45-1300ACV   AC 1300W   good         good    good
PS2     PWR-C45-1300ACV   AC 1300W   good         good    good

Power Supply    Max     Min     Max     Min     Absolute
(Nos in Watts)  Inline  Inline  System  System  Maximum
--------------  ------  ------  ------  ------  --------
PS1              800     260    1000     460    1300
PS2              800     260    1000     460    1300

Power Summary
 (in Watts)    Available   Used   Remaining
-------------  ---------  ------  ---------
System Power     1000       545      455
Inline Power      800         0      800
Maximum Power    1260       545      715

Power supplies needed by system : 1

                         Power Used  Power Used
Mod   Model               (online)   (in Reset)
----  -----------------  ----------  ----------
 1    WS-X4515               110           110
 2    Standby Sup            110           110
 3    WS-X4418-GB             80            50
 4    WS-X4448-GB-RJ45       120            72
 5    Not available            0             0
 7    WS-X4306-GB             35            30

I know the software is super old. We don't have a license or support contract to upgrade AFAIK. I inherited this network. My company is moving buildings in the next six months. I get all new hardware then. I just want to ensure I don't have any redundancy issues during peak prod times before then.

Answer 1

Redundancy on these Catalyst platforms operates in two modes: RPR (Route Processor Redundancy) and SSO (Stateful Switchover).

In RPR mode, the Standby boots up "partially". When the Active dies for whatever reason, the Standby takes a few minutes to finish booting up completely and then takes over as Active. This process takes a few minutes.

In SSO mode, the Standby is booted up completely and ready to take over within a few milliseconds.

Now, your system appears to be configured for RPR. Maybe SSO is not even supported on this vintage device, you need to verify this from the documentation.

In any case, right now your Standby does not appear to be working even in RPR mode. If it were, you would have seen "peer state: STANDBY COLD" in the output of "show redundancy". Instead you are seeing DISABLED. In all probability, it did not even get past the ROMMON and boot IOS.

Things to check:

(1) Check the syslog from the bootup to see if there are any messages pertaining to Slot 2

(2) Connect a console cable to the seemingly faulty supervisor, and then reset it (either by using the physical reset button on the front panel of the supervisor, or by simply removing and re-inserting it). Observe what is shown on the console. This is safe to do whilst the Active Supervisor is working in Slot 1.

(3) Even if you don't have a console cable, resetting the seemingly faulty supervisor as above may work. Observe any syslogs appearing on the active as the standby is attempting to come up.

(4) Upgrade. Your image is from 2004, i.e. you have 14 years of software vulnerabilities on your system :-)