5

Background

I recently upgraded a pair of Catalyst4500s from Sup6E to Sup7E. Before the upgrade, I pre-loaded IOS-XE 3.4.1SG on the Sup7Es, and loaded the migration configurations into bootflash: on the new sups. We needed to perform the upgrade as quickly as possible; and I had only a few hours between the time I cut the tape on the shipping boxes and putting the new Sup7Es into production.

After pulling the old Sup6s out, I started maintenance with one Sup7E in each chassis running 3.4.1SG (which I had pre-loaded); the other two Sup7Es were left on 3.2.2SG to conserve time while I prepared for maintenance. I manually copied 3.4.1SG to slavebootflash: and reloaded when I inserted the standby supervisors.

FWIW, copying IOS to slavebootflash: is much faster than copying with tftp on these Sup7Es.

Maintenance window results

I had a strange result by the end of the maintenance window:

  • Both Sup7E OOB mgmt interfaces on sw1 operated in mgmtVrf
  • One Sup7E OOB mgmt interface on sw2 operated in Mgmt-vrf (this one was ACTIVE)
  • The other Sup7E OOB mgmt interface on sw2 operated in mgmtVrf (this one was STANDBY)

Somehow three out of the four new Sup7Es came up with mgmtVrf on the OOB FastEthernet1 interface, but one was stuck with Mgmt-vrf. I could not find a way to change Mgmt-vrf -> mgmtVrf, or mgmtVrf to Mgmt-vrf. During my attempts to get consistent management VRFs inside my maintenance window, one of the supervisors went lights-out and crashed.

At the end of maintenance, I was:

  • Running out of time
  • Hesitant to keep putting time into this, because it took over five minutes each time I booted IOS-XE on Sup7E
  • Unwilling to risk a config mismatch between Sup7E ACTIVE / STANDBY oob FastEthernet1 (every time I saw this during the maintenance window, I could not successfully fail from ACTIVE -> STANDBY, and still ping FastEthernet1).

Lab work

Thus, I decided to take sw2 STANDBY Sup7E to our lab to try to fix it, hoping that I'd be able to replicate how I managed to get the others into mgmtVrf.

In my attempts to get sw2 STANDBY FastEth1 into Mgmt-vrf in IOS-XE 3.4.1SG, I've had no luck with:

  • Copying a new config with Mgmt-vrf from bootflash:
  • tftping a new startup-config with Mgmt-vrf into nvram: and reloading
  • tftping a new running-config with Mgmt-vrf into the sup
  • Changing the config to use Mgmt-vrf while the switch is running

Question

How can I reconfigure this supervisor to use Mgmt-vrf, or reconfigure the other supervisor to use mgmtVrf?

Console log from sw2

This is what I see when I boot 3.4.1SG on the misbehaving supervisor.

% Use 'vrf definition mgmtVrf' command
% Management interface VRF can not be changed.

... more console output here

Interface FastEthernet1's vrf mgmtVrf does not match with cfg vrf Mgmt-vrf     


Press RETURN to get started!

sw2#show mod | i SUP7
 3     4  Sup 7-E 10GE (SFP+), 1000BaseX (SFP)   WS-X45-SUP7-E      CAT12345678
sw2#
sw2#sh ver | i Software                                                 
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software 
(cat4500e-UNIVERSALK9-M), Version 03.04.01.SG RELEASE SOFTWARE (fc3)
sw2#
sw2# 
sw2#sh ip vrf                                                           
  Name                             Default RD          Interfaces        
  Liin-vrf                         <not set>                                   
  Mgmt-vrf                         <not set>                                                    
  mgmtVrf                          <not set>           Fa1    <---------
sw2#
5

Documenting to help future googlers, since I could find no information about Mgmt-vrf online... I managed to find the solution while I was typing the question above. I remembered that the Sup7Es had IOS-XE 3.2.2 loaded when I pulled them out of the boxes; the other important fact was that 3.2.2 would not turn up the OOB interface on FastEthernet1 while I was staging / licensing the supervisors.

Solution

First I loaded 3.2.2 on the switch again, and booted into it. Then I did a wr erase to zero nvram and reloaded. When the switch rebooted mgmtVrf was gone :-).

Switch#sh ip vrf                                                               
  Name                             Default RD          Interfaces              
  Liin-vrf                         <not set>                                   
  Mgmt-vrf                         <not set>           Fa1                     
Switch#sh ver | i Software                                                     
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software 
(cat4500e-UNIVERSAL-M), Version 03.02.02.SG RELEASE SOFTWARE (fc3)                      
Switch#
Switch#sh bootflash: | i SG                                                    
  1   87674776 Feb 12 2014 18:56:05 +00:00 cat4500e-universal.SPA.03.02.02.SG.1
50-2.SG2.bin                                                                   
  2  125216116 Feb 06 2014 17:13:26 +00:00 cat4500e-universalk9.SPA.03.04.01.SG
.151-2.SG1.bin                                                                 
Switch#

Now I did a second wr e just to be sure, and loaded 3.4.1SG again to check which VRFs came up by default in 3.4.1SG. As one might suspect, some crazy person at Cisco renamed the management interfaces between IOS releases (probably because they were scolded for using a non-standard oob vrf name, since every other Cisco I can think of uses mgmtVrf).

Switch#sh ver | i Software                                                     
Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e
-UNIVERSALK9-M), Version 03.04.01.SG RELEASE SOFTWARE (fc3)                    
Switch#sh ip vrf                                                               
  Name                             Default RD          Interfaces              
  Liin-vrf                         <not set>                                   
  mgmtVrf                          <not set>           Fa1                     
Switch# 

Now the path to fixing this is clear.

| improve this answer | |
  • You must not work with IOS much. :-) wr erase was the very first thing that came to mind. (before reading the entire question) ... and then paste your old config line-by-line. (I've done that on a 125k line config) – Ricky Beam Feb 12 '14 at 20:08
  • I did many wr erase commands in 3.4.1SG. What you missed is that wr e under 3.4.1SG never yields Mgmt-vrf, and wr e under 3.2.2SG never yields mgmtVrf. – Mike Pennington Feb 12 '14 at 20:13
  • No, I got that. wr e see the new default(s), and just go with it. Even if Cisco failed to document it anywhere. (also, this isn't the first time they've done something like this.) – Ricky Beam Feb 12 '14 at 20:18
  • I had a 45 minute window, and 3.2.2SG wasn't on any active sup (deleted when I copied 3.4.1SG over)... Thus your solution is useless for the maintenance window... but yeah, guess what command I used after rolling back to 3.2.2SG in the lab :-) – Mike Pennington Feb 12 '14 at 20:23
  • True... time was the real enemy here. There's only some many things we can think to test in the lab. – Ricky Beam Feb 12 '14 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.