11

I'm looking for some post-event advice so this event doesn't happen again.

We have a network core of two Cisco 4500x switches, configured for VSS redundancy. From those, we have iSCSI devices, our HP bladecenter for our vSphere, plus aggregated links to our user access switches, and a pair of 4948e switches for copper devices in our server room. Off the 4948es we have a pair of 2960 switches for two ISP links, and a pair of ASA as firewalls. Pretty decent redundancy, except for a lot of the devices that connect to the 4948e only have single NICs - only so much we can do.

We're preparing to replace our current user access switches (old Extremes) with Meraki. We're also implementing Meraki APs to replace our current Arubas. Part of the wireless project involves making some new VLANs and subnets, for AP management and guest wireless.

We had two defined VLANs (20 and 40) on the 4500x that were not used anywhere - confirmed that the subnets were empty, no ports using them, etc. I went into the 4500x and issued "no interface vlan 20", and then rebuilt it with the subnet I wanted. I then added it to the two 10Gb ports that are connected to the Meraki

switchport trunk allowed <previous list plus two VLANs above plus existing wireless VLAN>

I noticed that the 20 and 40 VLANs were shutdown, so I issued no shutdown on them. I lost access to the Merakis at that point, so I realized I hadn't added the VLANs to the port channel interface for that link.

Half of our environment became unreachable at this point

Our Internet link went extremely flakey. Our Avaya VoIP phones couldn't dial in or out. We have a couple of copper-connected iSCSI devices that became unavailable - no outage for anything user-facing, but our backups and mail archive became impacted. I went into the server room, and disconnected the Merakis from the 4500x (unplugged both 10Gb fiber ports) in case I had somehow created a loop - no change. I admit to simply staring at this for a while at that point.

I pulled up Orion and noted that one of our external switches (the Cat2960) and one of our ASA pair were down as well. Apparent that we had some sort of partial LAN connectivity loss, but the ASA pair are also connected with crossover to each other, and their uplinks didn't go down, so they didn't failover to what our internal devices could reach. I shut down the "down" ASA and the internet became reachable again.

I called TAC, and after a couple of hours of wresting with the tech who kept nitpicking every port config for each downed host I was showing him on the 4500x, I logged into one of our 4948e switches and showed how it couldn't ping things that were directly connected and up - one of our Windows-based copper iSCSI devices, an iLO interface on our bladecenter, etc.

He had looked over the logs and didn't find anything, but at this point he said "Looks like a spanning-tree bug even if I don't see that in the logs", so we rebooted the 4948e and all of its directly-connected hosts came right back up - including the Avaya cabinet, so our phones started working again. We still had problems in the 4500x fiber-connected devices - dead paths, since it was all redundant. He wanted to power-cycle it ungracefully, but this has all of our 10 Gbit iSCSI, and that would have made our vSphere environment (essentially all of our servers) have a bad week. I talked him into doing a graceful redundancy switchover, which took care of the remaining problems.

TL;DR : I made a fairly innocuous change to our core, and caused a hideous problem. Did I make a config mistake that should have been predicted to cause this - eg, if I had no-shutdown the VLANs first and added them to the portchannel and then the ports, would this have been avoided? The Cisco tech didn't say that; he said, with uptimes over a year and old IOS versions, situations like this aren't surprising.

4500x: Cisco IOS Software, IOS-XE Software, Catalyst 4500 L3 Switch Software (cat4500e-UNIVERSALK9-M), Version 03.04.05.SG RELEASE SOFTWARE (fc1) ROM: 15.0(1r)SG10

4948e: Cisco IOS Software, Catalyst 4500 L3 Switch Software (cat4500e-IPBASEK9-M), Version 15.0(2)SG10, RELEASE SOFTWARE (fc1) ROM: 12.2(44r)SG11

5

It sounds like you created a broadcast storm, and the only way to stop it is to power down the switches. Having lived through this several times, we have adopted some best practices recommended by Cisco:

  • You should only have a VLAN extend to a single access switch. You can have as many VLANs as you like on an access switch, but the VLANs on any access switch should not be trunked to any other access switch, only to the distribution switch. Enforce this by manually disabling all other VLANs on a trunk with the switchport trunk allowed vlan command.
  • A distribution switch should not have any access interfaces on it, only distribution trunk interfaces.
  • Do not use VTP (set all switches to transparent mode).
  • Your access interfaces should have portfast and bpduguard enabled. You can enable those globally for all your access interfaces, and your trunk interfaces will remain unaffected. If you accidentally connect a switch to an access interface, this will cause the interface to go into err-diable and prevent STP loops.
  • Do not connect an access switch to another access switch. Only connect access switches to distribution switches, and only on trunk interfaces.

These best practices will prevent almost all STP problems, and isolate any problems which do happen to a single access switch.

  • 2
    Ah yes. Someday, I hope to work on a network that has enough money, no "weird" (i.e., L2) applications, docile user community, and sufficient management support to follow all the recommended, good sense practices. Some day. – Ron Trunk Sep 28 '16 at 15:04
  • 1. The first suggestion about VLANs and access switches, I'm not sure I understand. – mfinni Sep 28 '16 at 15:42
  • 2. Our "distribution" is presumably our 4500x, which is mostly trunks but has some iSCSI fiber connections. – mfinni Sep 28 '16 at 15:43
  • 3. Avoid VTP - will consider, don't think anything is set "transparent" today – mfinni Sep 28 '16 at 15:43
  • 4. portfast and bdpuguard - will review this suggestion as well – mfinni Sep 28 '16 at 15:44
3

In addition to Ron Maupin's excellent advice above, I also found several posts on Cisco's forum about a potential big mistake I made in the process. I added the VLANs to the physical port interfaces first, not the port-channel interface that they were members of. The latter is the proper way to do it, and I may have caused the problem.

  • 2
    You can do it the way you did, if the member interfaces are down. In general, I have found that I want the member interfaces down, do all the configuration, including the port channel, then, once it is all the way I want it, bring things up. – Ron Maupin Sep 29 '16 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.