Strange flooding during installing new switch in network

Question

This is very strange issue i am noticing. In our datacenter we have every switch in rack has cisco vPC configured in short whole datacenter running vPC so there is no loop for spanning-tree anywhere period.

But have noticed one issue which i ignore earlier that when i plug brand new TOR switch in rack and attach to distribution switch and after that as soon as i bring up vPC, it flood traffic in network just like it mirring traffic.

I setup tcpdump on one of node and watching live packet stream on console and as soon as i configure vPC i noticed very high flooding on screen and in short i can see every hosts traffic on that host tcpdump command. like every machine sending traffic to that host and its happening everywhere not just single host ( duration of flooding is less than 10 second).

What do you guys think of this kind of behavior, is it normal or i should go deep?

Updates - 1

• We have very basic vPC configuration ( We are not using FEX )
• TOR switches are N3K and Distribution switches are N9K

[ Distribution switches ]

VPC configuration

vpc domain 4
  peer-switch
  role priority 10
  peer-keepalive destination 10.10.10.2 source 10.10.10.1
  auto-recovery
  ip arp synchronize

VPC peer link port channel

interface port-channel999
  description *** vPC Peer-Link ***
  switchport mode trunk
  switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
  spanning-tree port type network
  speed 40000
  no negotiate auto
  vpc peer-link

Link connected to TOR switches

interface port-channel403
  switchport mode trunk
  switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
  speed 40000
  mtu 9216
  no negotiate auto
  vpc 403

[ TOR switches ]

vpc domain 403
  role priority 10
  peer-keepalive destination 172.29.30.11 source 172.29.30.10
  auto-recovery

vPC peer-link ports

interface port-channel999
  description *** vPC Peer-Link ***
  switchport mode trunk
  switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
  spanning-tree port type network
  speed 40000
  vpc peer-link

Trunk connected to distribution switches

interface port-channel3
  switchport mode trunk
  switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
  speed 40000
  vpc 3

EDIT - 1

I have two type of servers, some servers has bonding configure with vPC and some servers doesn't have bonding configured and i can tell you i am seeing this behavior on those servers doesn't have bonding configured.

In above diagram server has two NIC,

• nic1 is configured for VLAN tagging to run multiple VLANs
• nic2 is configured for SR-IOV ( This server is openstack compute node )

I am seeing flood of traffic coming on nic1 but not nic2

Update - 2

We have 48 racks in datacenter and every rack has two TOR switches has described above, We have two host configuration.

1. bonding host ( These host are normal application host ) - ** Not ** getting etherror
2. non-bonding host ( openstack compute node using SR-IOV because SR-IOV doesn't support bonding) - getting Etherror

I am seeing this error where ever openstack compute node located in racks so it's everywhere not specific rack or host, feel like where openstack host that time seeing all of the traffic coming to them.. just like you sending unwanted traffic to host.

This is very clear this is happening on only non-bonding servers orphen ports because other host who has bonding configure they are not seeing this kind of flood.

Update - 3

Distribution (One connected to core is ROOT bridge for all VLAN )

show spanning-tree vlan 200

VLAN0200
  Spanning tree enabled protocol rstp
  Root ID    Priority    32968
             Address     0023.04ee.be01
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32968  (priority 32768 sys-id-ext 200)
             Address     0023.04ee.be01
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po23             Desg FWD 1         128.4118 (vPC) P2p
Po24             Desg FWD 1         128.4119 (vPC) P2p

TOR switch (N3K )

  VLAN0200
      Spanning tree enabled protocol rstp
      Root ID    Priority    32968
                 Address     0023.04ee.be01
                 Cost        2
                 Port        4110 (port-channel15)
                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

      Bridge ID  Priority    32968  (priority 32768 sys-id-ext 200)
                 Address     0023.04ee.be73
                 Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

    Interface        Role Sts Cost      Prio.Nbr Type
    ---------------- ---- --- --------- -------- --------------------------------
    Po15             Root FWD 1         128.4110 (vPC) P2p
    Po131            Desg FWD 1         128.4226 (vPC) Edge P2p
    Po136            Desg FWD 1         128.4231 (vPC) Edge P2p
    Po999            Desg FWD 1         128.5094 (vPC peer-link) Network P2p
    Eth1/12          Desg FWD 2         128.12   Edge P2p
    Eth1/13          Desg FWD 2         128.13   Edge P2p

N3K vPC connected to server

interface port-channel129
  switchport mode trunk
  switchport trunk native vlan 40
  switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
  spanning-tree port type edge trunk
  spanning-tree bpduguard enable
  speed 10000
  vpc 129

I kept vPC configuration on switch side but didn't configure bonding on server side so in short on switch its showing vPC is down and bonding is down to, reason i didn't do that because i thought if tomorrow i need bonding i don't need to re-configure switch..

Answer 1

I think I set the path to this answer in the other thread, it took me some time to follow it through.

... when a non-portfast port (regardless if individual or port-channel) goes "FWD", a spanning-tree topology change notification runs through the pertaining spanning tree(s).

See this part of the document: Understanding Spanning-Tree Protocol Topology Changes

In short: A TopologyChangeNotification is sent by a bridge that just had a port go into FWD mode. That TCN gets ack'd by the root bridge with a TopologyChangeAcknowledgement.

Then the root bridge floods a TC BPDU across the pertaining spanning tree(s).

In turn, this causes the bridges to cut short (from 300sec down to forward_delay) their CAM timers and to purge a lot of addresses from their MAC address table(s).

This then leads to a surge of unknown unicast flooding., which I believe is what you are seeing.

What can you do against this?

• make absolutely sure that all your edge ports have portfast or portfast trunk. The way you asked in the other thread makes me believe that this was not the case for host facing port-channels.

• (re)consider the size of your spanning-tree domains.

Addon:

Well actually, with RSTP, things with TC are a little different, but the fundamental principle remains: a Topology Change BPDU leads to switches flushing (parts of) their CAM tables, which in turn leads to a surge of unknown unicast flooding.

http://www.hackandtinker.net/2015/02/19/spanning-tree-protocol-pvstrapid-pvstmst/

RSTP still holds the concept of TCNs, as did legacy STP.  However, rather than having a TCN be sent to the root bridge (which STP only did because of the fact that non-bridges don’t really have a way of communicating changes to other bridges without asking the root to perform these notifications on their behalf), switches will immediately flood BPDUs with the TC flag set.  Any switch sending or receiving these BPDUs with the TC bit set will set a tcWhile timer which is the hello timer plus one second.  It then immediately floods all MACs learned on non-edge designated and root ports.  It then sends BPDUs with the TC flag set, and the process repeats, quickly informing all switches in the topology of the change.

See also here: https://www.cisco.com/c/en/us/support/docs/lan-switching/spanning-tree-protocol/24062-146.html#anc20

When a bridge receives a BPDU with the TC bit set from a neighbor, these occur:

It clears the MAC addresses learned on all its ports, except the one that receives the topology change.

It starts the TC While timer and sends BPDUs with TC set on all its designated ports and root port (RSTP no longer uses the specific TCN BPDU, unless a legacy bridge needs to be notified).

[Addon-2:] (after comment)

I still believe you have a host related issue, in that a non-portfast edge port (most probably a host's port or port-channel) goes into FWD mode, which then triggers its switch to flood a Topology Change BPDU, in turn triggering the flood of unknown unicast.

Resorting to switchport block unicast at this stage is fighting the symptom before finding and confirming the actual cause - not good engineering practice.

You'd also need to evaluate the consequences of Unknown Unicast Flood Blocking in your environment. It might actually be detrimental to some of the rather silent end systems, becoming unreachable to their peers for possibly extended periods of time. Once their CAM entry is ages out, while their peer's ARP cache entries are still valid (e.g. a router has a default ARP timeout of 4hrs), their peer's unicast attempts would have to be unknown unicast flooded by the switches. Still, these packets wouldn't reach the end systems because of UUF blocking, therefore end systems won't hear these frames, and won't react to them. Because they remain silent, the switch they're connected to can't (re)learn their MAC address and will continue to unicast-flood packets with their DstMAC addresses, which they won't hear... ad nauseam, until these peers - or any other peer - eventually decides to (ARP)-Broadcast again for the device's MAC address or the end systems will show some network activity themselves.

So: You first want to check if you actually do have TCNs in your spanning trees and when and where they came from (show spanning-tree vlan XXX detail | include "VLAN|change|from" ).

switch# show spanning-tree vlan 5 detail | include "VLAN|topo|from"
 VLAN0005 is executing the rstp compatible Spanning Tree protocol
  Number of topology changes 1 last change occurred 1979:48:37 ago
          from port-channel1133
  Times:  hold 1, topology change 35, notification 2
  Timers: hello 0, topology change 0, notification 0
 Port 4295 (port-channel200, vPC Peer-link) of VLAN0005 is designated forwarding 
 Port 5226 (port-channel1131, vPC) of VLAN0005 is root forwarding 
 Port 5228 (port-channel1133, vPC) of VLAN0005 is designated forwarding

After a flood event, recurse this investigation downward from the root, and see where the TCN came from. If it leads you to a ToR switch, then chances are that it must've been a host's non-portfast port going into FWD mode. Check the given ToR switch link up/line protocol up events.

Other than that, if you add new ToR Switches to the network, a topology change is perfectly normal, as the upstream switch will unavoidably have the downlink port/port-channel (which must NOT be an edge port) to the new switch go into FWD mode.

[/Addon-2:]