This is very strange issue i am noticing. In our datacenter we have every switch in rack has cisco vPC configured in short whole datacenter running vPC so there is no loop for spanning-tree anywhere period.
But have noticed one issue which i ignore earlier that when i plug brand new TOR switch in rack and attach to distribution switch and after that as soon as i bring up vPC, it flood traffic in network just like it mirring traffic.
I setup tcpdump on one of node and watching live packet stream on console and as soon as i configure vPC i noticed very high flooding on screen and in short i can see every hosts traffic on that host tcpdump command. like every machine sending traffic to that host and its happening everywhere not just single host ( duration of flooding is less than 10 second).
What do you guys think of this kind of behavior, is it normal or i should go deep?
Updates - 1
- We have very basic vPC configuration ( We are not using FEX )
- TOR switches are N3K and Distribution switches are N9K
[ Distribution switches ]
VPC configuration
vpc domain 4
peer-switch
role priority 10
peer-keepalive destination 10.10.10.2 source 10.10.10.1
auto-recovery
ip arp synchronize
VPC peer link port channel
interface port-channel999
description *** vPC Peer-Link ***
switchport mode trunk
switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
spanning-tree port type network
speed 40000
no negotiate auto
vpc peer-link
Link connected to TOR switches
interface port-channel403
switchport mode trunk
switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
speed 40000
mtu 9216
no negotiate auto
vpc 403
[ TOR switches ]
vpc domain 403
role priority 10
peer-keepalive destination 172.29.30.11 source 172.29.30.10
auto-recovery
vPC peer-link ports
interface port-channel999
description *** vPC Peer-Link ***
switchport mode trunk
switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
spanning-tree port type network
speed 40000
vpc peer-link
Trunk connected to distribution switches
interface port-channel3
switchport mode trunk
switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
speed 40000
vpc 3
EDIT - 1
I have two type of servers, some servers has bonding configure with vPC and some servers doesn't have bonding configured and i can tell you i am seeing this behavior on those servers doesn't have bonding configured.
In above diagram server has two NIC,
nic1
is configured for VLAN tagging to run multiple VLANsnic2
is configured for SR-IOV ( This server is openstack compute node )
I am seeing flood of traffic coming on nic1
but not nic2
Update - 2
We have 48 racks in datacenter and every rack has two TOR switches has described above, We have two host configuration.
- bonding host ( These host are normal application host ) - ** Not ** getting
etherror
- non-bonding host ( openstack compute node using SR-IOV because SR-IOV doesn't support bonding) - getting
Etherror
I am seeing this error where ever openstack
compute node located in racks so it's everywhere not specific rack or host, feel like where openstack host that time seeing all of the traffic coming to them.. just like you sending unwanted traffic to host.
This is very clear this is happening on only non-bonding servers orphen
ports because other host who has bonding configure they are not seeing this kind of flood.
Update - 3
Distribution (One connected to core is ROOT bridge for all VLAN )
show spanning-tree vlan 200
VLAN0200
Spanning tree enabled protocol rstp
Root ID Priority 32968
Address 0023.04ee.be01
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32968 (priority 32768 sys-id-ext 200)
Address 0023.04ee.be01
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po23 Desg FWD 1 128.4118 (vPC) P2p
Po24 Desg FWD 1 128.4119 (vPC) P2p
TOR switch (N3K )
VLAN0200
Spanning tree enabled protocol rstp
Root ID Priority 32968
Address 0023.04ee.be01
Cost 2
Port 4110 (port-channel15)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32968 (priority 32768 sys-id-ext 200)
Address 0023.04ee.be73
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po15 Root FWD 1 128.4110 (vPC) P2p
Po131 Desg FWD 1 128.4226 (vPC) Edge P2p
Po136 Desg FWD 1 128.4231 (vPC) Edge P2p
Po999 Desg FWD 1 128.5094 (vPC peer-link) Network P2p
Eth1/12 Desg FWD 2 128.12 Edge P2p
Eth1/13 Desg FWD 2 128.13 Edge P2p
N3K vPC connected to server
interface port-channel129
switchport mode trunk
switchport trunk native vlan 40
switchport trunk allowed vlan 10-11,20-21,28-31,40,50,100,200
spanning-tree port type edge trunk
spanning-tree bpduguard enable
speed 10000
vpc 129
I kept vPC configuration on switch side but didn't configure bonding on server side so in short on switch its showing vPC is down and bonding is down to, reason i didn't do that because i thought if tomorrow i need bonding i don't need to re-configure switch..