I am dealing with very strange issue but wanted to check with experts if i am doing anything stupid or not. I have small size EVPN+VxLAN clos network (OSFP+iBGP+Multicast+Arp-suppression (no anycastgateway) is in my setup) very standard configuration. My leafs in vPC for redendency. My leafs connected to HP c7000 bladecenter switch 6120XG.
Problem: On blade server i have two 10G nic (nic1 connected to blade-switch-A and nic2 connected to blade-switch-B) so i have configured bonding active-backup because 6120XG doesn't support MLAG. Everything working great at this point. I have Linux PXE kickstart server and i have dedicated VLAN 70 for PXE which i have untagged on 6120XG switch (because PXE doesn't support VLAN tagging). when i reboot blade server and go to PXE boot to kickstart i can see my PXE get ip address from DHCP but after that it stopped pinging that IP and kickstart failing saying no network connection (in short i can't ping that pxe IP).
I have all my VLAN/VNI configured for arp-suppression. if i remove arp-suppression for PXE VLAN/VNI then everything works, PXE successfully able to kickstart my all servers but as soon as i add arp-suppression it stopped working. Any idea how EVPN+VxLAN handle untagged VLANs?
Leaf NVE1 interface config
interface nve1
no shutdown
description ** VTEP/NVE Interface **
host-reachability protocol bgp
source-interface loopback1
member vni 10064
suppress-arp
mcast-group 239.1.1.1
member vni 10065
suppress-arp
mcast-group 239.1.1.1
member vni 10070
suppress-arp <---------- if i remove this it then PXE works.
mcast-group 239.1.1.1
member vni 10100
suppress-arp
mcast-group 239.1.1.1
member vni 10555 associate-vrf
Bladecenter switch untagged VLAN 70
vlan 70
name "pxe"
untagged 1-16
tagged Trk1
no ip address
exit
Update - 1
After debugging i found as soon as PXE boot load start loading CentOS linux at specific place where Linux set VLAN ID 0 on interface to make it native to allow access of all VLAN at that point i have noticed it flood my network with ARP broadcast on that specific VLAN and that cause packet loss on network and my installation stuck.
Look like this untagged frame getting in loop because of arp suppression not sure why. I do have other VLAN also and they are all tagged with VLAN which is working fine without any issue. I haven't seen any issue on them.
[root@pxe-server ~]# tcpdump -i bond0.70 -nn not port 22 and host 10.70.0.112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0.70, link-type EN10MB (Ethernet), capture size 262144 bytes
13:13:23.115377 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.115539 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.115974 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.116110 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.215306 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.215312 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.215823 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
Only way to break this broadcast storm is to remove suppress-arp option from VLAN 70. I think it could be a big in Cisco software because not sure what is the relation of arp with non-tagged frame.
This is my Leaf-1-1 config and Leaf-1-2 is second vPC peer which has pretty similar config. Its very simple network not fancy stuff, I have only 2 vPC pair Leaf as per diagram.
Question: I have enabled suppress-arp on VNI which is pure L2VNI it doesn't have anycast-gateway because my Cisco ASA is gateway for all VLAN. (I am assuming suppress-arp does with L2VNI only without Distributed gateway)
leaf-1-1# show run
cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay
ip domain-lookup
system qos
service-policy type network-qos jumboframes
fabric forwarding anycast-gateway-mac 0000.dead.beef
ip pim rp-address 10.255.0.123 group-list 239.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 1,64-68,70,100,444,555
vlan 64
name inside
vlan 65
vn-segment 10065
vlan 66
vn-segment 10066
vlan 67
vn-segment 10067
vlan 68
vn-segment 10068
vlan 70
name pxe_boot
vn-segment 10070
vlan 100
name public
vn-segment 10100
vlan 444
name BACKUP_VLAN_ROUTING_VPC
vlan 555
name L3VNI
vn-segment 10555
spanning-tree port type edge bpduguard default
spanning-tree loopguard default
spanning-tree vlan 64-68,70,100,555 priority 8192
route-map DIRECT-PERMIT-ALL permit 10
description ** Route-Map for BGP to redist route **
vrf context CUST1
vni 10555
rd auto
address-family ipv4 unicast
route-target both auto
route-target both auto evpn
vrf context management
ip route 0.0.0.0/0 172.30.0.1
hardware access-list tcam region vacl 0
hardware access-list tcam region arp-ether 256
vpc domain 1
peer-switch
role priority 10
peer-keepalive destination 172.30.0.32 source 172.30.0.31
delay restore 90
peer-gateway
delay restore interface-vlan 30
ip arp synchronize
interface Vlan1
no ip redirects
no ipv6 redirects
interface Vlan100
description ** Anycast Gateway For Public **
no shutdown
mtu 9216
vrf member CUST1
ip address 60.25.124.1/23
fabric forwarding mode anycast-gateway
interface Vlan444
description ** Underlay Backup over vPC Peer-Link **
no shutdown
ip address 192.168.1.1/30
ip ospf authentication-key 3 fa3ab8e90610229c
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface Vlan555
description ** L3VNI-For-IRB **
no shutdown
mtu 9216
vrf member CUST1
ip forward
interface port-channel111
description ** Link to enc-k001-1-a **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 111
interface port-channel112
description ** Link to enc-k001-1-b **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 112
interface port-channel121
description ** Link to enc-k001-2-a **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 121
interface port-channel122
description ** Link to enc-k001-2-b **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 122
interface port-channel211
description ** Link to enc-k002-1-a **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 211
interface port-channel212
description ** Link to enc-k002-1-b **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 212
interface port-channel221
description ** Link to enc-k002-2-a **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 221
interface port-channel222
description ** Link to enc-k002-2-b **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 222
interface port-channel311
description ** Link to enc-k003-1-a **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 311
interface port-channel312
description ** Link to enc-k003-1-b **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 312
interface port-channel321
description ** Link to enc-k003-2-a **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 321
interface port-channel322
description ** Link to enc-k003-2-b **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
vpc 322
interface port-channel999
description ** vPC Peer-Link **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100,444,555
spanning-tree port type network
speed 40000
no negotiate auto
vpc peer-link
interface nve1
no shutdown
description ** VTEP/NVE Interface **
host-reachability protocol bgp
source-interface loopback1
member vni 10064
suppress-arp
mcast-group 239.1.1.1
member vni 10065
suppress-arp
mcast-group 239.1.1.1
member vni 10066
suppress-arp
mcast-group 239.1.1.1
member vni 10067
suppress-arp
mcast-group 239.1.1.1
member vni 10068
suppress-arp
mcast-group 239.1.1.1
member vni 10070
mcast-group 239.1.1.1
member vni 10100
suppress-arp
mcast-group 239.1.1.1
member vni 10555 associate-vrf
interface Ethernet1/1
description ** Link to swt-enc-k001-1-a Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 111 mode active
interface Ethernet1/2
description ** Link to swt-enc-k001-1-a Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 111 mode active
interface Ethernet1/3
description ** Link to swt-enc-k001-1-b Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 112 mode active
interface Ethernet1/4
description ** Link to swt-enc-k001-1-b Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 112 mode active
interface Ethernet1/5
description ** Link to swt-enc-k001-2-a Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 121 mode active
interface Ethernet1/6
description ** Link to swt-enc-k001-2-a Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 121 mode active
interface Ethernet1/7
description ** Link to swt-enc-k001-2-b Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 122 mode active
interface Ethernet1/8
description ** Link to swt-enc-k001-2-b Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 122 mode active
interface Ethernet1/9
shutdown
mtu 9216
interface Ethernet1/10
description ** Link to VMware Host-1 **
switchport mode trunk
switchport trunk allowed vlan 64-65,70,100
spanning-tree port type edge trunk
mtu 9216
speed 10000
interface Ethernet1/17
description ** Link to swt-enc-k002-1-a Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 211 mode active
interface Ethernet1/18
description ** Link to swt-enc-k002-1-a Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 211 mode active
interface Ethernet1/19
description ** Link to swt-enc-k002-1-b Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 212 mode active
interface Ethernet1/20
description ** Link to swt-enc-k002-1-b Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 212 mode active
interface Ethernet1/21
description ** Link to swt-enc-k002-2-a Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 221 mode active
interface Ethernet1/22
description ** Link to swt-enc-k002-2-a Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 221 mode active
interface Ethernet1/23
description ** Link to swt-enc-k002-2-b Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 222 mode active
interface Ethernet1/24
description ** Link to swt-enc-k002-2-b Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 222 mode active
interface Ethernet1/33
description ** Link to swt-enc-1-a Port E18 **k003
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 311 mode active
interface Ethernet1/34
description ** Link to swt-enc-k003-1-a Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 311 mode active
interface Ethernet1/35
description ** Link to swt-enc-k003-1-b Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 312 mode active
interface Ethernet1/36
description ** Link to swt-enc-k003-1-b Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 312 mode active
interface Ethernet1/37
description ** Link to swt-enc-2-a Port E18 **k003
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 321 mode active
interface Ethernet1/38
description ** Link to swt-enc-k003-2-a Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 321 mode active
interface Ethernet1/39
description ** Link to swt-enc-k003-2-b Port E18 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 322 mode active
interface Ethernet1/40
description ** Link to swt-enc-k003-2-b Port E19 **
no cdp enable
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100
mtu 9216
speed 10000
channel-group 322 mode active
interface Ethernet2/1
description ** Spine-1 **
no switchport
mtu 9216
medium p2p
ip address 10.1.11.1/31
ip ospf authentication-key 3 XXXXXXX
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet2/2
description ** Spine-2 **
no switchport
mtu 9216
medium p2p
ip address 10.2.11.1/31
ip ospf authentication-key 3 XXXXXXX
ip ospf network point-to-point
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
no shutdown
interface Ethernet2/11
description ** vPC Peer-Link **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100,444,555
speed 40000
no negotiate auto
channel-group 999 mode active
interface Ethernet2/12
description ** vPC Peer-Link **
switchport mode trunk
switchport trunk allowed vlan 64-68,70,100,444,555
speed 40000
no negotiate auto
channel-group 999 mode active
interface mgmt0
vrf member management
no ip redirects
ip address 172.30.0.31/23
interface loopback0
description ** RID/BGP Overlay **
ip address 10.255.1.11/32
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
interface loopback1
description ** VTEP/Overlay **
ip address 10.255.255.11/32
ip address 10.255.255.10/32 secondary
ip ospf authentication-key 3 fa3ab8e90610229c
ip router ospf UNDERLAY-NET area 0.0.0.0
ip pim sparse-mode
cli alias name wr copy running-config startup-config
line console
line vty
boot nxos bootflash:/nxos.9.3.4.bin
router ospf UNDERLAY-NET
router-id 10.255.1.11
log-adjacency-changes
area 0.0.0.0 authentication
router bgp 65001
router-id 10.255.1.11
log-neighbor-changes
template peer VXLAN_SPINE
remote-as 65001
update-source loopback0
address-family ipv4 unicast
address-family l2vpn evpn
send-community
send-community extended
neighbor 10.255.0.1
inherit peer VXLAN_SPINE
description ** iBGP Peer to Spine-1 **
neighbor 10.255.0.2
inherit peer VXLAN_SPINE
description ** iBGP Peer to Spine-2 **
vrf CUST1
log-neighbor-changes
address-family ipv4 unicast
redistribute direct route-map DIRECT-PERMIT-ALL
evpn
vni 10064 l2
rd auto
route-target import auto
route-target export auto
vni 10065 l2
rd auto
route-target import auto
route-target export auto
vni 10066 l2
rd auto
route-target import auto
route-target export auto
vni 10067 l2
rd auto
route-target import auto
route-target export auto
vni 10068 l2
rd auto
route-target import auto
route-target export auto
vni 10070 l2
rd auto
route-target import auto
route-target export auto
vni 10100 l2
rd auto
route-target import auto
route-target export auto
Update-2:
After more testing i found culprit is DHCP packets, For testing purper i spun up DHCP server on other VLANs and noticed as soon as DHCP send/receive broadcast packet and at same time broadcast flood start. (If ARP suppression is enabled). I am going to create Anycast gateway to just prove this theory that arp-suppression doesn't support L2VNI. Please chime in if anyone has any history.
Notes: I am not using any kind of fancy DHCP-Relay feature.
