0

I am dealing with very strange issue but wanted to check with experts if i am doing anything stupid or not. I have small size EVPN+VxLAN clos network (OSFP+iBGP+Multicast+Arp-suppression (no anycastgateway) is in my setup) very standard configuration. My leafs in vPC for redendency. My leafs connected to HP c7000 bladecenter switch 6120XG.

enter image description here

Problem: On blade server i have two 10G nic (nic1 connected to blade-switch-A and nic2 connected to blade-switch-B) so i have configured bonding active-backup because 6120XG doesn't support MLAG. Everything working great at this point. I have Linux PXE kickstart server and i have dedicated VLAN 70 for PXE which i have untagged on 6120XG switch (because PXE doesn't support VLAN tagging). when i reboot blade server and go to PXE boot to kickstart i can see my PXE get ip address from DHCP but after that it stopped pinging that IP and kickstart failing saying no network connection (in short i can't ping that pxe IP).

I have all my VLAN/VNI configured for arp-suppression. if i remove arp-suppression for PXE VLAN/VNI then everything works, PXE successfully able to kickstart my all servers but as soon as i add arp-suppression it stopped working. Any idea how EVPN+VxLAN handle untagged VLANs?

Leaf NVE1 interface config

interface nve1
  no shutdown
  description ** VTEP/NVE Interface **
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10064
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10065
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10070
    suppress-arp     <---------- if i remove this it then PXE works. 
    mcast-group 239.1.1.1
  member vni 10100
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10555 associate-vrf

Bladecenter switch untagged VLAN 70

vlan 70
   name "pxe"
   untagged 1-16
   tagged Trk1
   no ip address
   exit

Update - 1

After debugging i found as soon as PXE boot load start loading CentOS linux at specific place where Linux set VLAN ID 0 on interface to make it native to allow access of all VLAN at that point i have noticed it flood my network with ARP broadcast on that specific VLAN and that cause packet loss on network and my installation stuck.

Look like this untagged frame getting in loop because of arp suppression not sure why. I do have other VLAN also and they are all tagged with VLAN which is working fine without any issue. I haven't seen any issue on them.

[root@pxe-server ~]# tcpdump -i bond0.70 -nn not port 22 and host 10.70.0.112
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0.70, link-type EN10MB (Ethernet), capture size 262144 bytes
13:13:23.115377 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.115539 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.115974 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.116110 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.215306 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.215312 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46
13:13:23.215823 ARP, Request who-has 10.70.0.112 (ff:ff:ff:ff:ff:ff) tell 0.0.0.0, length 46 

Only way to break this broadcast storm is to remove suppress-arp option from VLAN 70. I think it could be a big in Cisco software because not sure what is the relation of arp with non-tagged frame.

This is my Leaf-1-1 config and Leaf-1-2 is second vPC peer which has pretty similar config. Its very simple network not fancy stuff, I have only 2 vPC pair Leaf as per diagram.

Question: I have enabled suppress-arp on VNI which is pure L2VNI it doesn't have anycast-gateway because my Cisco ASA is gateway for all VLAN. (I am assuming suppress-arp does with L2VNI only without Distributed gateway)

leaf-1-1# show run

cfs eth distribute
nv overlay evpn
feature ospf
feature bgp
feature pim
feature fabric forwarding
feature interface-vlan
feature vn-segment-vlan-based
feature lacp
feature vpc
feature nv overlay

ip domain-lookup
system qos
  service-policy type network-qos jumboframes

fabric forwarding anycast-gateway-mac 0000.dead.beef
ip pim rp-address 10.255.0.123 group-list 239.0.0.0/8
ip pim ssm range 232.0.0.0/8
vlan 1,64-68,70,100,444,555
vlan 64
  name inside
vlan 65
  vn-segment 10065
vlan 66
  vn-segment 10066
vlan 67
  vn-segment 10067
vlan 68
  vn-segment 10068
vlan 70
  name pxe_boot
  vn-segment 10070
vlan 100
  name public
  vn-segment 10100
vlan 444
  name BACKUP_VLAN_ROUTING_VPC
vlan 555
  name L3VNI
  vn-segment 10555

spanning-tree port type edge bpduguard default
spanning-tree loopguard default
spanning-tree vlan 64-68,70,100,555 priority 8192
route-map DIRECT-PERMIT-ALL permit 10
  description ** Route-Map for BGP to redist route **
vrf context CUST1
  vni 10555
  rd auto
  address-family ipv4 unicast
    route-target both auto
    route-target both auto evpn
vrf context management
  ip route 0.0.0.0/0 172.30.0.1
hardware access-list tcam region vacl 0
hardware access-list tcam region arp-ether 256
vpc domain 1
  peer-switch
  role priority 10
  peer-keepalive destination 172.30.0.32 source 172.30.0.31
  delay restore 90
  peer-gateway
  delay restore interface-vlan 30
  ip arp synchronize


interface Vlan1
  no ip redirects
  no ipv6 redirects

interface Vlan100
  description ** Anycast Gateway For Public  **
  no shutdown
  mtu 9216
  vrf member CUST1
  ip address 60.25.124.1/23
  fabric forwarding mode anycast-gateway

interface Vlan444
  description ** Underlay Backup over vPC Peer-Link **
  no shutdown
  ip address 192.168.1.1/30
  ip ospf authentication-key 3 fa3ab8e90610229c
  ip ospf network point-to-point
  ip router ospf UNDERLAY-NET area 0.0.0.0
  ip pim sparse-mode

interface Vlan555
  description ** L3VNI-For-IRB **
  no shutdown
  mtu 9216
  vrf member CUST1
  ip forward

interface port-channel111
  description ** Link to enc-k001-1-a **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 111

interface port-channel112
  description ** Link to enc-k001-1-b **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 112

interface port-channel121
  description ** Link to enc-k001-2-a **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 121

interface port-channel122
  description ** Link to enc-k001-2-b **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 122

interface port-channel211
  description ** Link to enc-k002-1-a **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 211

interface port-channel212
  description ** Link to enc-k002-1-b **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 212

interface port-channel221
  description ** Link to enc-k002-2-a **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 221

interface port-channel222
  description ** Link to enc-k002-2-b **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 222

interface port-channel311
  description ** Link to enc-k003-1-a **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 311

interface port-channel312
  description ** Link to enc-k003-1-b **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 312

interface port-channel321
  description ** Link to enc-k003-2-a **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 321

interface port-channel322
  description ** Link to enc-k003-2-b **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  vpc 322

interface port-channel999
  description ** vPC Peer-Link **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100,444,555
  spanning-tree port type network
  speed 40000
  no negotiate auto
  vpc peer-link

interface nve1
  no shutdown
  description ** VTEP/NVE Interface **
  host-reachability protocol bgp
  source-interface loopback1
  member vni 10064
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10065
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10066
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10067
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10068
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10070
    mcast-group 239.1.1.1
  member vni 10100
    suppress-arp
    mcast-group 239.1.1.1
  member vni 10555 associate-vrf

interface Ethernet1/1
  description ** Link to swt-enc-k001-1-a Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 111 mode active

interface Ethernet1/2
  description ** Link to swt-enc-k001-1-a Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 111 mode active

interface Ethernet1/3
  description ** Link to swt-enc-k001-1-b Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 112 mode active

interface Ethernet1/4
  description ** Link to swt-enc-k001-1-b Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 112 mode active

interface Ethernet1/5
  description ** Link to swt-enc-k001-2-a Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 121 mode active

interface Ethernet1/6
  description ** Link to swt-enc-k001-2-a Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 121 mode active

interface Ethernet1/7
  description ** Link to swt-enc-k001-2-b Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 122 mode active

interface Ethernet1/8
  description ** Link to swt-enc-k001-2-b Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 122 mode active

interface Ethernet1/9
  shutdown
  mtu 9216

interface Ethernet1/10
  description ** Link to VMware Host-1 **
  switchport mode trunk
  switchport trunk allowed vlan 64-65,70,100
  spanning-tree port type edge trunk
  mtu 9216
  speed 10000

interface Ethernet1/17
  description ** Link to swt-enc-k002-1-a Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 211 mode active

interface Ethernet1/18
  description ** Link to swt-enc-k002-1-a Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 211 mode active

interface Ethernet1/19
  description ** Link to swt-enc-k002-1-b Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 212 mode active

interface Ethernet1/20
  description ** Link to swt-enc-k002-1-b Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 212 mode active

interface Ethernet1/21
  description ** Link to swt-enc-k002-2-a Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 221 mode active

interface Ethernet1/22
  description ** Link to swt-enc-k002-2-a Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 221 mode active

interface Ethernet1/23
  description ** Link to swt-enc-k002-2-b Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 222 mode active

interface Ethernet1/24
  description ** Link to swt-enc-k002-2-b Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 222 mode active

interface Ethernet1/33
  description ** Link to swt-enc-1-a Port E18 **k003
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 311 mode active

interface Ethernet1/34
  description ** Link to swt-enc-k003-1-a Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 311 mode active

interface Ethernet1/35
  description ** Link to swt-enc-k003-1-b Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 312 mode active

interface Ethernet1/36
  description ** Link to swt-enc-k003-1-b Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 312 mode active

interface Ethernet1/37
  description ** Link to swt-enc-2-a Port E18 **k003
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 321 mode active

interface Ethernet1/38
  description ** Link to swt-enc-k003-2-a Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 321 mode active

interface Ethernet1/39
  description ** Link to swt-enc-k003-2-b Port E18 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 322 mode active

interface Ethernet1/40
  description ** Link to swt-enc-k003-2-b Port E19 **
  no cdp enable
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100
  mtu 9216
  speed 10000
  channel-group 322 mode active

interface Ethernet2/1
  description ** Spine-1 **
  no switchport
  mtu 9216
  medium p2p
  ip address 10.1.11.1/31
  ip ospf authentication-key 3 XXXXXXX
  ip ospf network point-to-point
  ip router ospf UNDERLAY-NET area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet2/2
  description ** Spine-2 **
  no switchport
  mtu 9216
  medium p2p
  ip address 10.2.11.1/31
  ip ospf authentication-key 3 XXXXXXX
  ip ospf network point-to-point
  ip router ospf UNDERLAY-NET area 0.0.0.0
  ip pim sparse-mode
  no shutdown

interface Ethernet2/11
  description ** vPC Peer-Link **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100,444,555
  speed 40000
  no negotiate auto
  channel-group 999 mode active

interface Ethernet2/12
  description ** vPC Peer-Link **
  switchport mode trunk
  switchport trunk allowed vlan 64-68,70,100,444,555
  speed 40000
  no negotiate auto
  channel-group 999 mode active

interface mgmt0
  vrf member management
  no ip redirects
  ip address 172.30.0.31/23

interface loopback0
  description ** RID/BGP Overlay **
  ip address 10.255.1.11/32
  ip router ospf UNDERLAY-NET area 0.0.0.0
  ip pim sparse-mode

interface loopback1
  description ** VTEP/Overlay **
  ip address 10.255.255.11/32
  ip address 10.255.255.10/32 secondary
  ip ospf authentication-key 3 fa3ab8e90610229c
  ip router ospf UNDERLAY-NET area 0.0.0.0
  ip pim sparse-mode
cli alias name wr copy running-config startup-config
line console
line vty
boot nxos bootflash:/nxos.9.3.4.bin
router ospf UNDERLAY-NET
  router-id 10.255.1.11
  log-adjacency-changes
  area 0.0.0.0 authentication
router bgp 65001
  router-id 10.255.1.11
  log-neighbor-changes
  template peer VXLAN_SPINE
    remote-as 65001
    update-source loopback0
    address-family ipv4 unicast
    address-family l2vpn evpn
      send-community
      send-community extended
  neighbor 10.255.0.1
    inherit peer VXLAN_SPINE
    description ** iBGP Peer to Spine-1 **
  neighbor 10.255.0.2
    inherit peer VXLAN_SPINE
    description ** iBGP Peer to Spine-2 **
  vrf CUST1
    log-neighbor-changes
    address-family ipv4 unicast
      redistribute direct route-map DIRECT-PERMIT-ALL
evpn
  vni 10064 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10065 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10066 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10067 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10068 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10070 l2
    rd auto
    route-target import auto
    route-target export auto
  vni 10100 l2
    rd auto
    route-target import auto
    route-target export auto

Update-2:

After more testing i found culprit is DHCP packets, For testing purper i spun up DHCP server on other VLANs and noticed as soon as DHCP send/receive broadcast packet and at same time broadcast flood start. (If ARP suppression is enabled). I am going to create Anycast gateway to just prove this theory that arp-suppression doesn't support L2VNI. Please chime in if anyone has any history.

Notes: I am not using any kind of fancy DHCP-Relay feature.

7
  • I suspect a layer-3 problem but there's not enough information in your question. Is the PXE server connected to VLAN 70 directly? Have you tried ARPing/pinging the server from client interface? Does the VLAN show all required MAC addresses? What DHCP options do you use?
    – Zac67
    Nov 7 '20 at 9:28
  • Please share your BGP configuration as EVPN relies on that heavily. Are the peering sessions that carry EVPN address family up? It looks to me that might be the root cause of your problems. Nov 8 '20 at 12:27
  • Give me little time to post all my config. My all other VNI working fine except this untagged one. not sure what is the relation of arp with untagged but anyway let me post full config in few mins.
    – Satish
    Nov 9 '20 at 15:21
  • To maximize the chances someone will help I would suggest cutting down the repro to a minimum possible setup that still has that problem you are trying to solve Nov 10 '20 at 2:54
  • repro to minimum, i didn't get you what you trying to say.
    – Satish
    Nov 10 '20 at 2:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.