3

I have following scenario with vPC and HSRP (version 1) configuration.

enter image description here

I am seeing very strange issue, my host configured for bond + vlan and my bonding mode is active-backup, I have only VLAN 100 configured for HSRP, I am seeing my host not able to ping HSRP virtual IP but it can ping all other host on VLAN 100, this issue just recently started everything was working fine few weeks ago.

vPC config

vpc domain 1
  peer-switch
  role priority 10
  peer-keepalive destination 10.5.0.117 source 10.5.0.116
  peer-gateway
  auto-recovery
  ip arp synchronize

HSRP config

interface Vlan100
  description *** Public_1 VLAN ***
  no shutdown
  mtu 9216
  no autostate
  no ip redirects
  ip address 74.xx.xx.2/23
  no ip ospf passive-interface
  ip router ospf 100 area 0.0.0.0
  hsrp 1
    preempt
    priority 110
    ip 74.xx.xx.1

I have two VLAN configured on HOST machine vlan 10 and vlan 100 following is my host interface output.

bond0.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.10.1.146  netmask 255.255.0.0  broadcast 10.10.255.255
        inet6 fe80::6e3b:e5ff:feba:84e8  prefixlen 64  scopeid 0x20<link>
        ether 6c:3b:e5:ba:84:e8  txqueuelen 1000  (Ethernet)
        RX packets 18724100  bytes 861377042 (821.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1579  bytes 160270 (156.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

bond0.100: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 74.xx.xx.179  netmask 255.255.254.0  broadcast 74.xx.xx.255
        inet6 fe80::6e3b:e5ff:feba:84e8  prefixlen 64  scopeid 0x20<link>
        ether 6c:3b:e5:ba:84:e8  txqueuelen 1000  (Ethernet)
        RX packets 338156  bytes 15584262 (14.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 74  bytes 7230 (7.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

HOSR arp table

[root@host ~]# arp -n
Address                  HWtype  HWaddress           Flags Mask            Iface
74.xx.xx.171           ether   fc:15:b4:13:1e:40   C                     bond0.100
74.xx.xx.2             ether   fc:5b:39:f7:6d:4f   C                     bond0.100
74.xx.xx.170           ether   d8:9d:67:75:2a:98   C                     bond0.100
74.xx.xx.1             ether   00:00:0c:07:ac:01   C                     bond0.100
74.xx.xx.177           ether   6c:3b:e5:b0:f9:f0   C                     bond0.100

On SW1 switch ARP and MAC tables

sw1# show ip arp 74.xx.xx.179

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies       D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 1
Address         Age       MAC Address     Interface
74.xx.xx.179  00:02:35  6c3b.e5ba.84e8  Vlan100

SW1 MAC tables

sw1# show mac address-table address 6c3b.e5ba.84e8
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
*   10     6c3b.e5ba.84e8   dynamic  0         F      F    Po46

FHRP

sw1# show ip arp fhrp-non-active-learn

Flags: D - Static Adjacencies attached to down interface

IP ARP Table for context default
Address         Age       MAC Address     Interface

SW2

sw2# show ip arp 74.xx.xx.179

Flags: * - Adjacencies learnt on non-active FHRP router
       + - Adjacencies synced via CFSoE
       # - Adjacencies Throttled for Glean
       CP - Added via L2RIB, Control plane Adjacencies       D - Static Adjacencies attached to down interface

IP ARP Table
Total number of entries: 1
Address         Age       MAC Address     Interface
74.xx.xx.179  00:05:04  6c3b.e5ba.84e8  Vlan100          *

MAC table

sw2# show mac address-table address 6c3b.e5ba.84e8
Legend:
        * - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
        age - seconds since last seen,+ - primary entry using vPC Peer-Link,
        (T) - True, (F) - False, C - ControlPlane MAC
   VLAN     MAC Address      Type      age     Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+   10     6c3b.e5ba.84e8   dynamic  0         F      F    Po46

FHRP

sw2# show ip arp fhrp-non-active-learn

Flags: D - Static Adjacencies attached to down interface

IP ARP Table for context default
Address         Age       MAC Address     Interface
74.xx.xx.179  00:07:32  6c3b.e5ba.84e8  Vlan100

UPDATE - 1

We are seeing lots of following logs in show logging

2018 Jul  5 14:15:05 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:15:06 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:17:07 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:19:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:19:07 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul  5 14:19:07 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul  5 14:19:08 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0

UPDATE -2

Jul  5 14:31:13 10.5.0.116 : 2018 Jul  5 18:24:35 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:17 10.5.0.116 : 2018 Jul  5 18:24:39 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:25 10.5.0.116 : 2018 Jul  5 18:24:47 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:29 10.5.0.116 : 2018 Jul  5 18:24:51 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:33 10.5.0.116 : 2018 Jul  5 18:24:54 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 6c3b.e5b0.c998 in vlan 100 has moved between Po35 to Po36
Jul  5 14:31:35 10.5.0.116 : 2018 Jul  5 18:24:56 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:31:35 10.5.0.116 : 2018 Jul  5 18:24:56 UTC: %L2FM-3-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 100 for 120s due to too many mac moves
Jul  5 14:31:35 10.5.0.116 : 2018 Jul  5 18:24:56 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul  5 14:33:35 10.5.0.116 : 2018 Jul  5 18:26:57 UTC: %L2FM-3-L2FM_MAC_FLAP_RE_ENABLE_LEARN: Re-enabling learning in vlan 100
Jul  5 14:34:44 10.5.0.116 : 2018 Jul  5 18:28:06 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul  5 14:34:44 10.5.0.116 : 2018 Jul  5 18:28:06 UTC: %L2FM-3-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 100 for 120s due to too many mac moves
Jul  5 14:34:44 10.5.0.116 : 2018 Jul  5 18:28:06 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul  5 14:36:43 10.5.0.116 : 2018 Jul  5 18:30:05 UTC: %L2FM-3-L2FM_MAC_FLAP_RE_ENABLE_LEARN: Re-enabling learning in vlan 100

Question:

  1. why i am not able to ping HSRP VIP 74.xx.xx.1 ip address from host even on host arp table i can see its MAC address?

  2. why in MAC table i can't see VLAN 100 mac address ( But i can see vlan 10 MAC)

  3. what is fhrp-non-active-learn and why its only showing up on SW2 (standby HSRP instance)

  • Could it be this bug? CSCvb28186 - N9K: Mac Learning disable due to broadcast storm control – Ron Trunk Jul 5 '18 at 14:41
  • @RonTrunk check my UPDATE -2 i have increase logging and i am seeing something going on related MAC flooding, I have all my server in bond configuration and they are doing round-robin mode so sending traffic on both switch.. my vPC aggr switch seeing MAC are flapping, why its seeing mac are flapping? – Satish Jul 5 '18 at 18:40
  • 2
    round-robin might not be the best choice, here. RR will cause the host to send out its ethernet frames one by one, rotating through all NICs of the bond. In turn, the switching platform will see (and learn and re-learn) the source MAC address on all the switchports the host is connected to. Very soon, the switches will start to complain about MAC address flapping. I suggest to pick a bonding mode which makes sure that one given MAC address on the host (or one of its VMs) "sticky" to one of the bond members. Or see if the given setup allows for 802.3ad/LACP between switches and host. – Marc 'netztier' Luethi Jul 6 '18 at 8:18
  • What kind of switches are these and what software version are they running? Are those TOR switches actually FEXes? What happens if you move the port channels to the parent switches instead of the FEXs (to eliminate enhanced vPC)? It really looks like there's some inconsistency between cabling and port-channel configuration somewhere. – user3629081 Jul 18 '18 at 16:39
2

Resolution:

Problem was host were running on round-robin bonding configuration which cause lots of MAC flapping and that cause lockup MAC table and preventing new MAC entires.

We re-configured all Server to use active-backup to fix this issue.

Notes: Above diagram was not correct (we thought its active-backup but it was round-robin)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.