I have following scenario with vPC and HSRP (version 1) configuration.
I am seeing very strange issue, my host configured for bond + vlan and my bonding mode is active-backup, I have only VLAN 100 configured for HSRP, I am seeing my host not able to ping HSRP virtual IP but it can ping all other host on VLAN 100, this issue just recently started everything was working fine few weeks ago.
vPC config
vpc domain 1
peer-switch
role priority 10
peer-keepalive destination 10.5.0.117 source 10.5.0.116
peer-gateway
auto-recovery
ip arp synchronize
HSRP config
interface Vlan100
description *** Public_1 VLAN ***
no shutdown
mtu 9216
no autostate
no ip redirects
ip address 74.xx.xx.2/23
no ip ospf passive-interface
ip router ospf 100 area 0.0.0.0
hsrp 1
preempt
priority 110
ip 74.xx.xx.1
I have two VLAN configured on HOST machine vlan 10 and vlan 100 following is my host interface output.
bond0.10: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.1.146 netmask 255.255.0.0 broadcast 10.10.255.255
inet6 fe80::6e3b:e5ff:feba:84e8 prefixlen 64 scopeid 0x20<link>
ether 6c:3b:e5:ba:84:e8 txqueuelen 1000 (Ethernet)
RX packets 18724100 bytes 861377042 (821.4 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 1579 bytes 160270 (156.5 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
bond0.100: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 74.xx.xx.179 netmask 255.255.254.0 broadcast 74.xx.xx.255
inet6 fe80::6e3b:e5ff:feba:84e8 prefixlen 64 scopeid 0x20<link>
ether 6c:3b:e5:ba:84:e8 txqueuelen 1000 (Ethernet)
RX packets 338156 bytes 15584262 (14.8 MiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 74 bytes 7230 (7.0 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
HOSR arp table
[root@host ~]# arp -n
Address HWtype HWaddress Flags Mask Iface
74.xx.xx.171 ether fc:15:b4:13:1e:40 C bond0.100
74.xx.xx.2 ether fc:5b:39:f7:6d:4f C bond0.100
74.xx.xx.170 ether d8:9d:67:75:2a:98 C bond0.100
74.xx.xx.1 ether 00:00:0c:07:ac:01 C bond0.100
74.xx.xx.177 ether 6c:3b:e5:b0:f9:f0 C bond0.100
On SW1 switch ARP and MAC tables
sw1# show ip arp 74.xx.xx.179
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies D - Static Adjacencies attached to down interface
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface
74.xx.xx.179 00:02:35 6c3b.e5ba.84e8 Vlan100
SW1 MAC tables
sw1# show mac address-table address 6c3b.e5ba.84e8
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
* 10 6c3b.e5ba.84e8 dynamic 0 F F Po46
FHRP
sw1# show ip arp fhrp-non-active-learn
Flags: D - Static Adjacencies attached to down interface
IP ARP Table for context default
Address Age MAC Address Interface
SW2
sw2# show ip arp 74.xx.xx.179
Flags: * - Adjacencies learnt on non-active FHRP router
+ - Adjacencies synced via CFSoE
# - Adjacencies Throttled for Glean
CP - Added via L2RIB, Control plane Adjacencies D - Static Adjacencies attached to down interface
IP ARP Table
Total number of entries: 1
Address Age MAC Address Interface
74.xx.xx.179 00:05:04 6c3b.e5ba.84e8 Vlan100 *
MAC table
sw2# show mac address-table address 6c3b.e5ba.84e8
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link,
(T) - True, (F) - False, C - ControlPlane MAC
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+------------------
+ 10 6c3b.e5ba.84e8 dynamic 0 F F Po46
FHRP
sw2# show ip arp fhrp-non-active-learn
Flags: D - Static Adjacencies attached to down interface
IP ARP Table for context default
Address Age MAC Address Interface
74.xx.xx.179 00:07:32 6c3b.e5ba.84e8 Vlan100
UPDATE - 1
We are seeing lots of following logs in show logging
2018 Jul 5 14:15:05 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul 5 14:15:06 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul 5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul 5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul 5 14:17:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul 5 14:17:07 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul 5 14:19:06 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul 5 14:19:07 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
2018 Jul 5 14:19:07 swt1 %-SLOT1-5-BCM_L2_LEARN_DISABLE: MAC Learning Disabled unit=0
2018 Jul 5 14:19:08 swt1 %-SLOT1-5-BCM_L2_LEARN_ENABLE: MAC Learning Enabled unit=0
UPDATE -2
Jul 5 14:31:13 10.5.0.116 : 2018 Jul 5 18:24:35 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul 5 14:31:17 10.5.0.116 : 2018 Jul 5 18:24:39 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul 5 14:31:25 10.5.0.116 : 2018 Jul 5 18:24:47 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul 5 14:31:29 10.5.0.116 : 2018 Jul 5 18:24:51 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul 5 14:31:33 10.5.0.116 : 2018 Jul 5 18:24:54 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 6c3b.e5b0.c998 in vlan 100 has moved between Po35 to Po36
Jul 5 14:31:35 10.5.0.116 : 2018 Jul 5 18:24:56 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul 5 14:31:35 10.5.0.116 : 2018 Jul 5 18:24:56 UTC: %L2FM-3-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 100 for 120s due to too many mac moves
Jul 5 14:31:35 10.5.0.116 : 2018 Jul 5 18:24:56 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac 1458.d05a.f6d8 in vlan 100 has moved between Po43 to Po44
Jul 5 14:33:35 10.5.0.116 : 2018 Jul 5 18:26:57 UTC: %L2FM-3-L2FM_MAC_FLAP_RE_ENABLE_LEARN: Re-enabling learning in vlan 100
Jul 5 14:34:44 10.5.0.116 : 2018 Jul 5 18:28:06 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul 5 14:34:44 10.5.0.116 : 2018 Jul 5 18:28:06 UTC: %L2FM-3-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 100 for 120s due to too many mac moves
Jul 5 14:34:44 10.5.0.116 : 2018 Jul 5 18:28:06 UTC: %L2FM-4-L2FM_MAC_MOVE2: Mac fc15.b41f.59e0 in vlan 100 has moved between Po43 to Po44
Jul 5 14:36:43 10.5.0.116 : 2018 Jul 5 18:30:05 UTC: %L2FM-3-L2FM_MAC_FLAP_RE_ENABLE_LEARN: Re-enabling learning in vlan 100
Question:
why i am not able to ping HSRP VIP 74.xx.xx.1 ip address from host even on host arp table i can see its MAC address?
why in MAC table i can't see VLAN 100 mac address ( But i can see vlan 10 MAC)
what is
fhrp-non-active-learnand why its only showing up on SW2 (standby HSRP instance)
bondconfiguration and they are doinground-robinmode so sending traffic on both switch.. my vPC aggr switch seeing MAC are flapping, why its seeing mac are flapping?round-robinmight not be the best choice, here. RR will cause the host to send out its ethernet frames one by one, rotating through all NICs of the bond. In turn, the switching platform will see (and learn and re-learn) the source MAC address on all the switchports the host is connected to. Very soon, the switches will start to complain about MAC address flapping. I suggest to pick a bonding mode which makes sure that one given MAC address on the host (or one of its VMs) "sticky" to one of the bond members. Or see if the given setup allows for 802.3ad/LACP between switches and host.