2

We have a host which has a bonded Ethernet interface. only one is active, at each time, when the other fails, the MAC address moves to the other interface and traffic continues...

But at the SP level, what is the mechanism which causes this? I would have thought the access ports on the switch were blocked on one side, or would it be the case that STP hasn't blocked it, as it didn't see a loop to begin with?

I understand a layer 1 failure will cause convergence, but we have had a situation when you can electronically fail an Ethernet port, so the backup route is used, and it works fine.

co

3

As long as the host does not bridge traffic, there is no loop. portfast and bpduguard (Cisco speak, other vendors may have other terminology) are settings usually applied to host ports.

Continued traffic flow is ensured by probably the most fundamental function of a switch: MAC address learning.

Once the host fails over to the (former) standby interface, the ethernet frames it sends out automatically trigger the switches to update their CAM tables, upon arrival of the frame at the given switch port.

Since it is not ensured that all of the host's (unicast) frames reach all switches, there might be some slight delay until all switches have updated their tables. The first broadcast or multicast frame from that host or VM will however be seen by all switches, and since broadcasts do happen every so often, you might never actually notice a delay. However, there can be cases where an admin may run into the "old mac-address-table problem" - just as Darrel Root pointed out.

That's why some hypervisor hosts have a feature called "notify switches" (for example VMware ESXi and its vSwitches). If they reassign a VM to another NIC, e.g. in case of a NIC failover or failback, they will send out a burst of RARP broadcast messages, through the given VM's new uplink, containing each applicable VM's MAC address as source MAC address. All these RARPs are broadcast, so they travel through the entire network (or VLAN), and all switches can update their CAM tables accordingly. ESXi sends these broadcast bursts instantly once it sees "line protocol up", so having portfastor portfast trunk on an ESXi's switch port is essential. Without portfast [trunk], the 30sec timeout before the switchport port goes into FWD mode will kill all attempts to "notifiy switches". This 30sec delay even exists on non-portfast ports of Rapid-Spanning-Tree environments.

| improve this answer | |
  • Thanks for the info, as mentioned below we are having issues with old table problem and the MAC address timing out, even when there is continuous active traffic.... any ideas on where to look to fix this? – user10021657 Jul 19 '19 at 10:50
  • @user10021657 While in the "old table problem" situation, look at the CAM tables (a.k.a. mac address table) of all switches involved. Would the the given entry on one/any of the switches happen to be "sticky/static" or "dynamic"? The former might be a hint at a port security or 802.1x configuration being in place; under certain circumstances, these can keep a switch from (re)learning a MAC address on another port. – Marc 'netztier' Luethi Jul 19 '19 at 14:02
  • Just looking through the config now, and there is no mention of sticky/static or dynamic. Testing again tomorrow so I'll see if I can get the problem to happen again. – user10021657 Jul 22 '19 at 13:47
0

End hosts generally don’t speak the spanning tree protocol and generally don’t forward BPDU’s*, so both ports on the switches will be in forwarding state. You’ll probably have those ports configured as host ports (portfast) so rapid spanning tree functions properly.

You may suffer an old Mac-address-table problem on the switches after failover in one direction, so make sure to test failover in both directions.

  • I have responded to a few network outages caused by someone configuring an end-host to bridge. That could result in BPDUs going through the host.

EDIT REGARDING MAC-ADDRESS-TABLE FLOODING:

If you are suffering mac-address-table flooding on one switch under normal operation (as opposed to immediately after failover), that's a well understood problem with a full explanation already at Best practice for the combination of HSRP and ECMP

Here's the quick summary:

Naming your devices:

router-left
router-right  (assume this is HSRP primary)
switch-left
switch-right
host-left   (lets assume this one is "standby" right now)
host-right  ("active" right now)
  1. Traffic inbound from other networks is load balanced by your routing protocol (OSPF, IS-IS, whatever) between router-left and router-right.
  2. router-left and router-right both ARP for host-right.
  3. (edit: deleted)
  4. host-right responds to both ARP requests
  5. ARP response to router-right populates router-right's ARP table. It also passes through switch-right populating its mac-address-table
  6. ARP response to router-left populates router-left's ARP table. Is also passes through both switches populating both their mac-address-tables.
  7. Outbound traffic from host-right goes to router-right (HSRP primary) and passes through switch-right, continuing to refresh the timer in its mac-address-table. But switch-left does not see this traffic!
  8. Cisco default "arp timeout" varies by platform but I see documentation varying from 1500 seconds to 14,400 seconds.
  9. Cisco mac-address-table switch timeout is 300 seconds.
  10. 300 seconds after the initial ARP from router-left, mac-address-table timeout on switch left expires. Router-lefts ARP entry does not expire. So switch-left starts flooding inbound network traffic destined for host-right.

The solution I've used in the past is to reduce the ARP timeout to 290 seconds or increase the mac-address-table timeout to 14,500. Since this is a HA-failover environment, I'd reduce the ARP timeout in this case, even though that is an (inconvenient) interface configuration command.

| improve this answer | |
  • That's what I am seeing. Should the MAC address table timeout, even if I have active traffic on it? – user10021657 Jul 19 '19 at 10:49
  • @user10021657 no. As long as a switch sees incoming frames on a port with the given source MAC address, it will keep the CAM table entry, mapping this MAC to this port. After the last frame with this source MAC arrives at this port, the CAM table aging timer starts to decrement. As long as this entry exists, traffic towards that MAC address will be forwarded out of this port. – Marc 'netztier' Luethi Jul 19 '19 at 13:52
  • Once the CAM table aging timer has expired, the entry is flushed from the CAM table. If there are still frames coming into the switch (from elsewhere), destined to that MAC address, the switch will resort to unknown unicast flooding, and will forward these frames out of all ports (... of that VLAN), essentially "hoping" that it will eventually see a response frame incoming on one of its (other) ports, which would allow to update the CAM table. – Marc 'netztier' Luethi Jul 19 '19 at 13:55
  • So this is what's happening on it, even when there is valid traffic present, the MAC address disappears in the MAC address table after the 300 second timer... obviously the switch doesn't know where to send traffic then, so floods the switch...... This is a switching HWIC module in a cisco router... I'm starting to lean to a possible hardware issue or bug?? – user10021657 Jul 22 '19 at 13:51
  • 1
    Did you adjust the ARP timer on both router interfaces to be less than the Mac-address-timer on both switches? – Darrell Root Jul 22 '19 at 13:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.