18

The combination of ECMP (or other causes of asymmetric paths) and HSRP is broken by default in Cisco IOS; the default behaviour with this design floods unicast traffic excessively.

What is the best-practice for using HSRP with ECMP to prevent unknown unicast flooding?

Details / Background

We have a HSRP topology similar to the first diagram below for many of our facilities. Our Cisco WAN routers have equal-cost routes to all other sites; thus we can see asymmetric routing effects all the time. Normally we assign R1 to be the HSRP primary, but ECMP allow return traffic through either R1 or R2.

The issue is that when PC1 mounts a remote iSCSI drive across the WAN, the traffic leaves the site via R1, but could return via R2. As long as the iSCSI traffic returns via R1, there are no issues.

HSRP_Broken_00

The problem occurs when PC1's traffic returns via R2. Assume the iSCSI session starts at 8:00:00, and both routers and both switches learn PC1's mac simultaneously. Between 8:00:00 and 8:00:05, there are no flooding problems because both switches still have PC1's mac-address in their CAM table.

HSRP_Broken_01

Five minutes after the iSCSI session starts, S2's CAM entry for PC1's mac expires out of the CAM table and S2 floods PC1's traffic out all ports (in this case to Po1, Gi0/3 and Gi0/4). If PC1's iSCSI session consumes a lot of bandwidth, this unknown unicast flooding can suck non-trivial capacity from the links to PC3 and PC4.

Cisco IOS switches have a default CAM timer of 300 seconds...

S2# show mac address-table aging-time
Vlan Aging Time
---- ----------
1    300
17   300

However, Cisco IOS' default interface ARP timer is 4 hours...

R2# show interface gi0/0
GigabitEthernet0/0 is up, line protocol is up 
  Hardware is AmdP2, address is 000a.dead.beef (bia 000a.dead.beef)
  Internet address is 172.17.1.252/24
  MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, 
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  ARP type: ARPA, ARP Timeout 04:00:00       <--------------

Therefore, S2 starts flooding PC1's iSCSI traffic after five minutes.

HSRP_Broken_02

8
  • Why do people keep posting questions and then answering them, themselves? No as in, the searched and found the answer, they had it already? This is a Q&A site, not a blog (not that you haven't provided a good write up!)
    – jwbensley
    Jul 3, 2013 at 13:25
  • 8
    @javano: self-answering is explicitly encouraged by SE. ref meta.networkengineering.stackexchange.com/questions/4/… Jul 3, 2013 at 13:30
  • 1
    @CraigConstantine Yes I know, but I'm sure people post questions and then answer strait after, not some period of time later when they have figured out the answer to the question (even if it's only 5 minutes later), they answer strait away because they already knew the answer before posting the question. I find this a bit strange.
    – jwbensley
    Jul 3, 2013 at 13:40
  • 6
    Yet the fact remains, that writing a Q and an immediate answer is explicitly encouraged. Jul 3, 2013 at 13:42
  • 4
    @javano, If you solve a problem that you think other people will face, SE wants the search engine traffic for the resolution of that problem... they don't care whether I post the answer at the same time or not... in fact, there is a small checkbox at the bottom of the question web form to "Answer your own question – share your knowledge, Q&A-style" Jul 3, 2013 at 13:45

7 Answers 7

13

The simple answer is to make the CAM timer equal or slightly longer than the corresponding interface ARP timer, but there are at least three different options to select from...

Option 1: Lower all interface ARP Timers

This option works best if you have a decent-sized layer2 switched network, a reasonable number of ARP entries and few routed interfaces. This method also is preferable if you like to see PC mac entries age out of the topology quickly.

  • On all IOS ethernet interfaces facing an ethernet switch: arp timeout 240
  • On all IOS ethernet interfaces facing an ethernet switch: hold-queue 200 in and hold-queue 200 out to avoid dropping ARP packets during periodic ARP-refreshes (these limits could be higher, or lower depending on how many ARP refreshes you think that you'll need to handle at once). If you are adjusting Selective Packet Discard values, then you should follow the guidelines in the paper I linked.

This forces Cisco IOS to refresh the ARP table within four minutes, if it hasn't happened otherwise for a given ARP entry. The obvious disadvantage is that this doesn't scale well if you have lots of ARP entries... the limits vary by platform. I have used this with a few hundred ARPs per router on Catalyst 4500 / 6500 (the Layer3 SVIs) without any issues.

Option 2: Increase the switch CAM Timers

This option works best if you have a large number of ARP entries (i.e. thousands, such as an intense VMWare environment could see).

  • On all IOS switches: mac address-table aging-time 14400, or mac address-table aging-time 14400 vlan <vlan-id> for any Vlan that is of concern.

This change adjusts timers that most people assume are fixed at 300 seconds (on Cisco IOS), so be sure to include this in continuity docs. The side-effect of this is that CAM table entries linger for 4 hours after the PC is removed (which can be either good or bad, depending on your PoV). If 4 hours is too long, see the next option...

Option 3: Change both the interface ARP timers, and the switch CAM Timers

This option avoids hideously-long CAM timers in Option 2 at the expense of more configuration. You can choose whether you need 900 seconds, 1800 seconds, or whatever... just make sure your CAM and ARP timers match; thus, you will need to configure both Option 1 and Option 2 in your topologies.

5
  • 4
    We sorted this problem choosing the first proposed solution, but we weren't sure of the order in which IOS would clean the table, then we set ARP timeout to 293s ( the closest prime number below the mac-address table timeout ). Still do not know if this was a good choice or not Jul 3, 2013 at 12:09
  • 1
    Technically Cisco IOS fires the ARP walker on 60 second intervals so you should use 240... I neglected to include that in my answer... editing it in... i am curious why you picked a prime number... Jul 3, 2013 at 12:36
  • ACK. ARP timeout less or equal to MAC timeout should be BCP. There doesn't even need to be HSRP, just if there are two routers it can bite you and cause even loops.
    – ytti
    Jul 3, 2013 at 12:39
  • Didn't know. So our trick is totally useless. We picked a prime number to minimize the overlap of the timers. Jul 3, 2013 at 15:03
  • 4
    @MikePennington, thank you. Anyway you're right ARP timeout resolution is implemented in minutes Jul 3, 2013 at 15:44
1

To me, ECMP is the real issue here - so in addition to the above steps to limit unknown unicast flooding, you can also tune the route metrics toward the WAN so that R1 is preferred over R2 for the return traffic. One way to achieve this is via distribute list on R2 as follows: (EIGRP used for example only, the same can be achieved with OSPF or BGP with other commands)

!
ip prefix-list R1-PREFER seq 5 permit 172.17.1.0/24
!
route-map R1-PREFER-MAP permit 10
 match ip address prefix-list R1-PREFER
 set metric 1 1 1 1 1
... (allow all other routes)
!
router eigrp 1
 ....
 distribute-list route-map R1-PREFER-MAP out Ser1/0
 ....
!

This will result in the WAN forwarding all traffic for 172.17.1.0 to R1. If R1 Se1/0 fails, the route will be installed toward R2. You can further tune these metrics so the backup route to R2 is actually a feasible successor for faster failover. HSRP and tracking will take care of the egress traffic.

3
  • essentially youre answering the question that you want, instead of my question... which requires both fhrp and ecmp Jul 3, 2013 at 16:16
  • sorry about that - I'm getting used to this forum and missed that requirement!
    – smoothbSE
    Jul 3, 2013 at 16:53
  • No problem... welcome to NE :) Jul 3, 2013 at 17:04
0

The idea if not use ECMP if HSRP is in use may be ok for SERVERS where ingress traffic may be higher than egress traffic, in a PC situation IN GENERAL ingress traffic from the WAN (responses) is higher than egress traffic (ingress). We like most people just set the ARP timers. you can mess with CAM timers BUT if you have say an MDF with the layer 3 switch and an IDF with 2 collection switches and say 5 access switches, it is a LOT easer to configure on the L3 SVI than doing all access switches.

0

One could use a stack of switches to mitigate this issue of expiring MAC address entry in the second switch.

0

Ah, I remember this one. Weeks of fun was had dealing with this back a few jobs ago. One wrinkle is that STP events will put the vlans in fast-aging mode, so setting the MAC timer longer than the ARP timer doesn't help

I solved the issue by forcing ECMP back from the servers, by creating two floating HSRP gateways, with one primary on each router. We then configured both gateways on each host. By forcing host traffic to both R1 and R2 in this fashion, we would be sure that R2 would never age out the MAC addresses.

Ideally, this wouldn't be an issue if L2/3 switches purged ARP entries associated with aged-out MAC addresses. The next packet to the IP would then result in a new ARP request, populating both the ARP cache and the MAC table. I think Cisco eventually implemented this, but I can't say for sure.

0

Summary: MC-LAG or HSRP GARP

I've never been a fan of tweaking timers. Timers are set a certain way usually for many reasons. Altering them:

  • is potentially operationally intensive to maintain everywhere the same
  • makes things more complicated and difficult to troubleshoot
  • as a recent commentor showed, can have unexpected side effects
  • may not "play nice" with future Cisco enhancements

Alternately:

  1. Use MC-LAG (aka "MEC" in Cisco documentation). This is your best option, though you should understand the deployment scenarios where MC-LAG can be used (its not a universal solution, and should only be deployed after appropriate research and testing). MC-LAG variants are hardware dependent. Examples are:

    a. Stacking (Cat 3k)

    b. VSS (Cat4k/6k)

    c. VPC (Nexus)

    d. Pseudo mLACP (ASR1k)

    e. MC-LAG (ASR9k)

    f. Clustering (ASA)

  2. Enable HSRP to periodically send Gratuitous ARP packets. Granted, this is similar to altering timers, but it's a much more graceful alteration than manipulating the CAM table and ARP timers. (Note though that this depends on your hardware and software combination, not all HSRP implementations offer this.)

    By default, HSRP sends 3 GARPs, at 0, 2, and 4 seconds after the router becomes the forwarding gateway. However, there is a configuration parameter that allows you to choose the number of GARPs (including "infinite") and the interval.

I use MC-LAG pretty extensively, particularly VSS, VPC, and Clustering (I'm not a fan of stacking).

Where I can't use MC-LAG or GLBP, this is what I apply to my campus L2/L3 boundary routers (I have a 350-building campus so I use Cat6k pretty heavily):

Cat6k-v15(config)#interface vlan 100
Cat6k-v15(config-if)#standby arp ?
  gratuitous  Setup gratuitous ARP interval and count

Cat6k-v15(config-if)#standby arp gratuitous ?
  count     Set HSRP gratuitous ARP count
  interval  Set HSRP gratuitous ARP interval
  <cr>

Cat6k-v15(config-if)#standby arp gratuitous count ?
  <0-60>  Number of gratuitous ARPs to send after group is activated (0 for continuous)

Cat6k-v15(config-if)#standby arp gratuitous count 0 ?
  count     Set HSRP gratuitous ARP count
  interval  Set HSRP gratuitous ARP interval
  <cr>

Cat6k-v15(config-if)#standby arp gratuitous count 0 interval ?
  <3-1800>  Gratuitous ARP Interval (sec)

Cat6k-v15(config-if)#standby arp gratuitous count 0 interval 60 ?
  count     Set HSRP gratuitous ARP count
  interval  Set HSRP gratuitous ARP interval
  <cr>

Cat6k-v15(config-if)#standby arp gratuitous count 0 interval 60 

(I would post references to all these, but I don't have a high-enough "reputation" on this site to post more than two URLs.)

7
  • What you are calling MC-LAG is certainly an option, but its availability on classic IOS platforms is spotty. I am also missing how HSRP Gratuitous ARP helps solve this problem. Using the example in my question, could you elaborate on how HSRP Gratuitous ARP solves the ARP entry timeout of 172.17.1.1? Note that the default GW is 172.17.1.254 Jun 29, 2017 at 10:15
  • Long answer, so let me break this into two parts. Part 1... The problem with HSRP is that the router responds to an ARP query with the virtual MAC. However, when the router forwards a datagram to the host, it uses the physical MAC address. Switches clear their forwarding table fairly quickly (often 300sec, or 5min), but ARP entries often stick around far longer than that (8hr is common). Oct 9, 2017 at 5:00
  • Part 2... After the switches times-out the virtual MAC address from the forwarding table, the traffic from the server to the virtual MAC becomes "unknown unicast", where the switch defaults to hub-like behavior and floods the traffic out all ports. By periodically sending a GARP, the router refreshes the switch forwarding table. Additionally, by sending out a GARP, the ARP table on the server is refreshed, eliminating it's need to send an ARP query. Oct 9, 2017 at 5:01
  • Secondary to my 2-part answer, I just realized the question is asking from the opposite direction: the server's MAC address is getting flushed from the switches, not the router's virtual MAC. We had this specific problem and ended up solving it initially through MC-LAG (specifically VPC), and later since we were already using Nexus we switched to FabricPath aka TRILL, which made the problem go away. But, both of those are hardware- and topology-dependent. Oct 9, 2017 at 5:14
  • I just realized my original comment is valid - but woefully incomplete. Not just MC-LAG, but MC-LAG at both layers. Then you're dealing with a shared CAM table at both the the switch level and the router level. Dec 16, 2019 at 19:08
0

I just realized my original comment is valid - but woefully incomplete. Vendor-neutral design recommendation is to build in triangles, not rectangles. So:

  1. Not just MC-LAG, but MC-LAG at both layers. Then you're dealing with a shared CAM table at both the the switch level and the router level.

  2. If you can't do that, MC-LAG either the router or switch, and MC-LAG to the other layer with additional link (i.e. full-mesh between routers and switches). STP will ensure loop-free topology.

  3. If you can't do that, still full-mesh the routers and switches. STP will ensure loop-free topology, and the switch CAM tables will still know all the appropriate MAC forwarding rules. The server will always send it's MAC, and if you configure the HSRP GARPs on 1-min intervals the switches will also not forget the HSRP vMAC.

Preferred options are in that order. But at the very least, install that extra pair of links.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.