10

Alright - been battling this for at least 20 hours consecutive.. Sorry if this seems like a long rant, or a blog post, but I'm getting to the point of exhaustion.

So, here's the deal. We're using KEMP load balancers, who utilizes UCARP (a Linux clone of CARP, which is a VRRP clone) for HA heartbeat and persistent states. We want to utilize IGMP in our environment to prevent flooding across the datacenter.

We have two Dell PowerConnect 8124F switches running SW 5.1.1.7 acting as top-of-rack. These two are connected to a stacked pair of Cisco 3750-X, which is our core.

The problems began when we upgraded to the PowerConnect 5.1.x, where they apparently defaulted to leave IGMP snooping on unless you tell it otherwise. And behold - our load balancers went into split-brain, causing all sorts of warm fuzzy fun.

  • If I disable IGMP snooping on the VLAN where the load balancers do their multicast nothing happens, multicast is still dead
  • If I set up IP PIM on our core, the PowerConnect switches sees it on the same VLAN, but still no multicast traffic
  • If I enable flooding of all unregistered multicast traffic, it still does nothing at all.
  • If I disable IGMP snooping globally on the PowerConnect switches, all multicast traffic works. It works so great, that we get multicast traffic flooded to every single port that has the same VLAN tagged. Wonderful.

I noticed some strange MAC address entries on the VLAN at our core:

coresw#sh mac address-table vlan 367 | include 5e00
 367    0000.5e00.0101    DYNAMIC     Po13   seq_no:0

And I think.. Isn't that the multicast address? Why isn't this in the "sh mac address-table multicast"?

coresw#sh mac address-table multicast vlan 367
Vlan    Mac Address       Type        Ports
----    -----------       ----        -----
coresw#

And then I read this in the PowerConnect CLI guide:

Multicast traffic is traffic that is destined to a host group. Host groups are identified by the destination MAC address, i.e. the range 01:00:5e:00:00:00-01:00:5e:7f:ff:ff:ff for IPv4 multicast traffic or 33:33:xx:xx:xx:xx for IPv6 multicast traffic.

Seems like we are missing a "01" at the beginning of the MAC address, no? The dynamic MAC entry above start with "00". At this point I'm thinking about calling KEMP, and letting them know that their product is horribly misconfigured. But then I go read the RFC for VRRP - and behold:

The virtual router MAC address associated with a virtual router is an IEEE 802 MAC Address in the following format:

IPv4 case: 00-00-5E-00-01-{VRID} (in hex, in Internet-standard bit- order)

Alright - so switches doesn't normally pick up the multicast mac address range for VRRP. Fine, let's configure a static host group on the Dell switches. Nope.

Invalid input: Multicast MAC Address must be of format 01XX:XXXX:XXXX

OK then.. Next step, try to add a static mac entry:

osl-sys-swrack03(config)#mac address-table multicast ?

forbidden                forbid adding specific multicast addresses to
                         specific ports.

osl-sys-swrack03(config)#

So - no way to configure a static multicast MAC entry. If I try to do the same with a regular static MAC entry, I can only bind it to one port - this load balancing cluster runs across 4 different 10gig ports.

Update: There seems to be some confusion regarding MAC addresses. 172.30.1.0/24 is the front-facing loadbalancer network. 172.30.1.6 is the default shared VIP for the cluster, .7 is the management IP for the first load balancer and .8 is for the second load balancer. All other addresses (30, 40, 70, 80 etc) are all VIP with different services on them. When a failover occurs, all the VIP's change their MAC address to the second LB's physical MAC address. The multicast address in the bottom table does not change.

coresw#sh ip arp vlan 367
Protocol  Address          Age (min)  Hardware Addr   Type   Interface
Internet  172.30.1.6             78   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.40           204   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.80           167   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.70            38   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.66            12   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.35           185   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.60            97   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.30            80   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.61            33   0050.56b4.5004  ARPA   Vlan367    <- VIP - Loadbalancer1 physical MAC
Internet  172.30.1.7             27   0050.56b4.5004  ARPA   Vlan367    <- Management - Loadbalancer1 physical MAC
Internet  172.30.1.8             21   0050.56b4.08c2  ARPA   Vlan367    <- Management - Loadbalancer2 physical MAC

osl-sys-coresw#sh mac address-table dynamic vlan 367
          Mac Address Table
-------------------------------------------

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
 367    0000.5e00.0101    DYNAMIC     Po13   seq_no:0   <- multicast HA mac (UCARP)
 367    0050.56b4.08c2    DYNAMIC     Po13   seq_no:0   <- Loadbalancer1 physical MAC
 367    0050.56b4.5004    DYNAMIC     Po13   seq_no:0   <- Loadbalancer2 physical MAC

And that's the story. What on earth am I going to do with this?

Improve this question
8
  • What on earth am I going to do with this? <-- Tequila. Lots of it.
    – voretaq7
    Oct 31, 2013 at 21:44
  • You are confusing the multicast address used between the "virtual routers" (to tell who's there and who's the master), and the unicast address used for the virtual router itself (i.e. the MAC for the virtual IP.)
    – Ricky
    Oct 31, 2013 at 22:49
  • @RickyBeam Could you be a bit more specific? The reason why there (was) two mac addresses in the list above is because we have two pairs of loadbalancers, each with their own ID (the 01 and 02 at the end) - if that's what you're thinking of?
    – pauska
    Oct 31, 2013 at 22:51
  • 1
    No, you are still confused about the unicast MAC associated with the virtual router IP -- that's the MAC hosts use to talk to their load balanced service. A multicast address is what the load balancers used to know who's in charge. (read section 5.1.1)
    – Ricky
    Nov 1, 2013 at 1:10
  • @RickyBeam Sorry, it doesn't make any sense to me. The unicast MAC address for each load balancer (00:56, vmware) is totally different from the 0000.5e00.0101 I'm seeing when I disable IGMP snooping.
    – pauska
    Nov 1, 2013 at 1:43

2 Answers 2

Reset to default
5

I was able to resolve the issue. On the Kemp (with HA pair) you have the option of using a "Virtual MAC Address". If this box isn't checked, then the MAC of a load balancer VIP is that of the physical interface of the active Kemp unit. If this box is checked, then the MAC address of the VIP is a VRRP MAC. As you mentioned above the VRRP RFC states that the MAC being "00:00"{blah} with the last octet being the Router ID. The default Kemp HA [router] ID is 01. On my Powerconnects using Firmware 5.1.x.x I'm not using VRRP but I ran some traces and determined that the Powerconnect will drop a VRRP frame if the router ID is the same as itself. They do this EVEN if VRRP isn't configured and in that mode they default to 01. So changing the Kemp HA router ID to something like 22 (0x16) resulted in everything working.

Improve this answer
1
  • You are my hero. Thank you for finally figuring this out! Extremely poor wording of KEMP's part - they should give you a description that actually tells you that this translates to the VRRP router ID.
    – pauska
    Apr 18, 2016 at 18:00
2

The multicast address you're looking for is 224.0.0.18 [mac: 01005e.000012]. That is the control channel for all VRRP nodes. [edit] Unless KEMP changed the code, CARP (UCARP) originates traffic using the VRRP unicast MAC [00005e.0001xx] -- that's where a switch would naturally learn it.

If you don't have a querier on the network (presumably in every segment), then your switches will eventually forget what groups are where -- hosts don't send periodic memberships unless asked. [edit: Depending on configuration, a switch could drop unknown multicast, vs. flooding.] That can be a dedicated querier (it sends the packet, it doesn't care about any answers), or more commonly the multicast router within your infrastructure. In this simple case, a querier is all that's required as the VRRP messages are forbidden from crossing segments anyway.

I'm not familiar with Dell switches, so I don't know what cli commands you need.

[UPDATE]

Vlan    Mac Address       Type        Ports
----    -----------       --------    -----
367    0000.5e00.0101    DYNAMIC     Po13   seq_no:0   <- multicast HA mac (UCARP)

That is not the multicast mac. That's the unicast MAC source of the multicast traffic. The multicast mac will not show up in the mac address table. That's in a multicast group table. Cisco IOS has a show mac-address-table multicast (shows nothing on my router) and show ip igmp groups (shows 3 groups). That router is set to pim sparse-mode; nortel and cisco switches see it as a querier.

And the KEMP method is deeply flawed by using the host NIC MAC for virtual addresses. In your case, 5004 belongs to a nic. When 5004 disappears, everybody is still going to have "IP:6 == MAC:5004" in their tables; they will continue trying to talk to the dead host until that entry is replaced. KEMP is obviously gambling on gratuitous-arp being honored by everything in the network. HSRP, VRRP, and the OpenBSD designed CARP all use a virtual MAC for this very reason. (they appear to have failed to hack UCARP to use the nic mac instead of the VRRP virtual mac when transmitting it's multcast traffic.)

Given they hackery of UCARP, are you sure it's even using multicast?

Improve this answer
4
  • I've already set up a PIM router at our core on the same VLAN - shouldn't that eliminate the need of a querier?
    – pauska
    Nov 1, 2013 at 1:38
  • 1
    In theory, yes. Confirm the switch(es) are seeing a querier. (show ip igmp snooping querier vlan 367 for a cisco)
    – Ricky
    Nov 1, 2013 at 2:26
  • They don't see any querier, but they do see the mrouter (sh ip igmp snooping mrouter). Am I supposed to have a querier AND a mrouter? I thought the latter supersedes the first one..
    – pauska
    Nov 1, 2013 at 13:45
  • 1
    I don't know know Dell treats it. My cisco, hp, and adtran switches show the PIM mrouter as the querier, but they lack (or aren't configured for) PIM support themselves.
    – Ricky
    Nov 1, 2013 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.