1

I'm trying to do up a wiring diagram for a VMWare cluster in a remote data center. I typically will check the MAC addresses in VMWare VSphere client (vmnic[x]) and/or use esxcli to do so from SSH on the ESXi hosts.

I'll then log into the switch and look at the MAC table to see what port a MAC address was learned from and go from there.

Having an odd issue with Huawei CloudEngine 6800 series switches where the MAC address that's reported in the MAC table (display mac-addresses | include xxxx-xxxx-xxxx) is off on the last bit.

For example, vmnic0 is one port on a Broadcom QLogic 57800 2x10GbE + 2x1GbE quad adapter, which reports as xx:xx:xx:xx:xx:b5. Filtering on xxxx-xxxx-xxb5 produced zero results in the filtering options. However, there's an entry for xxxx-xxxx-xxb6.

I checked a few more NICs and it appears that it's just the Broadcom QLogic 57800 (and 57810) ones are the only ones with inconsistencies, both 10GbE and 1GbE: it's as if the MACs are all off by a bit (c0 shows up as c1 on the switch, c2 as c3 and so on).

Now I'm a bit rusty, but I know that converged interfaces can be "sliced up" into sub-interfaces, which would have different MAC addresses, but that would also be represented higher up in the "stack" in ESXi, so presumably the source MAC is being parsed out of the frame correctly.

Just not sure if this is a "UX" problem (the display mac-addresses command/switch has bugs) or potentially a driver/incompatibility issue or I'm missing something here.

Command Outputs:

vmnic02 is the only MAC address that's reporting "correctly". I guess you can assume that b5 is what the Huawei thinks is b6 and b7 is what the Huawei thinks is b8, but I have no explanation as to why.

Card is this from what I can tell, going by the PCI slots (quad card, not dual) http://accessories.euro.dell.com/sna/productdetail.aspx?c=ie&l=en&s=dhs&cs=iedhs1&sku=540-bbfh#Overview

Name    PCI Device    Driver  Admin Status  Link Status  Speed  Duplex  MAC Address         MTU  Description
------  ------------  ------  ------------  -----------  -----  ------  -----------------  ----  -------------------------------------------------------------
vmnic0  0000:01:00.0  bnx2x   Up            Up           10000  Full    11:22:33:44:55:b5  1500  Broadcom Corporation QLogic 57800 10 Gigabit Ethernet Adapter
vmnic1  0000:01:00.1  bnx2x   Up            Up           10000  Full    11:22:33:44:55:b7  1500  Broadcom Corporation QLogic 57800 10 Gigabit Ethernet Adapter
vmnic2  0000:01:00.2  bnx2x   Up            Up            1000  Full    11:22:33:44:55:b9 1500  Broadcom Corporation QLogic 57800 1 Gigabit Ethernet Adapter
vmnic3  0000:01:00.3  bnx2x   Up            Up            1000  Full    11:22:33:44:55:bb  1500  Broadcom Corporation QLogic 57800 1 Gigabit Ethernet Adapter
vmnic4  0000:83:00.0  bnx2x   Up            Up           10000  Full    aa:bb:cc:dd:ee:20  1500  Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter
vmnic5  0000:83:00.1  bnx2x   Up            Up           10000  Full    aa:bb:cc:dd:ee:22  1500  Broadcom Corporation QLogic 57810 10 Gigabit Ethernet Adapter


display mac-address | include 1122-3344-55
---- Flags: * - Backup
-------------------------------------------------------------------------------
MAC Address    VLAN/VSI                          Learned-From        Type
-------------------------------------------------------------------------------
1122-3344-55b6 15/-                               10GE1/0/1           dynamic
1122-3344-55b8 16/-                               10GE2/0/1           dynamic
1122-3344-55b9 19/-                               10GE2/0/9           dynamic
-------------------------------------------------------------------------------

EDIT 2017/06/28:

Checked iDRAC on host, two 10GbE ports are configured in Standard Mode and iSCSI is not enabled on the adapters nor in ESXi (portgroup has iSCSI Bindings disabled) so I don't think is a case of a "leaking" iSCSI frame: if it was, you'd likely see an extra MAC address, not an incorrect one as the total MAC count, despite being off incrementally, is still correct.

One thing I noticed is that ESXi vSphere Win32 client sees it as two dual adapters under host > Configuration > Network Adapters.

Checked driver, seems ok, so still thinking this is a Huawei issue. Think next step is to do a packet capture and/or call Huawei support.

Improve this question
9
  • Have you double-checked with the MAC addresses the other switches show for the Huawei switch links? If these are as expected there must be a problem with the display function.
    – Zac67 ♦
    Jun 23, 2017 at 23:11
  • I wonder if you're expecting the wrong MAC address value. In your example, xx:b5 and xx:b6 aren't off by one bit, they're off by two. 5 is 0x101 and 6 is 0x110. The last two bits have to change. So that would imply something other than a bad bit. I don't think anything is wrong with the switch.
    – Ron Trunk
    Jun 24, 2017 at 0:35
  • Unfortunately, questions about host, server, or VM configurations are off-topic here. You could try to ask this question on Server Fault for a business network, or on Super User for a personal network.
    – Ron Maupin ♦
    Jun 24, 2017 at 18:06
  • @RonMaupin this is purely a network question; obviously any switch in production is going to have hosts on it, so there's going to be some inevitable overlap.
    – gravyface
    Jun 26, 2017 at 18:32
  • @RonTrunk see edit re: MAC addresses and display output.
    – gravyface
    Jun 26, 2017 at 18:51

3 Answers 3

Reset to default
2
+100

Rereading your question, I'm pretty sure what the cause is: it's a dual NIC and you just mentioned a single MAC. Dual or quad NICs often (always?) have consecutive MAC addresses(*) and xxxb6 is just the 2nd port.

Depending on your vSwitch NIC teaming/load-balancing setup, the first port may not be in use - so it's invisible in the upstream switch. Open the vSwitch and check the Management Network NIC settings: it shows the MAC the host's network connection uses. On our hosts, it's that of the first NIC listed in vSwitch -> NIC Teaming.

If you do want to always use the first ports MAC you can set specific Management Network teaming with the second port only as fallback. Both ports active will arrange a somewhat random binding of Management Network to physical port.

(*) Our quad NICs with iSCSI option use every other MAC for normal Ethernet.

Improve this answer
10
  • But if the port isn't in-use, then the switch wouldn't have learned it's MAC address as it hasn't received a frame yet. Regardless, it still doesn't explain why it's learned MAC addresses that I cannot account for on the ESXi host and why they're off by 2: I did a partial filter on xxxx-xxxx and none of those MACs match what ESXi is reporting (via GUI or esxcli), but yet they're so close, they have to be the same interfaces.
    – gravyface
    Jun 26, 2017 at 18:51
  • If it isn't in use the switch can't learn the MAC, 100% correct. Define "isn't in use": it's unconnected? Configured as fall-back only? Possibly it's just a mix up of ports - the first port may just have the higher MAC and the 2nd the lower. Alternatively, the port order varies between figures on the slot and enumeration in the driver.
    – Zac67 ♦
    Jun 26, 2017 at 19:04
  • see my edit above for command outputs.
    – gravyface
    Jun 26, 2017 at 19:07
  • According to the datasheet, the card's got an iSCSI initiator - MACs alternate between plain Ethernet and iSCSI. Even if plain Ethernet isn't used the iSCSI part may be leaking out a frame or two.
    – Zac67 ♦
    Jun 26, 2017 at 19:11
  • interesting, I'll have to dig a bit further into the vmnic usage; may be that the "correct" MAC on the quad card is the only active Ethernet interface in an active/active pair on the vSwitch.
    – gravyface
    Jun 26, 2017 at 19:27
1

(Would prefer to comment but not enough reputation...)

If I was troubleshooting this on my network, first thing I'd do is run a packet capture from the vmware side. This will confirm what MAC is being sent to the Broadcom switches. https://pubs.vmware.com/vsphere-55/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-1BFB05E3-7171-464C-9603-8781C5A97CC6.html

You might also try plugging the same port into a different switch to see if the MAC changes.

I'd also be curious to view upstream switches to see what MAC the Broadcom switches are passing on (presumably the same +1 MAC, right?).

My intuition says that the vmnic is passing on the +1 mac (Cisco devices increment their MACs for each interface, maybe something similar for vmware?) or there's ahem human error involved at some point.

Improve this answer
2
  • can't plug in anything; remote data centre. +1 to sniffing, but not sure if ESXi supports tcpdump; was next on the Google hit list, however.
    – gravyface
    Jun 26, 2017 at 18:35
  • thanks for this too. Didn't tell me what I needed, but super handy nonetheless (but apparently I can't upvote this).
    – gravyface
    Jun 28, 2017 at 19:20
0

It's your filter that's wrong. Syntax for mac addresses varies. Sometimes its ff:ff:ff: ... and sometimes it's ffff.ffff ..., etc. Double check your syntax.

Improve this answer
3
  • It depends on the device. On example, Cisco displays MAC's as xxxx.xxxx.xxxx, HP SOHO - xxxxxx-xxxxxx, HP A/E - xxxx-xxxx-xxxx, Juniper - xx:xx:xx:xx:xx:xx
    – Andrey Prokhorov
    Jun 26, 2017 at 6:04
  • Huawei uses a dashed notation (xxxx-xxxx-xxxx), ESXi uses a semi-colon notation (xx:xx:xx:xx:xx:xx); double-checked many (many) times, not a filter/formatting issue.
    – gravyface
    Jun 26, 2017 at 18:36
  • well heck, that was my 2 cents. ...BTW 'x' is not a valid hex value :-)
    – Ronnie Royston
    Jun 26, 2017 at 20:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.