0

Posting this in response to the closed question: Network flooding without ff-ff-ff-ff-ff-ff (layer 2 broadcast) MAC address in destination?

I have a network that is experiencing the following problem: frames addressed to a particular machine (ip 192.168.107.125, mac bbbb.bbbb.bbbb) are sent to certain devices in the same VLAN.

For example, a wireshark capture on another machine (ip 192.168.107.10, mac aaaa.aaaa.aaaa) lists packets addressed to (ip 192.168.107.125, mac bbbb.bbbb.bbbb). The traffic is FTP traffic (including logins and passwords), so I am pretty sure that it has no place on 192.168.107.10.

I have also spotted that the mac address-table entry for bbbb.bbbb.bbbb is missing when the flooding happens. After a restart of bbbb.bbbb.bbbb the MAC entry comes back, but only for a while.

All pictured servers (network clients) are in the same VLAN

Switch models are shown on the diagram. Configuration is shown on the diagram and listed below.

Network DIagram

EDIT 1: The device using the MAC that gets lost definitely emits frames before the address disappears - we have wireshark captures of this. In fact, the device continues to emit frames even after the disappearance so I would expect the switch to create a new MAC address-table entry.

I have checked the switch logs, and there are no signs of link flapping on the port that loses MAC of the connected client. For example, on one occasion when the MAC address was lost I have restarted the client device. In logs I could see the interface going through the designated->blocking->learning->forwarding stages (within 30 seconds) and then no messages appear about that interface. After the restart the MAC entry was still missing! Only moving the device to another port has made it appear for a while.

EDIT 2:

Nexus 1 config:

version 8.2(5)
feature-set fex
switchname Core
                 
feature telnet
feature vrrp
feature scheduler
feature ospf
feature pim
feature msdp
feature eigrp
feature port-security
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature ptp
feature lldp
feature sla sender
feature sla responder

logging level aaa 5
logging level cdp 6
logging level hsrp 5
logging level interface-vlan 5
logging level monitor 6
logging level otm 5
logging level radius 5
logging level spanning-tree 6
logging level dhcp_snoop 5
logging level vpc 5

ip domain-lookup
service unsupported-transceiver
errdisable recovery cause link-flap
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause loopback
errdisable recovery cause storm-control
errdisable recovery cause security-violation
errdisable recovery cause psecure-violation
errdisable recovery cause vpc-peerlink
errdisable recovery cause failed-port-state

ip access-list accessblock121
  statistics per-entry
  11 deny ip 192.168.107.0/24 192.168.121.200/32 
  30 permit ip any any 
  
ip access-list cape
  statistics per-entry
  10 permit icmp 192.168.120.125/32 192.168.107.152/32 
  20 permit ip any any 

ip access-list tac
  statistics per-entry
  10 permit icmp 192.168.120.159/32 192.168.107.152/32 
  20 permit ip any any 
  
time-range 02:07:00

ip dhcp snooping
service dhcp
ip dhcp relay
ipv6 dhcp relay
ipv6 dhcp guard policy DHCP_CLIENT
!
class-map type qos match-any VLAN_QOS
policy-map type qos NFLINT
  class class-default
    police cir 200 mbps bc 200 ms conform transmit violate drop
    
fex 42
  pinning max-links 1
  debounce time 0
  description FEX_42
  
fex 45
  pinning max-links 1
  debounce time 0
  description FEX_45
  

ip pim rp-address 192.169.180.3 group-list 224.0.0.0/4
ip pim auto-rp mapping-agent Vlan107
ip pim ssm range 232.0.0.0/8
ip pim auto-rp forward
ip pim pre-build-spt
ip igmp any-query-destination

vlan 107
  name EEE

spanning-tree vlan 107 priority 4096
vrf context keepalive
vrf context management
  ip route 0.0.0.0/0 192.168.121.254
vpc domain 10
  peer-switch
  role priority 1500
  peer-keepalive destination 192.168.145.14 source 192.168.145.13 vrf keepalive
  peer-gateway
  config-sync
  ip arp synchronize

cfs eth distribute

interface Vlan107
  description EEE1
  no shutdown
  mtu 9216
  no ip redirects
  ip address 192.168.107.252/24
  no ipv6 redirects
  ip ospf passive-interface
  ip pim sparse-mode


interface port-channel1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  vpc peer-link


interface port-channel42
  description FEX_42
  switchport
  switchport mode fex-fabric
  fex associate 42
  mtu 9216

interface port-channel45
  description FEX_45
  switchport
  switchport mode fex-fabric
  fex associate 45
  mtu 9216


interface Ethernet4/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown

interface Ethernet4/3
  description VPC KeepAlive Link
  vrf member keepalive
  ip address 192.168.145.13/24
  no shutdown

interface Ethernet5/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown

interface Ethernet5/45
  description FLOODING_ADDRESSED_HERE
  switchport
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown

interface Ethernet7/46
  description NO_FLOODING_HERE_1                    
  switchport
                                                  
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown

interface Ethernet42/1/10
  description NO_FLOODING_HERE_2
  switchport
  switchport access vlan 107
                            
  no shutdown

interface Ethernet45/1/10
  description NO_FLOODING_HERE_3
  switchport
  switchport access vlan 107
  no shutdown

logging logfile messages 6
no terminal log-all
line console
  terminal width  80
line vty
                           
router ospf core
  network 192.168.107.0/24 area 0.0.0.0

monitor session 2 
  source interface Ethernet5/45 both
  destination interface Ethernet5/11
ip dhcp snooping vlan 107

scheduler logfile size 1024

Nexus 2 configuration:

version 8.2(5)
feature-set fex
hostname HOSTNAME

feature privilege
feature telnet
feature vrrp
feature scheduler
feature ospf
feature pim
feature msdp
feature eigrp
feature port-security
feature interface-vlan
feature hsrp
feature lacp
feature dhcp
feature vpc
feature ptp
feature lldp
feature sla sender
feature sla responder

logging level aaa 5
logging level cdp 6
logging level hsrp 5
logging level interface-vlan 5
logging level monitor 6
logging level otm 5
logging level radius 5
logging level spanning-tree 6
logging level dhcp_snoop 5
logging level vpc 5

ip domain-lookup
service unsupported-transceiver
errdisable recovery cause link-flap
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery cause loopback
errdisable recovery cause storm-control
errdisable recovery cause security-violation
errdisable recovery cause psecure-violation
errdisable recovery cause vpc-peerlink
errdisable recovery cause failed-port-state

ip access-list accessblock121
  statistics per-entry             
  11 deny ip 192.168.107.0/24 192.168.121.200/32                       
  30 permit ip any any
ip access-list cape
  statistics per-entry
  10 permit icmp 192.168.120.125/32 192.168.107.152/32 
  20 permit ip any any 

ip access-list tac
  statistics per-entry
  10 permit icmp 192.168.120.159/32 192.168.107.152/32 
  20 permit ip any any 
                           
ip dhcp snooping
service dhcp
ip dhcp relay
ipv6 dhcp relay
ipv6 dhcp guard policy DHCP_CLIENT
!
class-map type qos match-all trustme
fex 48
  pinning max-links 1
  debounce time 0
  description FEX_48
  
fex 54
  pinning max-links 1
  debounce time 0
  description FEX_54

ntp server 192.168.140.13
ntp server 192.168.140.14

ip pim rp-address 192.169.180.3 group-list 224.0.0.0/4
ip pim auto-rp mapping-agent Vlan107
ip pim ssm range 232.0.0.0/8
ip pim auto-rp forward
ip pim pre-build-spt
ip igmp any-query-destination

vlan 107
  name EEE                                       
vrf context keepalive
vrf context management
  ip route 0.0.0.0/0 192.168.121.254
vpc domain 10
  peer-switch
  role priority 1000
  peer-keepalive destination 192.168.145.13 source 192.168.145.14 vrf keepalive
  peer-gateway
  config-sync
  ip arp synchronize
cfs eth distribute


interface Vlan107
  description EEE1
  no shutdown
  mtu 9216
  no ip redirects
  ip address 192.168.107.254/24
  no ipv6 redirects
  ip ospf passive-interface
  ip pim sparse-mode


interface port-channel1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  vpc peer-link

interface port-channel48
  description FEX_48
  switchport
  switchport mode fex-fabric
  fex associate 48
  mtu 9216

interface port-channel54
  description FEX_54
  switchport
  switchport mode fex-fabric
  fex associate 54
  mtu 9216
  vpc 54

interface Ethernet4/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown
  
interface Ethernet4/3
  description VPC KeepAlive Link
  vrf member keepalive
  ip address 192.168.145.14/30
  no shutdown
  
interface Ethernet5/1
  description VPC Peer-Link
  switchport
  switchport mode trunk
  switchport trunk allowed vlan 107
  spanning-tree port type network
  channel-group 1 mode active
  no shutdown

interface Ethernet6/41
  description FEX_48
  switchport
  switchport mode fex-fabric
  fex associate 48
  mtu 9216
  channel-group 48
  no shutdown

interface Ethernet6/42
  description FEX_48
  switchport
  switchport mode fex-fabric
  fex associate 48
  mtu 9216
  channel-group 48
  no shutdown

interface Ethernet7/28
  description Link FEX54
  switchport
  switchport mode fex-fabric
  fex associate 54
  mtu 9216
  channel-group 54
  no shutdown

interface Ethernet48/1/3
  description FLOODING_RECEIVED_HERE_1
  switchport
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown

interface Ethernet48/1/8    
  description FLOODING_RECEIVED_HERE_2
  switchport
  switchport access vlan 107
  ipv6 dhcp guard attach-policy DHCP_CLIENT
  no shutdown
  
interface Ethernet54/1/10
  description FLOODING_RECEIVED_HERE_3
  switchport
  switchport access vlan 107
  no shutdown
  
logging logfile messages 6
no terminal log-all
line console
  terminal width  80
line vty
router eigrp 10
  router-id 192.168.133.253
  default-information originate
router ospf 1
router ospf core
  network 192.168.107.0/24 area 0.0.0.0
monitor session 2 
ip dhcp snooping vlan 107

scheduler logfile size 1024

EDIT 3: Hypothesis: the MAC address is missing because it ages out. Thank you, Zac67, I would like to test this further. When MAC address-table entry for bbbb.bbbb.bbbb was missing, I have exported ARP and CAM tables from both switches. ARP entry exists:

192.168.107.125 00:15:53 bbbb.bbbb.bbbb Vlan107

But CAM tables on BOTH switches don't contain this MAC entry! I know that if we see flooding only on one side one would conclude that the entry is missing only on that side, but that is not the case: flooding is happening only on one side, and both CAM tables are missing this entry.

Also, when the MAC address-table entry for bbbb.bbbb.bbbb was missing from the switch, I have taken a SPAN capture on the interface where this client was connected to, and saw the following: SPAN capture

I interpret this as proof that the switch has received frames with bbbb.bbbb.bbbb Source H/W address encapsulated within them. Even if the MAC entry has timed out, the switch should have re-created it, correct?

EDIT 4: A support case has been opened with Cisco by my colleagues. This is now under 'deeper' investigation, as they could not explain the cause of this behaviour straight away.

Improve this question
12
  • 1
    Please edit the question to include the switch configuration. Speculation and guessing are off-topic here, and that is all we could do without more information.
    – Ron Maupin ♦
    May 3, 2021 at 12:33
  • @RonMaupin, I did my best to include the details and provided more information in this new question. What exactly would you like me to add?
    – Tony Sepia
    May 3, 2021 at 12:37
  • Do you mean post the exact switch config in textual form?
    – Tony Sepia
    May 3, 2021 at 12:38
  • 1
    Something like this question. You should obfuscate any public addresses and passwords.
    – Ron Maupin ♦
    May 3, 2021 at 12:40
  • @RonMaupin I have added the switch details. Can you please re-open the question?
    – Tony Sepia
    May 4, 2021 at 13:48

1 Answer 1

Reset to default
2

frames addressed to a particular machine (ip 192.168.0.20, mac bbbb.bbbb.bbbb) are sent to certain devices in that VLAN.

Unless that MAC address is a multicast address, it needs to be unique in its VLAN - "certain devices" should be "a certain device".

[edit] You seem to be referring to devices attached to the right-hand Nexus - sorry about that.

a wireshark capture on another machine (ip 192.168.0.10, mac aaaa.aaaa.aaaa) lists packets addressed to (ip 192.168.0.20, mac bbbb.bbbb.bbbb). The traffic is FTP traffic (including logins and passwords), so I am pretty sure that it has no place on 192.168.0.10.

Switches don't care about IP addresses, the only thing relevant is the MAC address - source for learning, destination for forwarding.

As in your previous question, a MAC address is removed from its port association when

  • its associated port loses its link
  • it isn't seen for a certain time and aged out by the switch (that time is configurable usually)
  • it is seen as source on another port

When a destination MAC isn't present in its source address table, a switch floods that frame to all ports, like a broadcast, mimicking a repeater hub. However, all NICs not addressed by that MAC will simply ignore that frame - so it'll just waste bandwidth but won't cause issues.

So, make sure that MAC aging in the switches is configured appropriately and that the device using the MAC emits frames before the addressed is aged out. Also check the logs for any link flapping that might cause the MAC to be dropped prematurely. As a workaround, you could try triggering an ARP query every few minutes or so.

Since it's the right-side Nexus that losing the MAC address from its table, you should make sure that there's at least some traffic originating from 192.168.0.20 hitting that switch before the aging times out the address (defaults to 1800 seconds). Normally, somewhat frequent broadcasts (e.g. ARP) ensure reliable updating of all switches but that node might not do/need that. If you can't extend the aging, pinging across that switch in either direction should solve the problem.

You should try setting a longer MAC aging timeout:

mac address-table aging-time seconds [ vlan vlan_id ]

If that doesn't help, you might put a static mapping on the desired port:

mac address-table static address mac_addr vlan vlan_id [ interface type slot/port ]

See https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus7000/sw/layer2/command/cisco_nexus7000_layer2_command_ref/cisco_nexus7000_layer2_command_ref_chapter_011.html for details.

Improve this answer
4
  • Sorry Zac67, you must have misunderstood me - the frames addressed to machines are sent to certain devices (plural, as the diagram shows with the case of flooding unknown unicast). And thank you for your other suggestions - I have added the edits to clarify these points
    – Tony Sepia
    May 3, 2021 at 10:08
  • Oh, you mean "all devices" (attached to the right-hand Nexus)?
    – Zac67 ♦
    May 3, 2021 at 10:14
  • Yes, that is correct
    – Tony Sepia
    May 3, 2021 at 10:23
  • Thank you, @Zac67! I have added EDIT 3 in response to your recent edits
    – Tony Sepia
    May 4, 2021 at 15:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.