1. Why configuration at layer 2 at layer 3 device at different site of LAN caused problem at another part of the network ?!
  2. What kind of mechanism at CC1036 has been activated that cause gaps in the ARP tables across the entire network many devices further ?!

ARP issue among whole network caused by MikroTik CloudCore1036 router and HP 1810-24G switch missconfiguration

At above picture You can see part of my network scheme. Main router based on Mikrotik CloudCore1036 within one bridge and HP switches 1810-24G and 1810-8G - all default configured within default VLAN. All CC1036 interfaces were arp-proxy and arp-enabled mode.

  • CloudCore 1036 - main router with one bridge and included one link bond (causing issue balanced-rr and not causing issue 802.3ad,
  • swt-d-x switches - my distribution layer switches; all HP 1810-24G - J9803A with one default VLAN,
  • swt-a-x switches - my acces layer switches; all HP 1810-8G - J9802A with one default VLAN,
  • Server NAS- my local storage server,
  • Local station x - hosts connected to the network using DHCP server defined at CC1036 MikrotikRouter,
  • Bonding-1 - I bounded 3 links between CC1036 and swt-d-1 switch; at swt-d-1 I used trunks at E22,E23,E24 interfaces and at CC1036 at E1,E2,E3 I used balanced-rr protocol and I add that Bond to the bridge.

Root cause analysis:

At first look everything was fine - all hosts had access to the Internet and between them along the LAN.

First symptom was loosing connectivity to the NAS server from whole LAN site and from Internet using VPN.

Also some host from Local station 3 site couldn't localized LAN printers at Local station 1 - there was PING problems, etc. . The same was when I tried to connecto to different hosts at Local station 3 LAN site.

Also ARP tables at hosts at Local station 3 wasn't complete !!! They didn't know about MAC addresses of their closest neighbors !!!

I suspected that there is problem with swt-d-3.

When I disconnected at swt-d-3 switch link E23 to the CC1036 trying to replace that switch - monitoring server showed that problem disappear immediately.

It was strange because problem was isolated to the Local station 3 LAN part. Theoretically frames and packets shouldn't leave swt-d-3 switch because problematic traffic was limited to hosts connected to this switch ports.

When I reconfigured CC1036 and changed bonding protocol from balanced-rr to the 802.3ad between CC1036 and swt-d-1 the entire network began working well !!! All hosts began see each other, connectivity problems with Nas server disappeared, and all ARP tables have become complete.


Why configuration at layer 2 at layer 3 device at different site of LAN caused problem at another part of the network ?!

First, to be clear, since you mentioned that you are bridging the interfaces on the Cloud Core, this is part of the same L2 LAN, not a "different site."

Second, misconfiguring a network device can create all sorts of unintended problems. Part of being a good network troubleshooter is being able to separate the actual problem from the symptoms.

What kind of mechanism at CC1036 has been activated that cause gaps in the ARP tables across the entire network many devices further ?!

In your fix, you only mention changing the Cloud Core from balance-rr to 802.3ad. Did you change anything on swt-d-1? If so, what?

My guess is that you had a problem with STP, which can happen when you don't configure your link aggregations properly on both sides. Problems I have seen when this happen are L2 loops or situations where one device is sending traffic to the second while the second device has the port in a STP blocking state.

  • No i didn't change anything at swt-d-1. – user3799089 Jun 11 '15 at 9:40
  • Based on RouterOS documentation, RSTP is enabled by default on interface bridges. Based on the question and your comment, my guess is that the CC thought it was the root bridge and had all three links forwarding to swt-d-1, while swt-d-1 had two of those ports in blocking mode and dropping traffic causing traffic sent to them to be lost. When you switched it to LACP, with no LACP partner on the other side your bridged links are now actually running independently and only one is being used on both sides. No more lost traffic. – YLearn Jun 11 '15 at 13:53

With my knowledges and my Mikrotik adventures I advice to you :

  1. Use mode 802.3ad instead of balanced-rr if there's no major cause. If you read on Mikrotik WIKI - Bonding they say that its mode isn't 100% compatible with all of other networking products
  2. ARP table not need to be always full, may have only inputs from ip/mac targets which the host needs to talk

My diagnostics is mode balanced-rr is the root of your network problems

  • 1. I know that HP switch was incompatible with balanced-round robin protocol - thats why I write RCA but question is what kind of mechanism and how causes this kind of network behavior. \\ 2. But if You ping some device Your host must know the target and in this case "incomplete" record in ARP table is unacceptable :) – user3799089 Apr 4 '15 at 12:45
  • Small table is normal: en.wikipedia.org/wiki/Address_Resolution_Protocol#Example – gilito Apr 4 '15 at 12:49
  • Your entire network is in the same collision domain. It's probably that if ther's a layer2/3 problem, as bad bonding configuration for example, the entire network is faulting – gilito Apr 4 '15 at 12:56
  • You are wrong my friend. Entire network is one broadcast domain not collision one. Switch has single collision domain across one physical link - that approach eliminate collisions between hosts ;) Broadcast ARP shouldn't be cut off at "Local station 3" LAN part because that is under same switch management. 2 hosts should see each other regardless to the MikroTik configuration right ? – user3799089 Apr 4 '15 at 13:02
  • Ups yes with Broadcast domain, I was wrong. But with the rest I do not agree with you. However interesting discussion :) – gilito Apr 4 '15 at 13:35

Not the answer you're looking for? Browse other questions tagged or ask your own question.