1

We are making some changes to our network and after changing which router we use as our core router, there is a strange problem.

The problem is that there are two subnets (out of a total of more than 10) where hosts on those subnets can't be reached by hosts on other subnets, but as soon as I log on to the core switch/router (Catalyst 4507E+) and ping one of those hosts, then everyone can ping the host in question. Then about ten minutes after I've done the ping from the core router, the host that I enabled becomes unreachable from other subnets again.

The subnets are directly connected to the core.

So far my only guess is that there is some ARP problem. Pinging from the core helps these nodes successfully ARP and then the ARP cache entries time out and they are unreachable again, perhaps?

Edit:

I've been unwittingly monitoring Windows hosts on the subnet with the main problem all along and they have never been unavailable, so it only affects Linux hosts on that subnet. That makes me think maybe this is more a question for ServerFault, since the problem is OS-specific.

  • Do you have dynamic arp inspection enabled (DAI) or arp access-lists? Is all hosts connected directly to 4500? Or do you have some trunks to others switches? – Datagram.Network Feb 7 '17 at 22:40
  • Try adjust #mac-address-table aging-time for particual VLAN for much higher value and check it out. Is 4500 in VSS mode? If you have destination host going thru another switch. Does switch is connected into both 4500? or just one? – Datagram.Network Feb 7 '17 at 22:48
  • @Datagram.Network ARP inspection is disabled. None of the hosts are directly connected to the core, neither working nor non-working hosts. The core is not also an access or edge switch. The ARP entries are not aging out and disappearing on the core. I wonder if the ARP entries coming the other way on the non-working hosts might be aging out (which might be more of a serverfault question). I don't believe that the 4500 is in VSS mode because I don't know what that is. Keep in mind there are 8 other functioning subnets directly connected to the 4507. – Todd Wilcox Feb 7 '17 at 23:00
  • Do you have two 4500? How they are connected into access switch? Those other subnets are on diffrent vlan I assume? VSS mode: cisco.com/c/en/us/products/collateral/switches/… – Datagram.Network Feb 7 '17 at 23:01
  • @Datagram.Network No, I have a selection of Catalyst 3850s, 2960Xs, and Dell PowerConnect 6248s. There doesn't seem to be a correlation between non-working subnets and access switches. – Todd Wilcox Feb 7 '17 at 23:02
3

CAM and ARP table are populeted while you ping from 4500. Is just aging faster from CAM than from ARP. If you would wait 4 hours it would time out as well from ARP. Is those 4500 clustered? If yes I would say that VLAN is not getting to Active node for mac address population. All the request are coming to passive node and they are dropped. Maybe STP topology for particular VLAN is blocking state to Active node. I would compare STP topology of working vlan with not working one. Double check uplinks from access layer for this VLAN. Could be as well that VSL link between 4500 is not allowing particualr vlan. You need as well make sure that both 4500 have ports in this vlan. As well check floodset between 4500:

#show platform hardware floodset vlan [VLANID]

If both 4500's having floodset on required interfaces.

Temporary fix would be increase vlan aging time for exmaple:

#mac address-table aging-time 14400 vlan [VLANID]
  • There is no cluster. I'm going to try the aging time workaround for now and see if I can run down the real cause. Looks like the floodset output is correct. – Todd Wilcox Feb 9 '17 at 15:08
  • So in your core you are having single 4500? Or two seperate with HSRP, VRRP or GLBP? – Datagram.Network Feb 9 '17 at 15:13
  • Wow. How can I answer this question once and for all? Yes, just one 4500. – Todd Wilcox Feb 9 '17 at 15:23
  • Than I would check out STP topology for particual VLAN and use SPAN to confirm arp flows. temporary fix: you can apply #mac address-table aging-time 14400 vlan [VLANID] – Datagram.Network Feb 9 '17 at 15:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.