8

I have 2 force10 s25 switches that are used for a dark fiber link between two school buildings. One of my switches is filling its ARP table to the brim and then failing. This can happen as often as every 1.5 hours. I noticed that one of the addresses in the cache is an address from our previous web filter that has not been online in 6 months. We can't think of anything that would be holding an ARP cache but something seems to be feeding our force10 with addresses, but by running show arp summary, sometimes up to 50 new addresses per second are being added. What should I be looking for to figure out where this is coming from?

Here is my show ip route connected

MiddleSchool#show ip route connected
   Destination        Gateway                      Dist/Metric Last Change
   -----------        -------                      ----------- -----------
  C    10.4.0.0/16        Direct, Vl 400                       0/0    03:39:18
  C    192.168.1.0/30     Direct, Te 0/28                      0/0    03:39:20

The only static route is the default route (0.0.0.0 /0 over the tengig 0/28 port) over the fiber link since all traffic must leave this switch and go to the other building to reach the internet.

Improve this question
3
  • 4
    based on the very limited information we have now, my best guess is you have a machine(s) spoofing arps to overflow the mac address table. Another possibility is a machine with a virus / worm / network mapper that transit interfaces configured to use proxy arp for next-hop resolution. Please edit in "show ip route connected" (or whatever you use on Force10 as the equivalent of that Cisco IOS cmd), as well as any static routes that point to a connected interface instead of a next-hop ip address. Look at your mac address table for the source of many mac addresses that should be a single PC.
    – Mike Pennington
    Sep 16, 2013 at 17:08
  • Are there any patterns or repeats in the ARP table or are the majority of the MAC addresses entries random? Are there any manually configured ARP cache timeout entries beyond the defaults (which is typically four hours)? Are there any downstream switches that could potentially be looped? It may be worth tracing down to the edge if there are downstream switches.
    – one.time
    Sep 16, 2013 at 17:12
  • The switches appear to be receiving a broadcast storm, but we can't pinpoint it. We unplugged every connection to see if anything would cause the intense activity to stop or slow down but we were unsuccessful. If I look at the ARP table, the unusual part is that it is storing internet addresses for the IP addresses and they are all linked to the MAC address of the switch. Our network is 10.4.0.0 /16 in this building, and we will even see addresses that are something like 10.4.85.*, so very close to our subnet. There are also 192.168 addresses outside of our link route showing as well.
    – MooseBalm
    Sep 16, 2013 at 17:49

1 Answer 1

Reset to default
6

I looked at the configs in your stack overflow question.

By way of review, this is your topology...

      Ten0/28  Ten0/28
Bldg_L----------------Bldg_S
F10 S25               F10 S25
  |                     |
  Vlan200               Vlan400
  10.2.0.101            10.4.0.101/16

The problem is that building L's switch proxy-ARPs to resolve 10.4.0.0/16 and building S's switch proxy-ARPs for 10.2.0.0/16... interface TenGig0/28 (your transit link between the buildings) is answering proxy-ARPs requests. Remove those 10-net statics and use...

  • Building L: ip route 10.4.0.0 255.255.0.0 192.168.1.2
  • Building S: ip route 10.2.0.0 255.255.0.0 192.168.1.1

The reason that a route like ip route 10.4.0.0 255.255.0.0 TenGigabit0/28 proxy-ARPs is because you are essentially telling the switch that the entire /16 subnet is directly connected to TenGigabit0/28 when you static route out an interface like this. Using an IP next-hop only requires an ARP entry for that specific next-hop.

You probably need to move the default gateway to a new interface on the Building L switch, so the whole subnet can default through 10.2.0.101 and either reach 10.4.0.0/16 or the internet.

Sorry to say it, but you are leaving yourself wide open to ARP resource exhaustion problems when you assign a /16 as a connected subnet... ARP is an unauthenticated protocol, and anyone on the LAN can flood the switch with ARPs and it has no choice but to cache / answer them... even for phantom addresses.

Proactively, you might consider DHCP snooping and dynamic ARP inspection, if your version of FTOS supports it. These feature normally require some thought and testing before deployment; however they are well worth using if you have 100s of kids with nothing more exciting than showing off their "hacking" skills. I did a quick search to see if Force10 supports what Cisco calls port security, but I couldnt find it; port security can be used to limit the number of macs learned on a switch port.

Improve this answer
2
  • I meant to say that it was close to our scope. I know it isn't in the subnet, that is my fault. Our DHCP scope in this building is 10.4.50.1-10.4.51.254, so the 10.4.85.* is not a dhcp address we would issue. I have changed the routes so they use the IP addresses instead of the gateway. I can't disconnect everything until tonight, but currently the MAC address table is at 246 Dynamic addresses, and 0 static or sticky addresses.
    – MooseBalm
    Sep 16, 2013 at 19:07
  • After changing the routes, that seems to have fixed it. If anything changes, I will update the post, but for now my ARP table is sitting at 230 records and hasn't jumped from there in 10 minutes. Many thanks. Your help has been much appreciated.
    – MooseBalm
    Sep 16, 2013 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.