Cisco 2821 hangs with "No memory"

Question

I have one (pretty old) 2821 router, running BGP with 2 ISPs.

(C2800NM-ADVENTERPRISEK9_IVS_LI-M), Version 12.4(24)T5

Sometimes for unknown reason it just hangs, and as it is remote location I cannot provide more insight so far. One time an engineer was on site, and he managed to hook onto console and said it was showing alot of errors regarding to memory shortage.

Now, on that same site I've had this same problem before, we discovered one of ISPs randomly decided to send us full view in update, and obviously router didn't survive that. But after that accident I've put prefix-list on inbound updates, and it solved the problem.

Or so I thought, because now the problem is back (after 2 years) and I cannot blame BGP any more because prefix-lists should filter out anything but default route.

I have enabled syslog to another server to be able to read some log history, but the question is what exactly do I put into 'debug' to be able to see what was causing the issue or was happening right before it?

Any hints on what to look for to get more details on this problem will be much appreciated.

Answer 1

Short answer is this: You do not need to enable any debugs to have Traceback messages sent to syslog, provided the router can send those messages upon crashing (which does happen sometimes, thanks to modular software design). You should also ensure that you have BGP neighbor x.x.x.x soft-reconfiguration turned off because that will cause the router to store BGP advertisements even if a route-map rejects them.

Longer answer is this: You say that a couple years ago the router ran out of memory because it was storing full internet routes. You say that you added a route-map to reject all but the default route. If the router has the soft-reconfiguration feature enabled, then it was storing the routes twice--once in the memory for BGP, and once in the regular routing table. If this is so, then it's possible that the router is running out of memory a second time now (two years later) because there are more internet routes now than ever before, and the memory is filling up for a second time because of the routes that BGP is caching, even though the route-map rejected them.

If soft-reconfiguration is not on, then we take a step back and look for other causes. Generally speaking, it would be fair to say that routers crash for mostly two reasons--software bugs and being out of memory. We assume that you're not hitting any software bugs, since you didn't mention any software updates, config changes, etc. Usually running out of memory is caused only by a few scenarios: 1) the router has less memory than the software image requires, 2) the memory got all filled up by something, or 3) the memory is failing.

Perhaps you've had the second scenario in the past with BGP, but in a case like your current one where there is little evidence to make any conclusions at all, the third scenario seems the most likely. As @user56700 mentions, you should look in the log for any Traceback messages, and also look in the flash for any crashinfo files. Once the router has crashed, you will not usually be able to look at the log, therefore getting syslog to work correctly, as you mentioned, is a great first step. Generally, a router can perform some final actions when it knows that it's crashing, like sending syslog messages, and writing a memory dump to flash, so you have a fair chance at catching a Traceback message, if one is generated.

Unfortunately, that router model is end-of-life, so getting Cisco to look at a Traceback or a crashinfo file is basically zero. Might be time to upgrade that router!

Answer 2

Thank you for suggestions, the problem was solved.

As I said, I couldnt blame BGP this time, and soft-reconfiguration was already off after first time.

After babysitting this router I managed to catch the moment when it was almost dead, and show proc mem sorted allocated showed me that "L2TUN Application" service was the offender.

Then I used sh vpdn to confirm and some debugging to trace and discovered that someone was flooding with half-open pptp sessions (client starts a call and just ignores any further negotiation, maybe evil ISP blocking PPTP packets).

Radius accounting logs showed me the IP and confirmed that it was repeat offender for last 8 days, and after blocking it in ACL the problem was solved. I think it is a broken client (or ISP) rather than attacker. Need to add some monitoring to Radius to see if the number of failed attempts grows unexpectedly, my main problem here is I don't watch Radius at all and didn't think of it as a troubleshooting help.

Hope my findings help anyone in same situation. I still have no idea why does this cause router to leak all memory to VPN service, and what's more important - how to protect against such flood (be it intentional or not)