Cisco 2600 series router logging a lot of "Memory allocation failed" errors

Question

We are getting the below two error messages randomly on a 2600 series router. Syslog shows the router has been fine up until a few days ago. Would a reboot most likely cure the issue or is it most likely faulty memory in this ancient beast?

-Process= "SNMP ConfCopyProc", ipl= 0, pid= 81
-Traceback= 80476224 80478348 8047224C 803E39A0 803E114C 803E0DA0 80450AEC 80450B34 803E5DA4 8045575C 803E1444 803BDA5C 803BDF20 804B345C 804AEA50 804AEB58
May 28 10:06:02.286: %SYS-2-MALLOCFAIL: Memory allocation of 32768 bytes failed from 0x80472248, alignment 0
Pool: Processor  Free: 24472  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool


-Process= "Virtual Exec", ipl= 0, pid= 67
-Traceback= 80476224 80478348 8047224C 803E39A0 803E114C 803E0DA0 80450AEC 80450B34 803E5DA4 8045575C 803E1444 803BDA5C 803BDF20 803F1FDC 803F20FC 803E368C
May 28 10:06:38.739: %SYS-2-MALLOCFAIL: Memory allocation of 32768 bytes failed from 0x80472248, alignment 0
Pool: Processor  Free: 23512  Cause: Not enough free memory
Alternate Pool: None  Free: 0  Cause: No Alternate pool

Running 12.3(26) version of IOS. Device uptime is two months.

Answer 1

Do a 'show proc mem sorted' and check for biggest memory users. If that's something that is not related to configuration, it may be bug in the code. If that's something that's related to configuration, you may consider re/deconfigure unnecessary services.

As IOS can't defragment memory and you can't reclaim it unless some big allocating process frees memory, I'm afraid that to return to normal operation you need to reboot it. Then I'd start to monitor closely memory allocation in total and for the main processes that are using it. If it grows and router is not doing anything additionally, it's propably bug in the "growing" process. 12.3(26) has some memory oriented bugs, but you'll need to audit the config to check if you're affected.

Answer 2

If you are getting memory allocation errors, a reload should be in order to clear up whatever is going on, at least temporarily.

It looks like you are running out of memory (Cause: Not enough free memory). I would look at perhaps upgrading the memory, which is the cheapest option, or if this router is important enough, even upgrading to a new platform that can more readily support your needs.

Also, try upgrading the IOS. It may be a minor memory leak.

Answer 3

A reboot will surely alleviate the problem for the moment, but as time passes and router memory loads you will probably hit the same log error.

You can either remove some roles you have for this router, or a more uncommon method, would be to schedule an automatic reload at every 1 week or later.

Answer 4

I do not believe it is faulty memory. The errors listed are memory allocation failures. A reload will cure it temporarily, but will not fix the root cause. More information is needed to determine the source. See if you can find out what process is using up the memory. sh proc memory