TAC Response on the error( %SYS-2-MALLOCFAIL: Memory allocation)

Question

I hope you can offer me some assistance with my case, we have a Cisco router 2800 series ,having the error ( %SYS-2-MALLOCFAIL: Memory allocation), (and if restarted the router it works OK for while.

our admin who is responsible for opening case with TAC contacted them and the answer from them was as follows, but we did not understand it so, can you explain this?

---

SHOW PROCESS MEMORY NOTIFICATIONS (if any)

INFO: The output of 'show process memory' only shows the memory associated with the processor and does not identify other memory such as I/O, Fast, VM, etc. To receive a statistical analysis on these types of memory, submit the first page of output from the 'show memory' and 'show version' commands to Output Interpreter. NOTE: The types of memory vary depending on router platform and installed modules.

INFO: Processor memory utilization is 11.22801%. INFO: Processor memory or main memory stores the running configuration and routing tables. The Cisco IOS software executes from main memory. INFO: The amount of processor memory required by the router is affected by the Cisco IOS version used, the size of the network and by the access list configurations. Ensure that an optimal IOS version has been chosen.

INFO: The top 3 processes that are holding less than 1 MB of memory are: - 'Chunk Manager' is holding 964368 bytes - 'Exec' is holding 337640 bytes - 'LICENSE AGENT' is holding 37592 bytes

and

ERROR MESSAGE NOTIFICATIONS (if any)

ERROR: No Error message was found. TRY THIS: Ensure that the error message has the following format: %FACILITY-SEVERITY-MNEMONIC: Some examples would be: %SYS-2-MALLOCFAIL: %PIX-1-103001: Resubmit the output as necessary.

Answer 1

The most common reason I've seen for alloc failures has been serious misconfiguration of the default route -- ip route 0.0.0.0 0.0.0.0 f0/0 will cause this because it's proxy-arp, and the ARP table will grow to an insane size.

A distant second place goes to dynamic routing protocols causing severe memory fragmentation.

We (and TAC) will need a lot more information to debug this. The first place I would suggest looking is the output of show memory failures alloc after failures are happening. (before rebooting)

Note: because memory is relatively cheap (when not bought from Cisco), I tend to load my routers with as much memory as they'll hold.

Answer 2

I can tell you the last time I dealt with a MALLOCFAIL was caused by an IOS bug. I had to upgrade to a later version of IOS. But, if I recall correctly, that was on one of my 2901s. There are a lot more troubleshooting stops you would have to make before I would try that, though. If you want to do more troubleshooting try, show tech, show memory, and show buffers.

Answer 3

There are many reasons for such errors, and it may affect processor and I/O memory pools so the first step is to identify the affected pool.

Mallocs maybe due to a memory leak caused by a sw bug or misconfiguration, memory lack due to overwhelming the router with features and services or memory fragmentation which may also be caused by a bug.

In order for TAC to troubleshoot more details are needed and you'd probably need to take several iterations of "show proc mem sorted, show memory stat, etc.." and depending on the scenario it may be over a period of days.

I always keep the below document around for such cases :) http://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-151m-t/116467-technote-ios-00.html