IOS Memory allocation error after ping

Question

I have written a script in Tera Term for a Cisco IR809g which pings 3 different routers(IR 809g) in 2 different networks over the cellular interface. After +- 6 hours I'm getting a Memory allocation failure. And all the pings start to fail.

ping X.X.X.X Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to X.X.X.X, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 38/54/96 ms
ping X.X.X.X Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to X.X.X.X, timeout is 2 seconds: !!!!! Success rate is 100 percent (5/5), round-trip min/avg/max = 266/484/846 ms

*Sep 14 13:47:25.733 CEST: %SYS-2-MALLOCFAIL: Memory allocation of 780 bytes failed from 0x1B51F8, alignment 32 Pool: I/O Free: 17616 Cause: Memory fragmentation Alternate Pool: None Free: 0 Cause: No Alternate pool -Process= "Pool Manager", ipl= 0, pid= 6 -Traceback= 12582Az 11982Dz 1B51F8z 145C52z 1458FCz

ping X.X.X.X Type escape sequence to abort. Sending 5, 100-byte ICMP Echos to X.X.X.X, timeout is 2 seconds: ..... Success rate is 0 percent (0/5)

What I've found so far is this: post. So could it be a misconfiguration of the default routes or dynamic routing protocols ?

By launching show memory I get the following result:

  ¦  Type    ¦    Head  ¦   Total(b) ¦  Used(b)  ¦ Free(b)  ¦ Lowest(b) ¦ Largest(b) ¦
  ¦ -------- ¦ -------- ¦ ---------- ¦ --------- ¦ -------- ¦ --------- ¦ ---------- ¦
  ¦ Process  ¦ EB9FDD4  ¦ 189137452  ¦ 157931336 ¦ 31206116 ¦ 31040956  ¦  30727448  ¦
  ¦     I/O  ¦ B09FDD4  ¦  61865984  ¦  61848732 ¦    17252 ¦    17252  ¦       716  ¦

What could resolve the problem ?

Answer 1

General advice, not specific to your router model:

Pinging across NAT requires a NAT entry for each Identification value in the ICMP Echo Request header - ICMP doesn't use ports like common transport-layer protocols (UDP, TCP), and that's the only grip a NAT router has on the 'connection' (see RFC 5508 for details).

Since there is no connection concept in ICMP, the router can only age those NAT entries out by time. Pinging with a fairly high frequency in combination with a high ICMP aging value on the NAT router can eventually exhaust the router's NAT table ie. memory.

You need to either

• ping without requiring NAT, from a public IP address
• replace the router with a higher capacity model
• reduce the ICMP aging value to retire NAT table entries earlier, freeing memory faster
• reduce the ping frequency, decreasing the table growth rate
• explore ways to use a limited range of Identification values in the outgoing ICMP echo requests

Answer 2

What we tried to do in the end was creating a list of all cisco routers where we noted for every one of them:

1. IOS Software version
2. System Image file used
3. FPGA Version

For those routers that had a connection problem, we found similarities in the IOS version, image file and FPGA version. In the end, what the local distributor of the cisco router found out after contacting cisco is that it is most likely due to a software bug. He wrote: Cisco TAC believes it’s related to a bug with the DMVPN that causes traffic with a corrupted header to be punted to the cellular interface instead causing some kind of crash or malfunctioning of this interface/internal modem.

Answer 3

You are running out of I/O memory. This can be tweaked with the memory-size iomem configuration command. You appear to be using ~25%, so 30 or 40 might help. With only 30M free, I wouldn't go too far with that knob.

Check what's using IO memory: show memory io allocating-process totals

(update)

Smart Init is enabled
smart init is sizing iomem
  ID            MEMORY_REQ                 TYPE
0003E9          0X00474800 C2851 Mainboard
000063          0X0003C000 32 Port Async Module
000416          0X00000000 Eight Port FXS/DID Analog Voice Module
0004C8          0X00077D00 HWIC ADSL Annex A
                0X00284860 Onboard PVDM2 SIMM
                0X00262B0A Onboard VPN
                0X000021B8 Onboard USB
                0X002C29F0 public buffer pools
                0X00211000 public particle pools
-----------------------------------------------
TOTAL:          0X00EE5412

If any of the above Memory Requirements are 
"UNKNOWN", you may be using an unsupported
configuration or there is a software problem and
system operation may be compromised.
Rounded IOMEM up to: 16Mb.
Using 1 percent iomem. [16Mb/1024Mb]

At boot, smart-init runs to set the size of iomem. This is usually a "good guess", but not always perfect.

blue-gw-1#show memory 
                Head    Total(b)     Used(b)     Free(b)   Lowest(b)  Largest(b)
Processor   49302BE0   902812704    87109336   815703368   815324372   815336332
      I/O   3F000000    16777216     8526040     8251176     7958656     8132988

blue-gw-1#show memory io allocating-process totals 

Allocator PC Summary for: I/O

    PC          Total   Count  Name
0x403D3EE0    5722496       6  IDS SM
0x403D3F98    1906240      37  Normal
0x4039954C     705312     683  *Packet Data*
0x402BDFA8      69760       2  Init
0x403353C4      24704       2  C5510 DSP
0x44BF3CB0      21568       1  sfnt udm
0x4033556C      16512       2  C5510 DSP
0x401B9494      14400       1  dslsar hwic queue
0x43C40834      11680       1  *Init*
0x400504E4       8320       2  MV64340 Ethernet Tx Ring
0x44BFC664       6624       5  sfnt kblock
0x40050460       4224       2  MV64340 Ethernet Rx Ring
0x40836088       4160       1  HDLC32_TX
0x409E63C0       3072      32  USB Startup
0x44BFC114       2624       1  sfnt udm
0x40385450       1280       1  Init
0x408360E0       1088       1  HDLC32_RX

As you can see from my router, IDS takes up a lot of memory, and I barely use any IDS features. (ipv6 inspect) Packets are only #3 in the list.