Cisco 2611 router memory use-up

Question

We have an old old guy from year 2000 or so, named Cisco 2611 and he is going nuts lately.

It hangs every three days and about 10 - 20 hours. I wasn't believing my Observium monitoring system at first, here's the diagram:

And the peak happens to be exactly when it hangs. After that I did telnet to it and ran

cisco-router#show memory summary

And here's the top portion of the output:

cisco-router#show memory summary
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  80D02AA0   15717728    7384968    8332760    8322064    8325568
      I/O   1C00000    4194304    1832344    2361960    2344108    2361916

after few seconds...:

cisco-router#show memory summary
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  80D02AA0   15717728    7386864    8330864    8322064    8325568
      I/O   1C00000    4194304    1832344    2361960    2344108    2361916

and after five minutes:

cisco-router#show memory free
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  80D02AA0   15717728    7397072    8320656    8312776    8318048
      I/O   1C00000    4194304    1832344    2361960    2344108    2361916

The CPU load is stable and is under 5% always and interfaces are very lightly loaded.

Processor memory gets eaten up like crazy. I don't exactly understand what's happening, but I see something bad is happening. It even wiped out config few times.

Did any of you have similar issues? How can I troubleshoot further and how to understand the further output of my submitted command, I think I can find a villain there?

PS. almost forgot

cisco-router#show version
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IO3-M), Version 12.0(7)XK1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC:Home:SW:IOS:Specials for info
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Wed 15-Mar-00 05:51 by phanguye
Image text-base: 0x80008088, data-base: 0x80877DA8

ROM: System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
ROM: C2600 Software (C2600-IO3-M), Version 12.0(7)XK1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

cisco-router uptime is 11 hours, 3 minutes
System returned to ROM by power-on at 13:24:01 EET Thu Jan 27 2011
System restarted at 00:03:27 WET Wed Jun 18 2014
System image file is "flash:aaa1313.bin"

cisco 2611 (MPC860) processor (revision 0x203) with 28672K/4096K bytes of memory.
Processor board ID JAD04210DI1 (1380643539)
M860 processor: part number 0, mask 49
Bridging software.
X.25 software, Version 3.0.0.
2 Ethernet/IEEE 802.3 interface(s)
8 Low-speed serial(sync/async) network interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

Update:

Went further and looked at:

cisco-router#show processes memory
Total: 15717728, Used: 7445296, Free: 8272432
<unrelated>
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
  58   0  395438908  394183536    1268100          0          0 IP SNMP
<unrelated>

The Holding value is constantly growing. What's up with that? Could Observium do something nasty to it using SNMP?

Answer 1

You're using 12.0(7)XK1 which was released somewhere between creation of the Universe and dinosaurs. I can list at least 5 or 6 bugs connected with processing of the traffic on the interfaces and SNMP or telnet/SSH access that could have catastrophic effect on the stability of the box.

I believe 12.3 is the last line supported for the old non-XM 2600, you should try to upgrade to something like that before wasting time on troubleshooting this old code.