We have an old old guy from year 2000 or so, named Cisco 2611 and he is going nuts lately.
It hangs every three days and about 10 - 20 hours. I wasn't believing my
Observium monitoring system at first, here's the diagram:
And the peak happens to be exactly when it hangs. After that I did telnet to it and ran
cisco-router#show memory summary
And here's the top portion of the output:
cisco-router#show memory summary Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 80D02AA0 15717728 7384968 8332760 8322064 8325568 I/O 1C00000 4194304 1832344 2361960 2344108 2361916
after few seconds...:
cisco-router#show memory summary Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 80D02AA0 15717728 7386864 8330864 8322064 8325568 I/O 1C00000 4194304 1832344 2361960 2344108 2361916
and after five minutes:
cisco-router#show memory free Head Total(b) Used(b) Free(b) Lowest(b) Largest(b) Processor 80D02AA0 15717728 7397072 8320656 8312776 8318048 I/O 1C00000 4194304 1832344 2361960 2344108 2361916
The CPU load is stable and is under 5% always and interfaces are very lightly loaded.
Processor memory gets eaten up like crazy. I don't exactly understand what's happening, but I see something bad is happening. It even wiped out config few times.
Did any of you have similar issues? How can I troubleshoot further and how to understand the further output of my submitted command, I think I can find a villain there?
PS. almost forgot
cisco-router#show version Cisco Internetwork Operating System Software IOS (tm) C2600 Software (C2600-IO3-M), Version 12.0(7)XK1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) TAC:Home:SW:IOS:Specials for info Copyright (c) 1986-2000 by cisco Systems, Inc. Compiled Wed 15-Mar-00 05:51 by phanguye Image text-base: 0x80008088, data-base: 0x80877DA8 ROM: System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1) ROM: C2600 Software (C2600-IO3-M), Version 12.0(7)XK1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) cisco-router uptime is 11 hours, 3 minutes System returned to ROM by power-on at 13:24:01 EET Thu Jan 27 2011 System restarted at 00:03:27 WET Wed Jun 18 2014 System image file is "flash:aaa1313.bin" cisco 2611 (MPC860) processor (revision 0x203) with 28672K/4096K bytes of memory. Processor board ID JAD04210DI1 (1380643539) M860 processor: part number 0, mask 49 Bridging software. X.25 software, Version 3.0.0. 2 Ethernet/IEEE 802.3 interface(s) 8 Low-speed serial(sync/async) network interface(s) 32K bytes of non-volatile configuration memory. 8192K bytes of processor board System flash (Read/Write) Configuration register is 0x2102
Went further and looked at:
cisco-router#show processes memory Total: 15717728, Used: 7445296, Free: 8272432 <unrelated> PID TTY Allocated Freed Holding Getbufs Retbufs Process 58 0 395438908 394183536 1268100 0 0 IP SNMP <unrelated>
Holding value is constantly growing. What's up with that? Could
Observium do something nasty to it using SNMP?