Take the 2-minute tour ×
Network Engineering Stack Exchange is a question and answer site for network engineers. It's 100% free, no registration required.

We have an old old guy from year 2000 or so, named Cisco 2611 and he is going nuts lately.

It hangs every three days and about 10 - 20 hours. I wasn't believing my Observium monitoring system at first, here's the diagram:

Observium - Cisco 2611 memory

And the peak happens to be exactly when it hangs. After that I did telnet to it and ran

cisco-router#show memory summary

And here's the top portion of the output:

cisco-router#show memory summary
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  80D02AA0   15717728    7384968    8332760    8322064    8325568
      I/O   1C00000    4194304    1832344    2361960    2344108    2361916

after few seconds...:

cisco-router#show memory summary
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  80D02AA0   15717728    7386864    8330864    8322064    8325568
      I/O   1C00000    4194304    1832344    2361960    2344108    2361916

and after five minutes:

cisco-router#show memory free
               Head   Total(b)    Used(b)    Free(b)  Lowest(b) Largest(b)
Processor  80D02AA0   15717728    7397072    8320656    8312776    8318048
      I/O   1C00000    4194304    1832344    2361960    2344108    2361916

The CPU load is stable and is under 5% always and interfaces are very lightly loaded.

Processor memory gets eaten up like crazy. I don't exactly understand what's happening, but I see something bad is happening. It even wiped out config few times.

Did any of you have similar issues? How can I troubleshoot further and how to understand the further output of my submitted command, I think I can find a villain there?

PS. almost forgot

cisco-router#show version
Cisco Internetwork Operating System Software
IOS (tm) C2600 Software (C2600-IO3-M), Version 12.0(7)XK1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)
TAC:Home:SW:IOS:Specials for info
Copyright (c) 1986-2000 by cisco Systems, Inc.
Compiled Wed 15-Mar-00 05:51 by phanguye
Image text-base: 0x80008088, data-base: 0x80877DA8

ROM: System Bootstrap, Version 11.3(2)XA4, RELEASE SOFTWARE (fc1)
ROM: C2600 Software (C2600-IO3-M), Version 12.0(7)XK1, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1)

cisco-router uptime is 11 hours, 3 minutes
System returned to ROM by power-on at 13:24:01 EET Thu Jan 27 2011
System restarted at 00:03:27 WET Wed Jun 18 2014
System image file is "flash:aaa1313.bin"

cisco 2611 (MPC860) processor (revision 0x203) with 28672K/4096K bytes of memory.
Processor board ID JAD04210DI1 (1380643539)
M860 processor: part number 0, mask 49
Bridging software.
X.25 software, Version 3.0.0.
2 Ethernet/IEEE 802.3 interface(s)
8 Low-speed serial(sync/async) network interface(s)
32K bytes of non-volatile configuration memory.
8192K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

Update:

Went further and looked at:

cisco-router#show processes memory
Total: 15717728, Used: 7445296, Free: 8272432
<unrelated>
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
  58   0  395438908  394183536    1268100          0          0 IP SNMP
<unrelated>

The Holding value is constantly growing. What's up with that? Could Observium do something nasty to it using SNMP?

share|improve this question
add comment

1 Answer 1

up vote 7 down vote accepted

You're using 12.0(7)XK1 which was released somewhere between creation of the Universe and dinosaurs. I can list at least 5 or 6 bugs connected with processing of the traffic on the interfaces and SNMP or telnet/SSH access that could have catastrophic effect on the stability of the box.

I believe 12.3 is the last line supported for the old non-XM 2600, you should try to upgrade to something like that before wasting time on troubleshooting this old code.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.