6

We have a bunch of ISR 2800s on 12.4(25b) that basically are in production until they die. There are no plans to invest money in SmartNET or purchase replacement fans.

We are getting a ton of fan service syslog messages and it is a nuisance with the tens of thousands of messages received daily. I see you can rate limit the number of syslog messages per second but need something more along the lines of allowing one syslog message of a particular type per day.

Are there any commands on the router or other external methods that would accomplish something like that?

Apr 17 14:52:09.292: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:52:09.292: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:52:09.292: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:52:39.292: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:52:39.292: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:52:39.292: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:53:09.293: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:53:09.293: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:53:09.293: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:53:39.293: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:53:39.293: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:53:39.293: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:54:09.293: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:54:09.293: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:54:09.293: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:54:39.294: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:54:39.294: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:54:39.294: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:55:09.294: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:55:09.294: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:55:09.294: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Apr 17 14:55:39.295: %ENVMON-4-FAN_LOW_RPM: Fan 1 service recommended
Apr 17 14:55:39.295: %ENVMON-4-FAN_LOW_RPM: Fan 2 service recommended
Apr 17 14:55:39.295: %ENVMON-4-FAN_LOW_RPM: Fan 3 service recommended
Improve this question
5
  • Since you obviously don't care about these messages at all, would it be OK to filter them all or do you still want like one per day? Do you want to receive any ENVMON messages at all?
    – Daniel Dib
    Commented Apr 18, 2014 at 12:16
  • My understanding is that they do not want to completely eliminate the messages, just limit the number that is being produced to a very low number. My thought is that one a day would make them the happiest. We would still like to receive all the ENVMON messages.
    – Adam Loveless
    Commented Apr 18, 2014 at 12:26
  • Which syslog server are you using?
    – Mike Pennington
    Commented Apr 18, 2014 at 12:34
  • @Mike Pennington All the logging from our Cisco devices goes to Cisco Prime.
    – Adam Loveless
    Commented Apr 18, 2014 at 12:44
  • From what I can see you can either rate-limit it (per second) or drop it entirely. There may be a solution available via EEM or TCL but that's beyond me. Maybe Mike has an idea.
    – Daniel Dib
    Commented Apr 18, 2014 at 13:07

1 Answer 1

Reset to default
6

Summary

You should use Cisco's Embedded Syslog Manager. ESM can dynamically modify or throttle syslog messages when they are generated on the router.

ESM Demo

I built an example (see bottom of answer) of how to rate-limit configuration messages within a test time window; for the purposes of this demo, I substituted [regexp {CONFIG} $::orig_msg] instead of [regexp {FAN_LOW_RPM} $::orig_msg] so I could illustrate rate-limiting messages like %SYS-5-CONFIG_I: Configured from console by vty0.

I edited the tcl script at the bottom of the answer with [regexp {CONFIG} $::orig_msg], and tftp'd the script into flash...

DEN-EDGE-02#copy tftp://172.16.1.5/filterSyslog.tcl flash:
Destination filename [filterSyslog.tcl]?
%Warning:There is a file already existing with this name
Do you want to over write? [confirm]
Accessing tftp://172.16.1.5/filterSyslog.tcl...
Loading filterSyslog.tcl from 172.16.1.5: !
[OK - 684 bytes]

684 bytes copied in 0.104 secs (6577 bytes/sec)
DEN-EDGE-02#

Then I configured my router with the name of the script, and the syslog server's address (172.16.1.5).

logging filter flash:filterSyslog.tcl
logging trap debugging
logging host 172.16.1.5 vrf mgmtVrf filtered

Now when you go into configuration mode on the router, the syslog messages are rate-limited.

[mpenning@tsunami tftpboot]$ sudo tshark -ni eth0 udp and host 172.16.1.204
Running as user "root" and group "root". This could be dangerous.
Capturing on eth0
  3.472614 172.16.1.204 -> 172.16.1.5   Syslog 177 LOCAL7.NOTICE: 278: Apr 21 05:37:58.189
   CDT: %SYS-5-CONFIG_I: Configured from console by vty0 (172.16.1.5) - This message was
   rate-limited by ESM

How it works

The ESM script at the bottom of the answer rate-limits FAN_LOW_RPM messages. The example leverages the fact that NVMON-4-FAN_LOW_RPM messages are sent every 30 seconds. For simplicity, I use an absolute 30-second window between 23:59:30 and 23:59:59 to rate-limit the messages. This script assumes the syslogs are sent at a constant rate, and are not intermittent. In the attached script, I use timestamps in HHMMSS (24-hour) format so they would map easily to integers.

When a syslog message is ready to be sent, IOS stores it in $::orig_msg. I just built a quick series of if .. else clauses to detect whether the syslog message:

  • Matches the regular expression (in this case, FAN_LOW_RPM)
  • Occurs in the 30-second window between 23:59:30 and 23:59:59 (inclusive)

If the message contains FAN_LOW_RPM and is within the time window, the script sends the message. Other messages containing FAN_LOW_RPM messages are not sent. All other syslogs are sent (because we only want to silence the noisy messages).

FYI, for simplicity I intentionally avoided persisting timestamp values between the last NVMON-4-FAN_LOW_RPM syslog message seen, although someone could do that too.

ESM syslog rate-limit script:

Save this file in flash as flash:filterSyslog.tcl

## Filename: filterSyslog.tcl
proc forceInteger { x } {
    set count [scan $x %d%s n rest]
    if { $count <= 0 || ( $count == 2 && ![string is space $rest] ) } {
        # This is an error
        return "-1"
    }
    return $n
}
set time_start 235900
set time_end 235959
# See http://wiki.tcl.tk/498 for information about TCL's strange number-handling
set timestamp [forceInteger [clock format [clock seconds] -format %H%M%S]]

### Modify the regexp below and use any message you want to rate-limit...
if { [regexp {FAN_LOW_RPM} $::orig_msg] } {
    if {($time_start <= $timestamp) && ($timestamp <= $time_end)} {
        # Send syslog messages inside the $time_start and $time_end
        return "$::orig_msg - This message was rate-limited by ESM"
    } else {
        # Drop syslog messages outside $time_start to $time_end
        return ""
    }
} else {
    # Return all other syslog messages as usual
    return $::orig_msg
}
Improve this answer
4
  • Having a problem getting the messages to be sent within the time window. I modified the time_start to 122530 and time_end to 124459 to test while I was away at lunch. I uploaded the modified TCL script to flash and then put in "logging filter flash:filterSyslog.tcl" and "logging host 151.132.230.12 vrf nms filtered". The logging of the FAN_LOW_RPM messages has stopped and it does show the syslog from the few times I went into the running config but it does not show any FAN_LOW_RPM log messages in that 12:25:30 to 12:44:59 window. As a side note I did not see that ESM was supported by 2800s.
    – Adam Loveless
    Commented Apr 21, 2014 at 17:24
  • But it does appear to be working somewhat even though it is not listed in Feature Navigator.
    – Adam Loveless
    Commented Apr 21, 2014 at 17:25
  • I actually tested this answer on a c2821 running 12.4(21a) so I am sure it works on the c2800 series. Some questions: A) were you always using vrf-aware syslog? B) Did you have to overwrite an old copy of the tcl script? If so, did you remove the logging filter flash:filterSyslog.tcl command and add it again? IOS loads the ESM script into memory and doesn't notice when you load a newer version in flash.
    – Mike Pennington
    Commented Apr 21, 2014 at 17:30
  • Doing "B" from your previous comment cured the issue. I know we are not supposed to say "Thank you!" but Mike I really do appreciate the above and beyond you went to for this. Thanks!
    – Adam Loveless
    Commented Apr 21, 2014 at 18:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.