5

If I poll over SNMP for example Cisco IOS interface ifHCInOctets counter and last reading is lower than previous reading, then I know that either the device has reloaded, ifHCInOctets counter has wrapped, there was an online hardware insertion/removal(OIR) which affected this particular interface or interface was deleted and recreated(this is possible in case of VLAN interface, Port-Channel interface, etc). Now I would like to distinguish between router reload and all those other possibilities for ifHCInOctets to start from zero. At first snmpEngineTime(range 0 - 2147483647 according to Cisco SNMP object navigator) seemed to be a perfect solution as this counter wraps after 68 years, but it also starts from zero if SNMP agent is restarted, i.e. stopped(no snmp-server) and started(snmp-server community public RO). This means that one still needs to check sysUpTime, which as far as I know, starts from zero only in case system is restarted, but unfortunately wraps after every 497 days. This means that simple algorithm seen below would not work if sysUpTime wraps between the same checks when ifHCInOctets becomes zero:

if (( prev_ifHCInOctets > cur_ifHCInOctets )); then
  if (( prev_sysUpTime > cur_sysUpTime )); then
    echo "router reloaded"
  else
    echo "counter wrapped, OIR or interface recreated"
  fi
fi

It would be perfection itself if there is a "sysUpHCTime" counter, but looks like there is not. What options do I have? I guess one possibility is simply to ignore this highly unlikely situation where both cur_ifHCInOctets(current reading of ifHCInOctets counter) and cur_sysUpTime(current reading of sysUpTime counter) are smaller than previous readings because both counters wrapped within the same polling interval. However, just out of interest, what would be the options here? I guess at least one possible option is not to check if prev_sysUpTime > cur_sysUpTime, but to check if delta between prev_sysUpTime and cur_sysUpTime is roughly equivalent to script check interval? I mean for example let's imagine a situation where prev_sysUpTime variable was 42949500 and script knows that it got this value 300 seconds ago. Now the cur_sysUpTime read by script is 128. As a next step script checks if cur_sysUpTime+(42949672-prev_sysUpTime) is around 300(for example within range 295 - 305) and if it is, then it is 100% sure that sysUpTime started from zero because of counter wrap and not because device reload. 42949672 used in this formula is the maximum value of SNMP sysUpTime counter if milliseconds are not included, i.e. maximum value of SNMP sysUpTime is 2^32, but last two digits represent milliseconds so for example 4294967296 is 42949672 seconds(about 497 days) and 96 milliseconds.

Sorry for the long post and please let me know if anything is unclear.

  • 2
    If you are going to trust the hardware data (to the extent that you want to know about reloads) why not ask it to send you coldstart/warmstart traps? – Jeff Meden Jun 12 '15 at 12:41
  • Thanks! Unfortunately this is not an option in this case because devices are not under my management and all I can use are SNMP GET requests. In addition, downside of SNMP traps is more complicated script as it needs to run as daemon and listen for SNMP traps. – Martin Jun 12 '15 at 13:43
2
+50

I would approach it this way: fetch sysuptime, then calculate the boot date/time and the predicted next wrap date/time. Write both to the log. Calculate the next poll time and if it is after the predicted next wrap date/time by a little margin, write a 'wrap expected' bit to the log. Next time you fetch, look at the 'wrap expected' bit and the predicted next wrap time from the log, and if the bit is 1 and if the predicted wrap time and current boot times are pretty close (extremely close if your server and router are both using NTP) then you know it has wrapped and not rebooted. If not, you know it rebooted. If the wrap expected bit wasn't set, simply go back to the main script logic and calculate the new boot time and predicted wrap time, make your wrap prediction bit, and write it all to the log.

You are trying to guard against a reboot false positive by not anticipating the uptime wrap, AND a reboot true negative by assuming the counter wrapped when it actually rebooted. To do both you need pretty careful timing, and even then it just reduces the probability (down past 1 in 1,000,000), but it doesn't eliminate them.

If you want to go full tilt, you can do something like adding a second detection layer on top. For example look at the UDP traffic counter: since you are polling via snmp you will be constantly incrementing it a tiny bit. Since there probably arent many other SNMP polls taking place, it will not likely wrap very often (if at all compared to reboots for other reasons) so if you looked at sysuptime going down AND udp traffic count going down you can increase the confidence that you caught a reboot.

| improve this answer | |
  • As I understand your approach, the "wrapped" bit is stored in order to detect this particular polling interval when sysUpTime is expected to wrap? In addition, please explain the "if the predicted reboot time and current boot times are pretty close then you know it(sysUpTime) has wrapped" part. I mean what is this predicted reboot time? And by current boot time you mean the device boot date/time calculated earlier, e.g. device_uptime=$(snmpget -Ov -v 2c -c public device sysUpTime.0 | gsed -r 's/^.*\((.*)\).*$/\1/;s/.{2}$//'); cur_time=$(date +%s); boot_time=$((cur_time - device_uptime))? – Martin Jun 17 '15 at 14:22
  • 1
    My wording was inelegant, I fixed it to hopefully make more sense. Using the wrap prediction bit is probably unnecessary since if you have the next wrap prediction time in the log anyway you might as well just compare it to the current boot calculation. And yes, the boot time would be the date/time of the system boot as suggested by sysuptime (either the actual boot time or the time that sysuptime wrapped to 0). – Jeff Meden Jun 17 '15 at 14:36
  • I don't see a connection between predicted sysUpTime counter wrap time and current system boot time. For example let's say that (( wrap_expected_bit == 1 )) test is true and thus script checks if predicted wrap time is let's say +/-5 seconds from system boot time: (( next_wrap_time >= (boot_time-5) && next_wrap_time <= (boot_time+5) )). Router is not reloaded after setting the wrap_expected_bit to 1 and has uptime 5057293. boot_time variable has value 1429560604(cur_time - device_uptime) and next_wrap_time would be 1472510276. next_wrap_time is always system boot time + 42949672. – Martin Jun 18 '15 at 9:15
  • 1
    Making the connection is where storing the prediction for the next check is necessary. If you record the predicted wrap time (sysuptime_max-sysuptime)+now as a date, which for example evaluates "9:50:10AM June 20 2015", and then read this in on the next run. So, when you run next and read the prediction in, read the sysuptime, and you compare those two. You will take the evaulated boot time (now-sysuptime) so for example "9:50:09AM June 20 2015" and compare it to what you logged on the last run, and see that they are close enough to say with reasonable certainty that the counter wrapped. – Jeff Meden Jun 18 '15 at 12:42
  • 1
    Yes, that is a good way of reducing the formula to its most essential elements. I usually lean toward converting to human-readable formats early on to help with debugging and operation, since the computer is more than powerful enough to do a few extra things to make it easy on us. – Jeff Meden Jun 18 '15 at 20:30
2

What you want is the SNMP equiv of: System restarted at 15:12:52 EDT Wed Jun 3 2015 from show hardware. However, I am unaware of any MIB exporting that.

Another option would be tracking the syslog message counts. I don't know which routers actually support that; my 3745 running 12.4T doesn't.

| improve this answer | |
  • Yes, timestamp of last boot would be perfect, but as much as I searched, there is no such SNMP object. However, what's your opinion on not to check if prev_sysUpTime > cur_sysUpTime, but to check if delta between prev_sysUpTime and cur_sysUpTime is roughly equivalent to script check interval? Do you see any problems with this? – Martin Jun 16 '15 at 10:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.