Why we get high fluctuation on bandwidth measurement Cacti graph?

Question

We were on redundancy test of Etherchannel and Routing on our network. During this intervention we made some measurement. Our monitoring tool is Cacti for graph. The equipment monitored is a 4500-X on VSS. Each link is on a different physical chassis.

Schema :

Test chronology :
[t0] Link on te1/1/14 port was physically removed. The Te2/1/14 is active. Po1 is operational.
[t0+15] Link on Te1/1/14 port returned to service and checked that the port back in etherchannel Po1
[t0+20] Link on te1/1/14 port was physically removed. The Te2/1/14 is active. Po1 is operational.
[t0+35] Link on Te1/1/14 port returned to service and checked that the port back in etherchannel Po1

In our tests, we monitored traffic etherchannel Po1 through Cacti (graph below) and noticed a significant change in the value of the flow when we disabled the te1/1/14 link (link te2/1/14 assets) rather stable during the reverse. We checked too the counters on int Po1 and these were maintained fairly stable.

Two interface of 10G are bundled on Etherchannels with LACP configured. Inside the etherchannel their is 2 vlans. One for Multicast traffic and another for Internet/All Traffic.

Do you know a possible cause of this behavior ?

Answer 1

To extend ytti's comment.

Your poll interval seems really small, every 10 seconds if I'm reading right. There's a few reasons you could get that result.

Equipment side:

• Bad choice of counters, if you're using 32-bit counters they could be rolling over every ~3.4 seconds if you're running a 10g interface at line rate
• Counter updating, many larger devices only update counters two or three times a minute, and they can never be relied on to be in sync. Every 30 seconds is as low as I'd bother polling, and even then I'd always want at least two points before triggering any alert or taking action
• There can be a gotcha as packets sent for CPU processing (netflow perhaps) may be counted straight away vs those not going to RE being batched (have seen this on Juniper MX)

Poller side:

• Is the poller polling accurately at the interval, and if not is it injecting its result with the actual polling time (eg, x bits in y.z seconds) so a sensible rate can be calculated
• What happens when counters reset, or SNMP GET's aren't responded to, different tools respond to these in different ways

Answer 2

Your problem is as such, that your router sampling and your own polling are not hitting the same moment. That is, even though polling interval is static, polling intervals contain different amount of samples, which your math does not take into account.
Consider you've polled t1,t2,t3 but router didn't sample anything at t1,t2 interval, so all of the traffic between t1,t3 ended up at t2,t3 polled value. Causing your rate to be 0 at t1,t2 and over linerate at t2,t3

Now I'm going to suggest one solution, but please verify this with someone who has cursory understanding of math.

First figure out interface you're interested in (if ge-1/1/1):

snmpbulkwalk SWITCH ifDescr|grep ge-1/1/1

Then you'll see its ifIndex number, let's assume it is '42'.

Then do something like:

while true; do
  snmpbulkwalk SWITCH ifHCInOctets.42 >> DATA
  date >> DATA
  sleep 1
done

Now analyze results to determine how often on average the counters are actually being updated. (I can produce script for the analysis if needed)

Then comes the part where we'd need math, but I'll suggest one naive solution.

If your update interval is 10s, poll box every 5s, i.e. twice as often as it is updated. Then your samples would be

t0, t5, t10, t15, t20, t25, t30

Now this would be your raw data, which you wouldn't use, but you'd rather recover actual samples from it like this

s1 = (t0+t5+t10)/3
s2 = (t10+t15+t20)/3
s3 = (t20+t25+t30)/3

Rationale here is, that we want to leak over the borders to reduce the effect of inaccurate polling intervals at your switch.

You'd then plot the s1, s2, s3 and you should have much more smooth/accurate result than what you are seeing now.

However I'm sure this is not novel problem and I'm sure there is formal solution how to recover optimal accuracy, unfortunately producing that solution is out of my skill set. Something math.stackexchange people would be better equipped to tackle.

Answer 3

Since you are polling at the same rate as the counters are updated you are likely out of sync.

By configuring

snmp-server hc poll <<hundredths of a second>>

you can reduce the interval in which the SNMP counters are updated to something like 1 second. This should result in a more accurate value for the throughput when you are polling every 10 seconds.

FYI, this is a hidden command.