14

We were on redundancy test of Etherchannel and Routing on our network. During this intervention we made some measurement. Our monitoring tool is Cacti for graph. The equipment monitored is a 4500-X on VSS. Each link is on a different physical chassis.

Schema :

etherchannel 1

Test chronology :
[t0] Link on te1/1/14 port was physically removed. The Te2/1/14 is active. Po1 is operational.
[t0+15] Link on Te1/1/14 port returned to service and checked that the port back in etherchannel Po1
[t0+20] Link on te1/1/14 port was physically removed. The Te2/1/14 is active. Po1 is operational.
[t0+35] Link on Te1/1/14 port returned to service and checked that the port back in etherchannel Po1

In our tests, we monitored traffic etherchannel Po1 through Cacti (graph below) and noticed a significant change in the value of the flow when we disabled the te1/1/14 link (link te2/1/14 assets) rather stable during the reverse. We checked too the counters on int Po1 and these were maintained fairly stable.

Graph

Two interface of 10G are bundled on Etherchannels with LACP configured. Inside the etherchannel their is 2 vlans. One for Multicast traffic and another for Internet/All Traffic.

Do you know a possible cause of this behavior ?

7
  • How long did you took each test?
    – laf
    Jun 4, 2013 at 6:54
  • Each port disconnection take 15min as you can see on chronology.
    – cgasp
    Jun 4, 2013 at 7:01
  • What is your port-channel config and load-balance type on both sides? What can you tell us about your test suite and paramters that generated that traffic -- one flow, multiple flows, protocol, etc. Jun 4, 2013 at 7:15
  • Two interface of 10G are bundled on Etherchannels with LACP configured. Inside the etherchannel their is 2 vlans. One for Multicast traffic and another for Internet/All Traffic. Question updated.
    – cgasp
    Jun 4, 2013 at 7:22
  • The test was on a generalist redundancy test on routing protocols and etherchannels. If a link go down what happen. All test run as expected but we wonder why this behavior on measurement of bandwitdh.
    – cgasp
    Jun 4, 2013 at 7:24

3 Answers 3

11

To extend ytti's comment.

Your poll interval seems really small, every 10 seconds if I'm reading right. There's a few reasons you could get that result.

Equipment side:

  • Bad choice of counters, if you're using 32-bit counters they could be rolling over every ~3.4 seconds if you're running a 10g interface at line rate
  • Counter updating, many larger devices only update counters two or three times a minute, and they can never be relied on to be in sync. Every 30 seconds is as low as I'd bother polling, and even then I'd always want at least two points before triggering any alert or taking action
  • There can be a gotcha as packets sent for CPU processing (netflow perhaps) may be counted straight away vs those not going to RE being batched (have seen this on Juniper MX)

Poller side:

  • Is the poller polling accurately at the interval, and if not is it injecting its result with the actual polling time (eg, x bits in y.z seconds) so a sensible rate can be calculated
  • What happens when counters reset, or SNMP GET's aren't responded to, different tools respond to these in different ways
2
  • 1
    Even if you poll very accurately every N, the box may not poll HW counters at accurate intervals, making it seem like t1,t2 see no traffic increase and t2,t3 see over linerate. Now what is most accurate results you can get is maybe in realm of math.stackexchange, but I believe best you can do is 2*the_slowest_update_interval, if box updates every 10s, you could have measurement data every 20s. But probably with some statistic magic you can make it closer to the 10s (the problem here is that the update interval isn't accurately timed)
    – ytti
    Jun 4, 2013 at 9:20
  • 1
    Also, which poller you are using with Cacti matters at a 10 second polling interval. I've had bad experiences with the default poller at those low polling intervals. No mention is made if they're using Spine or the default poller. Jun 4, 2013 at 10:39
6

Your problem is as such, that your router sampling and your own polling are not hitting the same moment. That is, even though polling interval is static, polling intervals contain different amount of samples, which your math does not take into account.
Consider you've polled t1,t2,t3 but router didn't sample anything at t1,t2 interval, so all of the traffic between t1,t3 ended up at t2,t3 polled value. Causing your rate to be 0 at t1,t2 and over linerate at t2,t3

Now I'm going to suggest one solution, but please verify this with someone who has cursory understanding of math.

First figure out interface you're interested in (if ge-1/1/1):

snmpbulkwalk SWITCH ifDescr|grep ge-1/1/1

Then you'll see its ifIndex number, let's assume it is '42'.

Then do something like:

while true; do
  snmpbulkwalk SWITCH ifHCInOctets.42 >> DATA
  date >> DATA
  sleep 1
done

Now analyze results to determine how often on average the counters are actually being updated. (I can produce script for the analysis if needed)

Then comes the part where we'd need math, but I'll suggest one naive solution.

If your update interval is 10s, poll box every 5s, i.e. twice as often as it is updated. Then your samples would be

t0, t5, t10, t15, t20, t25, t30

Now this would be your raw data, which you wouldn't use, but you'd rather recover actual samples from it like this

s1 = (t0+t5+t10)/3
s2 = (t10+t15+t20)/3
s3 = (t20+t25+t30)/3

Rationale here is, that we want to leak over the borders to reduce the effect of inaccurate polling intervals at your switch.

You'd then plot the s1, s2, s3 and you should have much more smooth/accurate result than what you are seeing now.

However I'm sure this is not novel problem and I'm sure there is formal solution how to recover optimal accuracy, unfortunately producing that solution is out of my skill set. Something math.stackexchange people would be better equipped to tackle.

3

Since you are polling at the same rate as the counters are updated you are likely out of sync.

By configuring

snmp-server hc poll <<hundredths of a second>>

you can reduce the interval in which the SNMP counters are updated to something like 1 second. This should result in a more accurate value for the throughput when you are polling every 10 seconds.

FYI, this is a hidden command.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.