4

Our office is running a Cisco Meraki Z1 Security Appliance and recently (March 5th) the internet has been dropping out intermittently. The appliance remains online but the internet either doesn't work or is unbearably slow. It begins when people start showing up to the office, but then speeds will pick up an hour or so later, but then they may drop again another hour later.

Here is the uplink details for the network currently:

uplink

The Z1 datasheet says it supports up to 5 clients. this seems quite low. Our office averages 80-100 clients. In this case, the fact we would have this device is absurd. Unfortunately the guy who set up the Meraki Z1 is no longer with the company, so I can't ask him what his intentions were by buying a device that is so limited.

Here is the traffic analysis from today:

traffic analysis

As you can see, around 9:00 when people arrive at the office, the data loss sky rockets. This makes complete sense to me seeing how we have about 100 clients running on a device which only supports 5 client devices. My question is, why is this only happening now? This device was installed nearly 3 years ago and this has never happened before until now.

No configuration for the device has changed, and we have done the following troubleshooting:

  • Checked power cable to ensure sufficient power supply
  • Checked data cables to ensure they are healthy and working
  • Reset the Z1 device to factory settings
  • Connected to ISP modem and ensure a healthy connection is present
  • Rolled back firmware to see if problem persists, it did.

I am looking at buying either the MX64 (50 clients) or MX84 (200 clients) but first I am asking the question, why is this problem only happening now after using this device for 3 years?

Update:

I should also mention that the appliances received a firmware updated on March 3rd, however Cisco support team has said that this is not the cause of the problem. We rolled back the firmware to test, and the problem seemed to persist.

  • That is a cloud managed device, and it may have been updated from the cloud just before the problem started. – Ron Maupin Mar 7 '18 at 22:18
  • @RonMaupin Updated the posting. The appliances received a firmware updated on March 3rd, however Cisco support team has said that this is not the cause of the problem and that the firmware update is stable. We rolled back the firmware to test, and the problem seemed to persist. – coopwatts Mar 7 '18 at 22:21
  • The device is designed for remote employees ("teleworker"). It doesn't remotely have the power to deal with a 100 worker office. As to "why now", in every case I've dealt with, there's something that changed in the network. Something you don't know of, or are ignoring/dismissing. – Ricky Beam Mar 7 '18 at 22:53
  • @RickyBeam Thanks for the insight. The thing I could be dismissing is that recently we have had about 5-7 new employees join the office which would result in an average of 15 additional devices connecting. Cisco also said the CPU is operating extremely high. As for the configuration side, nothing has changed- do you know of a way to check error logs etc for these teleworker gateways? – coopwatts Mar 7 '18 at 22:59
  • I think you're making a (understandable) mistake in saying that this problem occurs during the day. Your users have become inured to the performance of this device, which is not good, even at night. The packet loss you have to Google's DNS servers should be a flat line at zero. The occasional blip (one, or at most two in a day) may be understandable, but not this graph. The latency also seems a bit high. It looks like the low value there is 50ms, and it reaches towards 1s during the day. I'd be amazed if you saw that on a standalone computer connected directly to your ISP, but test it. – Slartibartfast Mar 27 '18 at 5:41
1

Having used undersized routers for a while myself, I found that the most precious resource in an access router (besides bandwidth) is free memory/free space in the connection tables.

Monitor the current number of connections for TCP and UDP, the amount of free memory and how they relate to the loss ratio. My bet is that once you cross a certain boundary, packet loss shoots up. Additionally, a simple traceroute should be able the verify if the Z1 is the source of the drops.

You can somewhat soften the problem by decreasing UDP and TCP aging periods - according to your workload - but for a true solution you'd need more powerful hardware.

0

copy running-config startup-config could be your culprit.. I've seen lot's of tiny config changes go unsaved and rear their ugly heads years later after a reboot or a "I just need to move this wire for a second..."

Here are my top guesses in order of probability.

Not the Z1 - I see it get blamed all of the time for problems caused by operator. And I have seen 50 user offices use it so there is no set limitation, only a recommendation.

Duplex mismatch - Auto + Full = big problems.

Local Routing. If you made a recent routing change things might be bouncing off the wrong router or the Merkai.

Heat. Stuff gets old, dust and bugs get in, glue and solder crack. Open it up and check your heatsinks, and check for anything that feels over 130F, because that's where things start throttling CPU.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.