High Jitter and Data Loss on Cisco Meraki Z1

Question

Our office is running a Cisco Meraki Z1 Security Appliance and recently (March 5th) the internet has been dropping out intermittently. The appliance remains online but the internet either doesn't work or is unbearably slow. It begins when people start showing up to the office, but then speeds will pick up an hour or so later, but then they may drop again another hour later.

Here is the uplink details for the network currently:

The Z1 datasheet says it supports up to 5 clients. this seems quite low. Our office averages 80-100 clients. In this case, the fact we would have this device is absurd. Unfortunately the guy who set up the Meraki Z1 is no longer with the company, so I can't ask him what his intentions were by buying a device that is so limited.

Here is the traffic analysis from today:

As you can see, around 9:00 when people arrive at the office, the data loss sky rockets. This makes complete sense to me seeing how we have about 100 clients running on a device which only supports 5 client devices. My question is, why is this only happening now? This device was installed nearly 3 years ago and this has never happened before until now.

No configuration for the device has changed, and we have done the following troubleshooting:

• Checked power cable to ensure sufficient power supply
• Checked data cables to ensure they are healthy and working
• Reset the Z1 device to factory settings
• Connected to ISP modem and ensure a healthy connection is present
• Rolled back firmware to see if problem persists, it did.

I am looking at buying either the MX64 (50 clients) or MX84 (200 clients) but first I am asking the question, why is this problem only happening now after using this device for 3 years?

Update:

I should also mention that the appliances received a firmware updated on March 3rd, however Cisco support team has said that this is not the cause of the problem. We rolled back the firmware to test, and the problem seemed to persist.

Answer 1

Having used undersized routers for a while myself, I found that the most precious resource in an access router (besides bandwidth) is free memory/free space in the connection tables.

Monitor the current number of connections for TCP and UDP, the amount of free memory and how they relate to the loss ratio. My bet is that once you cross a certain boundary, packet loss shoots up. Additionally, a simple traceroute should be able the verify if the Z1 is the source of the drops.

You can somewhat soften the problem by decreasing UDP and TCP aging periods - according to your workload - but for a true solution you'd need more powerful hardware.

Answer 2

copy running-config startup-config could be your culprit.. I've seen lot's of tiny config changes go unsaved and rear their ugly heads years later after a reboot or a "I just need to move this wire for a second..."

Here are my top guesses in order of probability.

Not the Z1 - I see it get blamed all of the time for problems caused by operator. And I have seen 50 user offices use it so there is no set limitation, only a recommendation.

Duplex mismatch - Auto + Full = big problems.

Local Routing. If you made a recent routing change things might be bouncing off the wrong router or the Merkai.

Heat. Stuff gets old, dust and bugs get in, glue and solder crack. Open it up and check your heatsinks, and check for anything that feels over 130F, because that's where things start throttling CPU.