I've been dealing with an intermittent issue in my network for several weeks now.
Symptoms:
Periodic "Destination host unreachable.", and similar errors when pinging. The following is indicative:
Pinging 10.0.64.16 with 32 bytes of data:
Reply from 172.31.0.18: Destination host unreachable.
Reply from 172.31.0.18: Destination host unreachable.
Reply from 10.0.64.16: bytes=32 time=1004ms TTL=62
Reply from 10.0.64.16: bytes=32 time<1ms TTL=62Our JavaEE server periodically looses connection to our SQL Server (Microsoft).
An internal piece of software written on Microsoft Access, and backed by the SQL Server, periodically experience significant performance degradation.
Our virtual machines experience periodic latency spikes when accessing their storage, over iSCSI.
Various latency sensitive tools get disconnected (SSH, XenCenter), while latency insensitive tools experience no issues (web based dashboards).
Oddly enough, it doesn't appear that we experience all of the above at the same time, it's more like a rolling issue.
We have a Netgear XS728T at the core of our network. It handles inter-VLAN routing. We have 3 racks, each of which has a stack of 2 Netgear S3300 switches, which connect to the XS728T using LCAP enabled LAGs of 10G.
Users connect through stacks of Netgear S3300 switches, connected to the core using LACP enabled LAGs of 10G.
We have tried enabling Flow Control between the stacks at the top of the racks and the core. When we enabled this, we did start seeing pause frames being exchanged, but the situation didn't appear to improve.
We have tried doubling the number of 10G links in the connections between the rack stacks and the core (from 2x10G to 4x10G). This also didn't appear to improve the situation.
We have been reviewing the switch logs regularly, and are not seeing anything out of the ordinary.
The network is free of loops, except in the stacking, STP is disabled. We enabled it briefly, and it did not disable any links.
I feel like we have one or more switches with an MAC table issue.
Any suggestions on ways to collect additional information? Any troubleshooting suggestions?
Thank you,
Dominic Hilsbos
ping
.