1

I've been dealing with an intermittent issue in my network for several weeks now.

Symptoms:

  1. Periodic "Destination host unreachable.", and similar errors when pinging. The following is indicative:
    Pinging 10.0.64.16 with 32 bytes of data:
    Reply from 172.31.0.18: Destination host unreachable.
    Reply from 172.31.0.18: Destination host unreachable.
    Reply from 10.0.64.16: bytes=32 time=1004ms TTL=62
    Reply from 10.0.64.16: bytes=32 time<1ms TTL=62

  2. Our JavaEE server periodically looses connection to our SQL Server (Microsoft).

  3. An internal piece of software written on Microsoft Access, and backed by the SQL Server, periodically experience significant performance degradation.

  4. Our virtual machines experience periodic latency spikes when accessing their storage, over iSCSI.

  5. Various latency sensitive tools get disconnected (SSH, XenCenter), while latency insensitive tools experience no issues (web based dashboards).

Oddly enough, it doesn't appear that we experience all of the above at the same time, it's more like a rolling issue.

We have a Netgear XS728T at the core of our network. It handles inter-VLAN routing. We have 3 racks, each of which has a stack of 2 Netgear S3300 switches, which connect to the XS728T using LCAP enabled LAGs of 10G.

Users connect through stacks of Netgear S3300 switches, connected to the core using LACP enabled LAGs of 10G.

We have tried enabling Flow Control between the stacks at the top of the racks and the core. When we enabled this, we did start seeing pause frames being exchanged, but the situation didn't appear to improve.

We have tried doubling the number of 10G links in the connections between the rack stacks and the core (from 2x10G to 4x10G). This also didn't appear to improve the situation.

We have been reviewing the switch logs regularly, and are not seeing anything out of the ordinary.

The network is free of loops, except in the stacking, STP is disabled. We enabled it briefly, and it did not disable any links.

I feel like we have one or more switches with an MAC table issue.

Any suggestions on ways to collect additional information? Any troubleshooting suggestions?

Thank you,

Dominic Hilsbos

Improve this question
15
  • If it's a ARP cache issue, it will be on the XS728T. Turn STP back on and leave it on. It's good protection. It's conceivable that you have an intermittent STP loop.
    – Ron Trunk
    Sep 9, 2020 at 17:04
  • Layer-2 switches do not use ARP, only layer-3 devices, e.g. routers and hosts, use ARP.
    – Ron Maupin ♦
    Sep 9, 2020 at 17:11
  • Graph discards and errors on all ports using SNMP and your preferred polling & graphing software. Check if incidents of errors correspond to observations of bad behavior. Use PingPlotter or similar to get a time-line of the observations you're currently doing in an adhoc manner with ping.
    – Jeff Wheeler
    Sep 9, 2020 at 17:39
  • 1
    Can you edit your question to include the switch configuration(s)? Perhaps there is a clue there.
    – Ron Trunk
    Sep 9, 2020 at 23:41
  • 1
    @DominicHilsbos 20K file? Now I'm even more suspicious of the configuration. You obviously have other features enabled besides LAGs. It's going to come down to three things: A faulty device, a bug, or a misconfiguration. Maybe you could include a representative sample of the configuration.
    – Ron Trunk
    Sep 10, 2020 at 11:45

2 Answers 2

Reset to default
1

Turn STP back on and leave it on. It's conceivable that you have an intermittent loop. Besides, it's good insurance.

If you have an ARP cache issue, you will only see it on the XS728T, as it's the only device doing routing.

See if your logs will show MAC address changes. Something may be flapping between ports.

Improve this answer
3
  • Ok, why / how would I be seeing an intermittent loop?
    – Dominic Hilsbos
    Sep 9, 2020 at 17:27
  • Perhaps one of your workstations is bridging intermittently.
    – Ron Trunk
    Sep 9, 2020 at 17:30
  • An intermittent loop causes high broadcast rates and generally higher port utilization (when uninterrupted to a point that reasonable traffic fails).
    – Zac67 ♦
    Sep 9, 2020 at 18:58
2

It's been a while since I participated in this question.

I believe I located 90% of our problem; we did indeed have a loop. A member of IT had plugged in an additional cable between a switch and a wall port, thus generating a loop.

Since the new link was only gigabit, as compared to the LAGged 10G that is common for inter-switch connections in our network, the symptoms I expected to see weren't present.

Improve this answer
2
  • You should accept your answer as correct so it won't keep popping up in the list looking for an answer. Glad you solved it.
    – Ron Trunk
    Sep 17, 2020 at 17:30
  • @RonTrunk I accepted your answer, is that not sufficient?
    – Dominic Hilsbos
    Sep 17, 2020 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.