How to Troubleshoot Network Latency Step by Step

Question

Does anyone have an idea of how to troubleshoot high network latency step by step?

For example, let's say we have a remote office. And all users are from that office are having high latency accessing a file server from another remote office.

Here's how I would start troubleshooting this issue.

1. Check the physical connections from access switch all the way to the border routers and ISP equipment and beyond. If there's no errors, proceed with number 2.
2. Check involved devices if they have high CPU utilization. If this is not the case, proceed with number 3.
3. Check if the circuit is not over utilized. If this is true, check who are the top talkers.
4. If there's top talkers, check what applications they are using while high latency is happening.
5. Ask top talkers if they can access the applications during off hours.

Thanks

Answer 1

Very likely the WAN connection is the major problem. There's little you can do except provide a local server or change the ISP. Additionally, the possibly weak WAN link may be congested and may need an upgrade. Also, the VPN router make be too weak (I've seen Gbit WAN links with a VPN router barely able to handle 20 Mbit/s).

In the local network, make sure routing is working correctly. While ping and traceroute (by IP) are no exact tools they can usually provide a good starting point for further research.

If you've got redundant WAN/VPN links you'll want to watch out for asymmetric routes - replies coming from another link that the requests went out of. If you use asymmetric routes you need to make sure your network, devices and policies can handle them.

Next, make sure DNS is working AOK. With Windows servers you'll need to use Windows DNS or put all the required AD records on the DNS server used locally. Usually, an ISP DNS server is not a good choice. Name resolution over broadcast does not work across VPN locations.

If everything fails you'll need to run a packet trace and watch a problematic connection in detail. This requires significant insight into the protocols used and you may need to hire a consultant.

For more detailed suggestions you'd need to provide more details on your network: a diagram, (sanitized) configurations of all relevant devices and a more exact description of your problem.