Hi Fellow Network Enthusiasts!
I'm having a hard time troubleshooting a Real Time Stocks Trading Application. Here are the Steps I've done so far:
Apparently, this is the only Real-Time Application in the company I'm working at that's experiencing slowness and everyone really likes to blame the network guy(Me) whenever an issue like that happens.
Performed Traceroute from App to DB, Client to App, DB to App, traceroute averages at 4ms to 9ms.
Monitored CPU Utilization, Memory Usage, Packet Loss and Link Utilization in Solarwinds
The Maximum Stats on SOME network devices are:
CPU Utilization: 30%
Memory Usage 73%
Packet Loss 0%
Link Utilization: 60%
This doesn't seem helpful except for baseline purposes since it looks okay, yet they're still experiencing slowness
I have also performed Sniffing using Wireshark. At first I only sniffed one device at a time and that didn't give me the details so I sniffed two devices at a time: The Switch closest to Trade Application Server and the Switch closest to the Database Server. I compared their I/O graphs and I've found out that the delay of packet transmission from App to DB is the same on both switches but the latency of DB to App is different on both switches. The switch closest to the db shows 10ms or less latency while the switch closest to the Trade server shows 100 - 254ms delay!.
One problem I'm having with wireshark though is that all our switches (except those that connects to the client's PC) perform loads balancing so it's hard to predict which packet it is. Also, the SQL and the FiX protocol is not fully supported by Wireshark yet so it's hard to check that. I'm currently just looking at the delta times. However, I don't know how to check if the high delta time is because of the Application Response time or the Delay in Network transmission. The Server guys don't know how to check the ART -_-'
So I'm currently investigating the very high delay. My problem is, the are a lot of network devices in between the App, DB and the Client's PC, around 30 devices including the 2 two firewalls. I'm planning on sniffing at least all the devices between App and DB excluding the firewalls and that would take me 6 Laptops running at the same time.
Before I perform that though, I have a few questions:
Am I doing the troubleshooting process correctly?
Are there other things I could look at/check?
Does the number of network hops increases the latency? If yes, does it vary between switches, routers and firewalls?
The vendor for the trading application recommends to implement QoS for this trading application. Is that really necessary?
Thank you in advance for all your help and sorry for the long block of text. :)