We have a proplem with server-to-server latency. How can I determine the reason for it? These servers are connected to different Cisco switches. Althought the reason for this problem might not be network related. How can I check that?
topology is: VMWare virtual server 1 -> ESXi -> IBM BladeCenter Switch -> Cisco Nexus9k -> another IBM BladeCenter Switch -> ESXi -> VMWare virtual server 2
First off I'd want to identify exactly how the latency in question is being identified. Is it a particular application running slowly or a wide variety of traffic that can be specifically observed with either long transit times or loss?
If it is something specific to one type of application traffic and it isn't the actual application (a very common theme, unfortunately) then I'd want to understand the nature of the traffic that was slow and try to see if I could isolate it. Is it high-bandwidth / large packet that's causing the issue? Lots of small packets?
Finally, some basic isolation might shed some light - can you send the same type of traffic experiencing high latency between two hosts on the same vSwitch (i.e. exclude the physical switch)? Between hosts on two different hypervisors on the same blade switch (exclude the Nexus and its connections)?
The other common causes should also be ruled out - taking errors on any physical interfaces (in either direction), weird STP/L2 configuration problems (incredibly common on blade switch configurations, unfortunately), tell-tale error messages, counters for packet drops on anything in line, etc.
Latency issues might not be specific enough. You can certainly get round trip (millisecond) response times using basic ICMP tools like ping and traceroute. However, poor application performance / response times could be rooted in higher layer issues. If the network looks good via ICMP, try looking at your application logs and/or debugging tools for a clue.
A packet capture at the 9k should do the trick. You can see when each packet comes in and heads out, which will tell you the latency. I would suggest one on the interface where "Virtual Server 1" comes in and one on the interface where "Virtual Server 2" comes in. This will give you a good idea of how long the packets spend in your switch. Odds are the answer will be "essentially zero time."
A capture at each of the IBM BladeCenter switches would be good as well, the interfaces leading to your VMWare cluster. Same idea, compare one to the other. It should at least tell you where the delay isn't. ie, if all the packets on the left-side switch enter and leave within 2-3ms of each other, that server's not the problem.
Finally, and this is where I'd start, a capture at each server may be your best and simplest option (others will disgree, but I find captures from Cisco switches to be laborious). If your server receives a packet and doesn't respond for several seconds, guess what? You've found your problem!
