2

We have a proplem with server-to-server latency. How can I determine the reason for it? These servers are connected to different Cisco switches. Althought the reason for this problem might not be network related. How can I check that?

topology is: VMWare virtual server 1 -> ESXi -> IBM BladeCenter Switch -> Cisco Nexus9k -> another IBM BladeCenter Switch -> ESXi -> VMWare virtual server 2

closed as too broad by Ron Maupin Jul 26 '18 at 2:35

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    A basic test is traceroute that may (or not) give you a hint, otherwise, performing packet capture (wireshark, tcpdump..) on each nodes should give you better clue. – JFL Mar 23 '17 at 8:58
  • 1
    I would suggest that if you can connect the servers back-to-back and make exact same test and each time add the other network elements - then it should be your next option to troubleshoot (after traceroute). – alexlev2004 Mar 23 '17 at 9:23
  • 1
    There are things like IP SLA that can be run on your network devices to give you statistics about the network. – Ron Maupin Mar 23 '17 at 12:56
  • 1
    Do those Bladecenter switches connect to the Nexus as a Fex? Or they are independent switches? – John K. Mar 23 '17 at 19:43
  • "Do those Bladecenter switches connect to the Nexus as a Fex?" Yes, those Bladecenter connect to FEX and FEX connect to Nexus – Vladimir Fomin Mar 29 '17 at 6:52
3

First off I'd want to identify exactly how the latency in question is being identified. Is it a particular application running slowly or a wide variety of traffic that can be specifically observed with either long transit times or loss?

If it is something specific to one type of application traffic and it isn't the actual application (a very common theme, unfortunately) then I'd want to understand the nature of the traffic that was slow and try to see if I could isolate it. Is it high-bandwidth / large packet that's causing the issue? Lots of small packets?

Finally, some basic isolation might shed some light - can you send the same type of traffic experiencing high latency between two hosts on the same vSwitch (i.e. exclude the physical switch)? Between hosts on two different hypervisors on the same blade switch (exclude the Nexus and its connections)?

The other common causes should also be ruled out - taking errors on any physical interfaces (in either direction), weird STP/L2 configuration problems (incredibly common on blade switch configurations, unfortunately), tell-tale error messages, counters for packet drops on anything in line, etc.

2

Latency issues might not be specific enough. You can certainly get round trip (millisecond) response times using basic ICMP tools like ping and traceroute. However, poor application performance / response times could be rooted in higher layer issues. If the network looks good via ICMP, try looking at your application logs and/or debugging tools for a clue.

1

A packet capture at the 9k should do the trick. You can see when each packet comes in and heads out, which will tell you the latency. I would suggest one on the interface where "Virtual Server 1" comes in and one on the interface where "Virtual Server 2" comes in. This will give you a good idea of how long the packets spend in your switch. Odds are the answer will be "essentially zero time."

A capture at each of the IBM BladeCenter switches would be good as well, the interfaces leading to your VMWare cluster. Same idea, compare one to the other. It should at least tell you where the delay isn't. ie, if all the packets on the left-side switch enter and leave within 2-3ms of each other, that server's not the problem.

Finally, and this is where I'd start, a capture at each server may be your best and simplest option (others will disgree, but I find captures from Cisco switches to be laborious). If your server receives a packet and doesn't respond for several seconds, guess what? You've found your problem!

  • Did I understand correctly, you suggest me connect two laptop with Wireshark in Nexus9k and mirror traffic to those laptops from virtual servers interfaces? How can I determine the latency? – Vladimir Fomin Mar 29 '17 at 6:59
  • 1
    Two laptops or (usually sufficient) one laptop and test it twice, one from each end. You can tell latency by comparing timestamps (how tightly you can get the laptops synched will also determine whether one or two). If the switch is causing latency, you will see it when testing on each side. If only one server is causing delay, you will see it when testing kibosh one side. – Nanban Jim Mar 29 '17 at 7:28

Not the answer you're looking for? Browse other questions tagged or ask your own question.