2

In many architectures there are load-balancing mechanisms such as ECMP routing or LAG between various interfaces. This load-balancing is usually per-flow, based on a hash of various criteria, such as source/destination address, protocol, port information, etc. What I'm wondering is, how do you troubleshoot non-deterministic routing situations like this?

For example, in a deterministic routing scenario where traffic takes only one path, there can be cases of packet loss or increased delay on a link, and it's relatively easy to identify this. If this happened in a non-deterministic routing scenario, I imagine identifying problems with a particular member interface could be difficult to discern.

Obviously, in normal situations, there would be no need for this, but I'm wondering if there are any commands on platforms like Cisco IOS (XE, XR, etc.) or JunOS, etc. which could reveal the result of the hashing algorithm, or if anyone has experience with issues like these and has a particular troubleshooting methodology for it.

Thanks

3
  • 2
    "how do you troubleshoot non-deterministic routing situations like this?" The hashing algorithm is deterministic. If it were not deterministic, you could not guarantee that every packet in a flow takes the same path.
    – Ron Maupin
    Jan 8 at 19:53
  • @RonMaupin if you don't know the algorithm, it may as well be non-deterministic! OP is asking how to see or run the algorithm
    – user253751
    Jan 8 at 21:22
  • 2
    "if you don't know the algorithm, it may as well be non-deterministic!" That is a patently false statement. You depend on a single flow following the same path, and that can only happen if the algorithm is deterministic. If it were not then any packet from any flow could follow any path, and that could be very bad. Just because you do not know the algorithm, does not mean anything. There are proprietary algorithms that are secret, yet many people depend on them. For example, Skype, many people and businesses depended on it, but it uses a proprietary algorithm, and dialing is deterministic.
    – Ron Maupin
    Jan 8 at 21:32
3

There are some things to test in certain circumstances, but the basic idea is that PAgP or LACP aggregations are between two directly-connected devices, and anything sent from one device to the other ends up on the other device, regardless of the path of the flow.

Cisco has a command to test the interface used for a port channel:

You can determine which interface in the EtherChannel forwards traffic, with the frame distribution policy as a basis. Issue the remote login switch command to log in remotely to the Switch Processor (SP) console in order to make this determination. Then, issue the test etherchannel load-balance interface port-channel number {ip | l4port | mac} [source_ip_add | source_mac_add | source_l4_port] [dest_ip_add | dest_mac_add | dest_l4_port] command.

Cisco and Juniper have various documents that go into some detail about the hashing algorithms for LACP, mostly describing which parameters are used for the algorithm, but there does not seem to be any specific testing like the above command.

2

I've been using this command on NX-OS, on a set of N3K-3164Q:

switch# show routing hash <srcIP> <dstIP> ip-proto <protocolNumber> <srcPort> <dstPort> vrf <vrfName>

It helped quite a bit to gain insight into an ECMP polarization problem.

1

For ECMP you can trace the route based on ICMP errors.

TTL is not included in the hashing algorithm, so as long as you keep the fields that are included in the hash consistent you can trace the route by sending packets with increasing TTLs and monitoring the ICMP errors that come back.

Facebook wrote a specialist traceroute tool for use in ECMP scenarios that loops over a range of source ports to exercise different routes and report packet losses for each route at each hop.

https://github.com/facebook/fbtracert

Afaict this won't work for LAGs though, because the L3 hops will be the same for all the paths.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.