1

I need to calculate the time it takes between:

  1. a request sent from a machine
  2. the request is received at another machine

I can do it by logging, but the clocks at these 2 machines are possible have a time gap, so I cannot just subtract: (Time_Received - Time_Sent) - need to pay attention to delay machines clocks one from another.

What is the best practice to calculate this? ( It's just a REST request from Spring Boot applications deployed at multiple machines, using slf4j libs stack)

  • 4
    "the clocks at these 2 machines are possible have a time gap" this is what NTP is made for. Otherwise, which precision do you need? – JFL Dec 12 '18 at 10:34
  • So, is the answer is: "Use NTP to synchronize clocks, collect logs from both machines, make a subtraction: (Time_Received - Time_Sent)", right? I need a precision within 10 milliseconds – Axel Coon Dec 12 '18 at 11:48
  • Yes, that could do it, but only if your logs have time logged precisely enough. Some logs only goes down to seconds, others to hundredth of a second. – JFL Dec 12 '18 at 12:33
  • @AxelCoon Measuring one-way-delay is hard without a side channel with well-known timing/latency properties. With a good NTP setup (end systems have proper RTCs, clocks are being disciplined by NTP, not stepped by ntpdate), you can get sync to the millisecond between the machines. But that's a bit of an art in itself. Are you certain that you cannot derive the answers to your question from time to response analysis in packet traces from both client and server side? (e.g. with Wireshark, see comment below). The difference between the values will contain exactly the network path's RTT. – Marc 'netztier' Luethi Dec 18 '18 at 10:56
  • Hi Axel and welcome to NE. How far apart are the machines? – jonathanjo Dec 18 '18 at 14:30
2

The one-way delay problem has a lot of attention over the years. It's especially important for analysing asymmetric routes but can be important for many applications.

The basic idea is always the same: establish a reference clock (with for example NTP) and then make measurements.

Obviously the first driving factor is what accuracy you need: 10 ms is what you say in comments. NTP can deliver 1 ms in good conditions.

The critical thing you don't say is how far apart your hosts are -- if they are all on a local network your results will be vastly much better than if distributed around the internet. Hosts on a LAN sharing the same NTP master will be in much closer sync than generic NTP hosts. Hosts which are adjacent can use the same pulse-per-second signal of a master clock and be synced better than the hosts can measure. If the hosts are under your physical control but not adjacent, you can consider adding reference clocks (GPS, atomic) to them, otherwise a bit of work with careful NTP configuration goes a long way. An example idea, if practical, is sync the client to the server by NTP.

Another thing to think about is whether you need constant measuring, or just periodic measurements or averages. For some situations I've just put hourly NTP offsets into syslog and work things out manually when required. Assuming you control the application and client requests, I'd consider adding client-clock timestamp to the request, and allowing the server to calculate an estimate; for more sophistication, add the client's clock and NTP statistics (especially jitter) and allow the server to present an answer with an error estimate. Normal public clients (assuming such a thing) might not send the information and server can simply omit the calculation. (Alternatively, have some other method to inform server of the one-way lag, and use this information in much the same way.) This approach means you wouldn't have to change any of your logging infrastructure to having higher-precision timestamps.

If you only need averages, obviously statistics can smooth out your answers to much, much, better than the direct NTP accuracy.

Other things to note:

  • The "One-way Active Measurement Protocol" defined in RFC 4656 is a protocol for establishing the one-way network time between servers. Open source implementations are available.
  • For the wider internet, there are a number of large measurement networks. A good overview conference presentation is: "Measurement Networks and deployment status", 2015 SANOG, Herath. https://www.sanog.org/resources/sanog25/Measurement_Networks_Herath_SANOG25.pdf
  • Of special note are RIPE Atlas, NLNOG Ring, Perf Sonar Here is a comparison page
  • Historically RIPE ran a Test Traffic Measurement Service until 2014, with units which contained GPS receivers, you might find useful material from that.
  • There is a lot of academic research on one-way measurements. Just a couple of papers which might be interesting to you: "Time Weaver: Opportunisctic One Way Delay Measurement via NTP", Durairajan et al, 2018, https://arxiv.org/pdf/1801.02123.pdf and
  • If you need vastly better accuracy than NTP, consider the Precision Time Protocol, but it's a lot more effort than NTP.
  • Large cloud services have pretty good clocks available for NTP eg AWS
1

With a RESTful interface based on HTTP/TCP, you could use a packet tracer and stop the time between the request being sent out and the corresponding TCP ACK. This of course includes the time the ACK requires to travel back, but it'd give a precise figure - and often, the round-trip latency is a more useful figure than a one-way delay.

Another way is to use a second connection with a low-overhead protocol (UDP) and send time stamped IDs back and forth to figure out the time gap.

edit to comment:

Note that even if you synchronize both machine clocks using NTP or similar, the protocol needs to take the RTT into account - so the "synchronized" time precision depends on it.

The only physically precise method is to have an known-latency path between the machines (or to a third one taking the measurement, e.g. using packet capturing) and use that as a reference. However, this isn't usually practical.

  • Thank you, but I need exactly a one-way delay. – Axel Coon Dec 18 '18 at 7:59
  • One step further: Wireshark can report and graph http.time (see osqa-ask.wireshark.org/questions/46537/… ) when analyzing HTTP-ish flows. By capturing the same request(s) on both initiator and responder, you get pretty clear answers. They show intake-processing-output delay of the server, while at the other end, you can collect the request-output-waiting-receiving delay as experienced by the initiator ("client"). The difference between the values from either end will show path RTT pretty well. But the OP did ask explicitely for one way latency, so... – Marc 'netztier' Luethi Dec 18 '18 at 10:38
  • @Marc'netztier'Luethi You still need to synchronize the clocks which was the start problem... – Zac67 Dec 18 '18 at 19:22
  • @Zac67 http.time is reported as (relative) "delta time since request", as observed by the capturing system. Assuming the capturing system's local clock can count milliseconds halfway decently, sync is only needed if particular requests must be singled out and analyzed, and if the only available identifier is the given frame's capture timestamp (in absolute time). Quite a few upper layer protocols expose transaction or request IDs which are understood by Wireshark and can be reported and graphed based on local delta times. With SSL enabled protocols, well... Things become difficult. – Marc 'netztier' Luethi Dec 18 '18 at 21:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.