The one-way delay problem has a lot of attention over the years. It's especially important for analysing asymmetric routes but can be important for many applications.
The basic idea is always the same: establish a reference clock (with for example NTP) and then make measurements.
Obviously the first driving factor is what accuracy you need: 10 ms is what you say in comments. NTP can deliver 1 ms in good conditions.
The critical thing you don't say is how far apart your hosts are -- if they are all on a local network your results will be vastly much better than if distributed around the internet. Hosts on a LAN sharing the same NTP master will be in much closer sync than generic NTP hosts. Hosts which are adjacent can use the same pulse-per-second signal of a master clock and be synced better than the hosts can measure. If the hosts are under your physical control but not adjacent, you can consider adding reference clocks (GPS, atomic) to them, otherwise a bit of work with careful NTP configuration goes a long way. An example idea, if practical, is sync the client to the server by NTP.
Another thing to think about is whether you need constant measuring, or just periodic measurements or averages. For some situations I've just put hourly NTP offsets into syslog and work things out manually when required. Assuming you control the application and client requests, I'd consider adding client-clock timestamp to the request, and allowing the server to calculate an estimate; for more sophistication, add the client's clock and NTP statistics (especially jitter) and allow the server to present an answer with an error estimate. Normal public clients (assuming such a thing) might not send the information and server can simply omit the calculation. (Alternatively, have some other method to inform server of the one-way lag, and use this information in much the same way.) This approach means you wouldn't have to change any of your logging infrastructure to having higher-precision timestamps.
If you only need averages, obviously statistics can smooth out your answers to much, much, better than the direct NTP accuracy.
Other things to note:
- The "One-way Active Measurement Protocol" defined in RFC 4656 is a protocol for establishing the one-way network time between servers. Open source implementations are available.
- For the wider internet, there are a number of large measurement networks. A good overview conference presentation is: "Measurement Networks and deployment status", 2015 SANOG, Herath. https://www.sanog.org/resources/sanog25/Measurement_Networks_Herath_SANOG25.pdf
- Of special note are RIPE Atlas, NLNOG Ring, Perf Sonar Here is a comparison page
- Historically RIPE ran a Test Traffic Measurement Service until 2014, with units which contained GPS receivers, you might find useful material from that.
- There is a lot of academic research on one-way measurements. Just a couple of papers which might be interesting to you: "Time Weaver: Opportunisctic One Way Delay Measurement via NTP", Durairajan et al, 2018, https://arxiv.org/pdf/1801.02123.pdf and
- If you need vastly better accuracy than NTP, consider the Precision Time Protocol, but it's a lot more effort than NTP.
- Large cloud services have pretty good clocks available for NTP eg AWS