10

I am implementing a system where one device on a network queries at a very high frequency (hundreds or thousands of queries per second) through sending a small UDP packet with 8 or so bytes of data. This is received by another application, perhaps on another device, which does some very simple processing and sends off the few-byte-large result wrapped in another UDP packet.

I would like to know what sort of round trip times are feasible with typical hardware, where the communicating systems are perhaps connected via wired Ethernet a few meters apart, taking into account propagation and transmission delays, etc.

Other thoughts and suggestions are also welcome.

2
  • 2
    you could see latencies in tens or hundreds of microseconds... definitely sub millisecond latency... based on your description, it sounds like you are considering a financial trading system... the latencies you ask for are very dependent on your specific hw, and you are much better off conducting tests than asking for free advice Jul 5, 2013 at 23:10
  • Many thanks for your response. It is not actually for a financial trading system in this case. I just wanted a vague idea of what was possible before starting the implementation, more as a feasibility study than anything else.
    – John Smith
    Jul 6, 2013 at 7:58

2 Answers 2

11

An an example Juniper MX80 has ingress->egress delay of about 8us, on low-latency cut-through switch it can be <1us (maybe 0.7us). (Remember that cut-through switch can't do cut-through 100% of time, only when egress port happens to be idle!)

1km in fibre is about 5us latency (again, single direction).

Serialization delay @10G for minimum sized payload (46B) is about 67ns (0.067us), by increasing link-speed, you decrease serialization delay.

IP header is 20B, UDP header is 8B, you data is 8B, so you have just 36B of data, which means your ethernet payload will include 10B of trash you MUST send, i.e. if you have something to add to your payload add it, it has 0 latency cost.

I hope you can extrapolate RTT from these, by multiplying the device delay by device count and adding 5us for every kilometer of fibre and then multiply that by 2.


I can't resist adding some thoughts aboutg HFT.

According to this HFT volume halved between 2009 and 2012. Suggesting that the easy wins are gone. I'd love to see some scientific paper or just real data about HFT latency and its affect on profit. I suspect latency which affects trade profits are on another magnitude than latency we're talking about right now. Friend of mine who builds network for one of the largest exchange seems to think it's just customer who do 'lower == better' without understanding the scales.
I can completely understand how HFT was useful when few people were doing it, when you could observe marketA not seeing change marketB does see and capitalize on it. Some are talking about using regulation to stop HFT by taxing each trade making it expensive for everyone, I don't think it'll be needed, I think the window of opportunity is already closing.

4
  • Excellent reply and very informative, thanks. Even a few personal thoughts on HFT!
    – John Smith
    Jul 6, 2013 at 8:03
  • 1
    @JohnSmith, do not neglect to consider delay introduced inside your endpoints (such as the OS scheduler, or kernel processing)... this can contribute significantly to the delay I mentioned in a previous comment. Jul 6, 2013 at 14:13
  • Excellent point about using full minimum packet size at 0 latency cost. Sep 12, 2013 at 8:39
  • Worth mentioning that on a typical server, PCIe adds ~3us latency and interrupt/context-switch/kernel adds ~5us. The latter is easy to eliminate, but the former isn't. Also people who care about latency use infiniband, not Ethernet. Typical IB switch latency is 100ns
    – Navin
    Aug 6, 2022 at 7:18
1

I think on conventional semi-tuned hardware you should be able to:

  1. Go out your host network stack
  2. Up your next network stack
  3. Back out that network stack with your 'new' packet

In ~ 10 us at 10gig. If you really lock things down, that number can be significantly lower.

Almost ALL of the latency you're going to see is not from networking hardware/cables, but from your host systems. A reasonable cut through switch, (Arista, Gnodal, New Cisco, etc) is going to be sub 1us.

Start with ensuring the processes consuming the UDP packets are pinned to the same core as your NIC interrupts. From there, ensure that coalescing on your NIC is disabled, and from there make sure you've got MSI-X and DCA on.

If you're more serious...check out SolarFlare's OpenOnload. They have a great suite of tools for testing/verifying performance as well.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.