0

I'm benchmarking latency of sending a TCP packet from a client to a server. The payload of the packet is the nanosecond timestamp (using clock_gettime() on Linux) right before the send() socket function. Basically, my client/server pair measures the time from the client calling send(), to when the server have recv().

The client/server programs are run on 2 PCs in the same room, same network/router, and both use Solarflare X2522 25Gigabit/s network cards. The server use Solarflare EF_VI kernel bypass, the client uses BSD C Socket with OpenOnload kernel bypass. Both programs use memory pinning + CPU pinning + core isolation + locked CPU clock.

Currently, my latency is ~1.8 microsecond. Each of my TCP packet has size exactly 94 bytes, and I use TCP_NODELAY option in C socket to force send a packet as soon as send() is called.

Problem is, IDK how much time is spent in each OSI layer (or software/hardware). So I don't know if performance is limited by hardware (NIC, routers, optical fibres, etc) or software (network stack). Is there anyway to benchmark this?

Improve this question
3
  • 1
    No OS implements the OSI model. The IP Services model is closer to what is used, but it is also just a model, and the real world often differs from it.
    – Ron Maupin ♦
    May 30, 2022 at 15:00
  • I agree with all the comments - however you say "... performance is limited by hardware (NIC, routers, optical fibres, etc) or software (network stack). " I suppose if its hardware you can upgrade , if its software you can try different drivers - So what is the problem and what do you want to improve
    – Ross
    Jun 1, 2022 at 11:53
  • @Ross Sorry the question description is bad. I mean how to measure the time spend in each software/hardware component (like time from CPU to network card, card to switch, etc) Basically, I want to find WHERE the current bottleneck is
    – Huy Le
    Jun 1, 2022 at 13:10

2 Answers 2

Reset to default
2

I don't think focusing on the layers is really practical.

While measuring the impact of external devices is fairly easy - just remove, reinsert and replace them from the measured path -, internal overhead is very hard to come by. Most of the processing happens inside the NIC or inside the OS's IP stack - normally without any form of precise monitoring.

Also, most better NICs feature offloading of higher-layer functionality to the hardware, so a precise benchmarking through the various layers would only be possible if all offloading is deactivated - likely not what you're looking for.

I'm afraid you'd have to benchmark each NIC/driver/platform/CPU/BIOS setting by itself and then derive a practical way to interpret the data. Basically, it boils down to benchmarking multiple configs and selecting between them.

Regarding the potential bottleneck: Those 1.8 μs are end-to-end / process-to-process? With a single switch in between? That's actually pretty good already: The pure serialization delay of 94 bytes user(?) data over TCP over IPv4 over 25G is (94+20[TCP]+20[IPv4]+26[Ethernet])*8/25G *2 = ~.1 μs (twice the single delay for a single L2 hop across a store-and-forward switch). 1.5 μs for the switch is already pretty fast, so there's not much else.

Improve this answer
4
  • In this case, is there anyway to find where the bottleneck is?
    – Huy Le
    May 30, 2022 at 11:08
  • The bottleneck implies that there's a single stage with a huge delay. I don't think that's the case in your scenario - all components cause small delays that sum up to some point. I've edited the answer to better reflect that.
    – Zac67 ♦
    May 30, 2022 at 13:13
  • Is there any document that measure/benchmark switch latency? My latest code has 1.5us median latency (process to process). Also 94 is the size of the entire packet (payload is only 24 bytes)
    – Huy Le
    Jun 1, 2022 at 7:53
  • 1
    @HuyLe Measuring switch latency isn't easy and usually requires very expensive hardware. You need to either believe the vendor's datasheet, temporarily remove the switch from the path, or replace the switch model for comparison. The packet/frame size in that range doesn't really have much impact as you can see above.
    – Zac67 ♦
    Jun 1, 2022 at 8:46
2

Your computer software, including the operating system, does not use the OSI model. So it’s impossible to measure. The models are just that—models. Real computers don’t follow it exactly.

In addition, there is no sharp distinction between what happens in hardware vs software. The interaction can be complex.

Improve this answer
2
  • Sorry the question description is bad. I mean how to measure the time spend in each software/hardware component (like time from CPU to network card, card to switch, etc)
    – Huy Le
    Jun 1, 2022 at 8:29
  • You would have to disassemble the drivers and calculate it. Again, the interaction is complex. It may be hard to define what "time spent in a component" means.
    – Ron Trunk
    Jun 1, 2022 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.