I'm benchmarking latency of sending a TCP packet from a client to a server. The payload of the packet is the nanosecond timestamp (using clock_gettime()
on Linux) right before the send()
socket function. Basically, my client/server pair measures the time from the client calling send()
, to when the server have recv()
.
The client/server programs are run on 2 PCs in the same room, same network/router, and both use Solarflare X2522 25Gigabit/s network cards. The server use Solarflare EF_VI kernel bypass, the client uses BSD C Socket with OpenOnload kernel bypass. Both programs use memory pinning + CPU pinning + core isolation + locked CPU clock.
Currently, my latency is ~1.8 microsecond. Each of my TCP packet has size exactly 94 bytes, and I use TCP_NODELAY
option in C socket to force send a packet as soon as send()
is called.
Problem is, IDK how much time is spent in each OSI layer (or software/hardware). So I don't know if performance is limited by hardware (NIC, routers, optical fibres, etc) or software (network stack). Is there anyway to benchmark this?