2

I have to emulate a wide area network. My setup contains two servers with MT27500 Family [ConnectX-3] Infiniband cards. Default latency between both nodes is ~0.4ms and iperf test shows throughput of 4Gbps per iperf connection. I was able to get max of 33Gbps by using 15 parallel iperf connections. To simulate RRT of 50ms between both nodes I am using following command.

sudo tc qdisc add dev ib0 root netem delay 25ms

However iperf throughput perstream drops to 154Mbps. I have another identical setup which have Ethernet adapters instead of Infiniband. On that setup iperf shows around 480Mbps per iperf connection which is 3 times more than that I am getting from IPoIB. Is it expected behavioir that IPoIB peformes poor than ethernet over large latency? If so what is the reason behind this behavior?

Improve this question
3
  • You need to edit your question to include more information about the network. How are the network devices connected (a diagram would be great), what are the network device models, and what are the network device configurations?
    – Ron Maupin ♦
    Jul 27, 2018 at 18:40
  • I'm not sure if emulating latency on an end system is the right way to go. Latency on a network comes from the propagation time across the medium (5 microseconds per km on optical fiber) and from deep buffers (also see the "buffer bloat problem", you'll find tons of resources about it on the net). Just adding 25ms of delay on one of the end systems (assuming that is what you do) might not give the same result as having an actual 25ms buffer in the form of a network path which keeps "bytes in flight". Be sure to understand how this 25ms delay feature works, especially w/regards to buffering.
    – Marc 'netztier' Luethi
    Jul 28, 2018 at 12:17
  • (part 2) Either way, the bandwidth * delay product of a given network path must be accounted for by the (TCP based) application, by scaling up the send/receive windows. switch.ch/network/tools/tcp_throughput/… might give you a clue how large the send/rececive buffers have to be, so that you can saturate a "long fat pipe". iperf's -w command line parameter lets you override the defaults (sometimes there's an upper limit given by the operating systems parametrisation), so that the send/receive buffers grow and TCP window scaling is used.
    – Marc 'netztier' Luethi
    Jul 28, 2018 at 12:21

1 Answer 1

Reset to default
3

The most common mistake made when using netem to emulate network delay is to not increase the packet limit. You didn't. Therefore you tail drop a lot of packets and tcp cannot build a queue long enough to emulate the path.

try limit 1000000 or more. Monitor your netem stats to see if you drop any packets.

Plug: we use (bufferbloat project) flent.org's test tool a lot. It wraps netperf and irtt to make reproduceable tests easier.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.