I have to emulate a wide area network. My setup contains two servers with MT27500 Family [ConnectX-3] Infiniband cards. Default latency between both nodes is ~0.4ms and iperf test shows throughput of 4Gbps per iperf connection. I was able to get max of 33Gbps by using 15 parallel iperf connections. To simulate RRT of 50ms between both nodes I am using following command.
sudo tc qdisc add dev ib0 root netem delay 25ms
However iperf throughput perstream drops to 154Mbps. I have another identical setup which have Ethernet adapters instead of Infiniband. On that setup iperf shows around 480Mbps per iperf connection which is 3 times more than that I am getting from IPoIB. Is it expected behavioir that IPoIB peformes poor than ethernet over large latency? If so what is the reason behind this behavior?
-wcommand line parameter lets you override the defaults (sometimes there's an upper limit given by the operating systems parametrisation), so that the send/receive buffers grow and TCP window scaling is used.