How to avoid microbursts causing drops?

Question

I have a network issue that has to do with microbursts. I will first tell a bit about the network:

There are 4 Aruba/HP ProCurve switches. 1 Core switch and 3 downstream switches. All connections in this network are 1 Gbit/s connections except the connection to a server. The server is connected to the core switch with 2x 10Gbit/s.

In several locations there are Clients that stream video data from the server. All together 1Gbit/s bandwith in average is enough. Unfortunately, I noticed that there are always drops on the 1Gbit/s downstream connections on the core switch. I thought microbursts could be an issue here and was right. I tested it with wireshark (you can see the I/O graph here: https://www.dropbox.com/s/fcvvqevv32i8wdy/wireshark_D2B607E8-6C5C-41CD-8539-58F35C6A2FFD_20180725140635_a08312.png?dl=0)

The wireshark output was traced on the server. On the first part of the graph where you can see the bursts I mirrored one affected downstream port to the server. The second part (without big bursts) looks different because I stopped mirroring while the trace was running. The problem is that I have no idea how to stop this bursts and the resulting drops. I tried to play a little bit with the settings of the servers network card (buffer size etc.) but I don't have experience with problems like this. What I don't want to do is connecting the server with 1 instead of 10 Gbit/s.

I hope to find some ideas here.

Answer 1

Your network is becoming too thin too fast between the server and the clients. While the server can send well over 1 Gbit/s to each client switch, those switch uplinks are bottlenecks - more than 1 Gbit/s on a link results in dropped traffic.

Your graph shows the bursts to carry too much volume to be buffered within the core switch. You should check the switch statistics to make sure. You'll need to either moderate the flows to decrease bandwidth (hard or even impossible) or increase bandwidth between core and downlink switches.

The best solution is a 10 Gbit/s link to each downlink switch. The second best is to aggregate at least two 1 Gbit/s links. Since link aggregation doesn't provide proper load balancing you may need more than two links.

Edit: As a last resort, you could activate flow control on the core switch and the server NICs. Flow control generates pause frames when egress buffers in the core switch fill up. Those cause the sender to cease transmission for a short period. This moves the buffering to the server and may help.