1

I have a network issue that has to do with microbursts. I will first tell a bit about the network:

There are 4 Aruba/HP ProCurve switches. 1 Core switch and 3 downstream switches. All connections in this network are 1 Gbit/s connections except the connection to a server. The server is connected to the core switch with 2x 10Gbit/s.

In several locations there are Clients that stream video data from the server. All together 1Gbit/s bandwith in average is enough. Unfortunately, I noticed that there are always drops on the 1Gbit/s downstream connections on the core switch. I thought microbursts could be an issue here and was right. I tested it with wireshark (you can see the I/O graph here: https://www.dropbox.com/s/fcvvqevv32i8wdy/wireshark_D2B607E8-6C5C-41CD-8539-58F35C6A2FFD_20180725140635_a08312.png?dl=0)

The wireshark output was traced on the server. On the first part of the graph where you can see the bursts I mirrored one affected downstream port to the server. The second part (without big bursts) looks different because I stopped mirroring while the trace was running. The problem is that I have no idea how to stop this bursts and the resulting drops. I tried to play a little bit with the settings of the servers network card (buffer size etc.) but I don't have experience with problems like this. What I don't want to do is connecting the server with 1 instead of 10 Gbit/s.

I hope to find some ideas here.

|improve this question
  • Welcome to Network Engineering! Drops are a normal part of networking. Do users notice the problem, or is it just you? – Ron Trunk Nov 11 '18 at 17:31
  • It is my management software that noticed it for me. The problem is that there are so many drops that if I just ignore the issue I will not able to notice other drops that could have more impacts. And we are talking about uplinks so this are important connections. – Otto Nov 11 '18 at 17:38
  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can provide and accept your own answer. – Ron Maupin♦ Dec 25 '18 at 9:56
2

Your network is becoming too thin too fast between the server and the clients. While the server can send well over 1 Gbit/s to each client switch, those switch uplinks are bottlenecks - more than 1 Gbit/s on a link results in dropped traffic.

Your graph shows the bursts to carry too much volume to be buffered within the core switch. You should check the switch statistics to make sure. You'll need to either moderate the flows to decrease bandwidth (hard or even impossible) or increase bandwidth between core and downlink switches.

The best solution is a 10 Gbit/s link to each downlink switch. The second best is to aggregate at least two 1 Gbit/s links. Since link aggregation doesn't provide proper load balancing you may need more than two links.

Edit: As a last resort, you could activate flow control on the core switch and the server NICs. Flow control generates pause frames when egress buffers in the core switch fill up. Those cause the sender to cease transmission for a short period. This moves the buffering to the server and may help.

|improve this answer
  • Unfortunately, I don't have the opportunity to increase the downstream bandwith. Otherwise I would already have done this because this was my first idea. Aren't there some tricks to "spread" the traffic a bit more to avoid the short bursts? – Otto Nov 11 '18 at 17:33
  • Out of ports? Consider getting a larger or a 2nd core switch. – Zac67 Nov 11 '18 at 17:37
  • Money is the main problem – Otto Nov 11 '18 at 17:39
  • 1
    You will either have to decrease the server bandwidth or increase the capacity of your network devices. You simply have more demand than your network can provide. – Ron Trunk Nov 11 '18 at 17:43
  • I've added flow control to the answer - it may be helpful in this situation. – Zac67 Nov 11 '18 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.