0

In our datacenter I'm having throughput issues if a server is connected on a 10G port on the switch. For debugging I already removed our router and asked our ISP for a small IP range that I can use directly. This way I can rule out a problem with the router.

So, right now the setup is very simple but the problem is still there.

Current setup:

ISP (1G link) -> Ubiquity ES-16-XG switch -> 10G Servers

Nothing more. Nothing less.

Now, when doing an iperf3 test from a remote location I'm only getting an average 200mbit in download (seen from the remote location) and 900mbit while uploading (seen from the remote location).

If I force any server from 10G to 1G on the switch, the throughput problem is gone. I get 900mbit in both directions.

This made me think that maybe the switch was the problem. The thing is: when I test locally, between a server on 1G and another server on 10G, iperf3 shows 940mbit in both directions. So the switch seems okay.

I hope someone knows what's happening here and what I can do about it.

Thanks

Improve this question
6
  • Check the switch for port errors. Are your light levels within range?
    – Ron Trunk
    Aug 4, 2022 at 12:42
  • This ubiquity switch has a counter called "OutDiscPkts" which are the packets it discarded. Looking at the interface where my ISP is connected this counter doesn't change when doing an iperf3 test with a server on 1G. But while doing an iperf3 with a server on 10G the counter increases at a rate of approx. 15 packets per second. But why is it discarding packets? Or is this amount of packets normal because it's comming from a 10G server and going to a 1G port?
    – user419650
    Aug 4, 2022 at 12:47
  • You are oversubscribed. If the server is sending at 10 Gbps, the 1 Gbps interface to the ISP will drop up to 90% of the frames.
    – Ron Maupin ♦
    Aug 4, 2022 at 12:50
  • what throughput do you have between 2 servers connected in 10G?
    – JFL
    Aug 4, 2022 at 13:12
  • With 2 servers on 10G I'm getting 8Gbps
    – user419650
    Aug 4, 2022 at 14:30

1 Answer 1

Reset to default
1

Some discards are completely normal, esp. when link speeds differ. Is Ethernet flow control active across the switch (ie. on the ISP router, the switch and the server)? Flow control may be detrimental due to head-of-line blocking and you should try without. Disabling on the switch alone should be sufficient.

Another possibility is that you see a resonance between TCP congestion control, switch buffering, and ISP router buffering.

If you can, try another switch with a deeper buffer. If you need 10G link internally and don't want to slow everything, a possible approach would be to use an additional 1G port on each server with an additional 1G switch for WAN connectivity - crude but likely effective.

Improve this answer
7
  • Flow control is disabled. I'm not sure how big the buffer of this ubiquity ES-16-XG switch is. I read on ubiquity's forum that it's only 2MB shared. Isn't that a very small buffer?
    – user419650
    Aug 4, 2022 at 14:30
  • You said I can try with a different switch. But I don't have one. I can buy a second hand Dell PowerConnect 8024F switch. Maybe this one is better? It seems to have 16Mb of packet buffer memory
    – user419650
    Aug 4, 2022 at 14:31
  • 2 MB is fairly small, esp. for 10G, but doesn't need to be a problem unless you have extremely bursty traffic. Large buffers can be more of a problem when they cause resonance (16 MB isn't 'large' for 10G either). We cannot recommend any products here, sorry. If your budget is tight, a separate 1G switch might be cheapest, assuming the servers have unused 1G ports.
    – Zac67 ♦
    Aug 4, 2022 at 14:36
  • I'm testing with iperf3 only for now. So it's not what you call bursty traffic, right? Adding a 1G network besides this one creates more issues because the servers will be VM hosts. All internet traffic passing between VM's on different hosts will be limited to 1G. Which is not what I want. So I want to solve the issue correctly
    – user419650
    Aug 4, 2022 at 14:43
  • I'm not even sure if the buffer size of the switch is the problem at all. So maybe I should just go for the 8024F and try? If it's not helping, the 8024F sounds like a more decent switch compared to the ubiquity one. It will probably perform better in many ways
    – user419650
    Aug 4, 2022 at 14:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.