10

I have limited knowledge of this and I'm still learning, so I hope someone can give me advice on this.

I have a 10GbE NAS, a 10GbE NIC in my PC and a switch with two 10GbE ports, which connect all together with fiber. Then there is also another 100Mbit cable attached which connects the switch to the router, which is also the standard gateway. This one also provides the internet connection for all connected devices.

I noticed that a direct connection to the NAS gives me speeds that 10GbE can handle, but when I put the switch in between (all standard VLAN, same subnet and standard gateway), the speeds to the NAS is limited to a max of 1GbE.

Is because the standard gateway is the central where all traffic comes together? How can I manage a peer to peer connection directly to the NAS via the switch, so the traffic goes through the 10GbE cable to the switch and from there through the 10GbE to the NAS?

Extra details:

IP NAS: 192.168.10.120
IP switch: 192.168.10.125
IP PC: 192.168.10.103
Standard gateway: 192.168.10.1
Subnet: 255.255.255.0
Switch: D-link 1510-52

Thank you

  • Which switch are you using? Is it a 10GbE switch? – Gerben Oct 20 '17 at 11:54
  • 1
    It's a switch with two sfp+ ports. The nas and PC are connected in these – EarthMind Oct 20 '17 at 12:57
  • 1
    What type of SFP are you using? – MerlinTheMagic Oct 20 '17 at 16:31
  • 1
    Follow up. I see you state it’s a 10GBase-R in another comment, but if you provide the model of the SFP we could help you further. For example it’s possible you are driving a short jumper with a ZR SFP or the opposite a long cable with an SR module. – MerlinTheMagic Oct 20 '17 at 17:43
  • 1
    @MerlinTheMagic See my last comment to Peter Green's post for the root of the cause :-) – EarthMind Oct 22 '17 at 11:03
16

I don't think the default gateway has anything to do with your problem. Local traffic between two devices on the same subnet doesn't flow through the default gateway and in any case the speeds you are reporting for your local traffic are 10 times higher than the speed you report for the link to your default gateway.

A couple of possibilities spring to mind.

Firstly are you in the correct ports? from some searching it seems only two of the four fibre ports on that switch are 10G.

Secondly I have heard reports of some early switches which have 10G ports but which can't handle a full 10G for a single flow because of their internal forwarding design.

  • 1
    My thoughts too. I'd check the netmasks and unplug the router to remove it from the problem. Also check you've no port mirror on the switch. – jonathanjo Oct 20 '17 at 12:56
  • 1
    I'm using the right ports yes, because they are sfp+ ports and I'm using the right transceivers to have a connection. These ports are advertised as uplink/stacking ports but no limitations have been described in the specs. Full duplex 10Gbase-R – EarthMind Oct 20 '17 at 13:01
  • 1
    ... I'd also try locking the port speeds on the switch, NAS, PC to see if they might be changing speed for some reason. – jonathanjo Oct 20 '17 at 13:17
  • 2
    The max switching capacity is 140 Gbps from (us.dlink.com/products/business-solutions/…) so I believe it should easily be able to manage it... – djsmiley2k - CoW Oct 20 '17 at 14:55
  • 2
    AIUI the issue on some early switches with 10G uplinks was that the switch engine was only designed for 1G ports. So the switch manufacturer used the link aggregation features of the switch engine to implement the 10G ports limiting the bandwidth of any one flow to 1G. This is not too much of a problem when using the ports for the intended purpose as uplinks but it is a problem in your usage scenario. – Peter Green Oct 20 '17 at 15:02
3

Looking at the Data Sheet for the D-Link 1510-52 switch:

It mentions switch being capable of "140Gbps" of throughput - using typical marketing "full-duplex" numbers this would mean that on a 50x1G and 2x10G switch like the 1510-52 that the switch is capable of running line-rate through every interface simultaneously. Eg: (1Gbps x 50) + (10Gbps x 2) = 70Gbps x 2 (Full Duplex) = 140Gbps.

Similarly the Packets-Per-Second (pps) numbers for the box are listed as 104.16Mpps, which equates nicely to line rate:

1Gbps = 1,000,000,000 bits/s = (1,000,000,000 bits/s) / (8 bits/byte)= 125,000,000 bytes/s

10Gbps = 10,000,000,000 bits/s = (10,000,000,000 bits/s) / (8 bits/byte)= 1,250,000,000 bytes/s

PPS on 1G port = (125,000,000 bytes/s) / (84 bytes/packet) = 1,488,095 pps

PPS on 10G port = (1,250,000,000 bytes/s) / (84 bytes/packet) = 14,880,952 pps

50 x 1,488,095 pps + 2 x 14,880,952 pps = 104,166,654 pps ~ 104.16Mpps

So, if the data sheet is to be believed, the switch should be capable of this easily.

The one thing you might want to confirm though is that there are two 1G SFP ports and two 10G SFP+ ports, so make sure that you are connected to ports 51 and 52 on the switch and that you are seeing Orange LED status on both ports (which means 10G). If you're seeing Green LEDs this means you're only getting a 1G link.

  • Thank you for the information. I shouldn't be able to use the SFP ports with the SFP+ plugs, because they are not backwards compatible. SFP can be used in SFP+ generally, but not vice versa. So this was why I was sure from the start that I didn't confuse the ports. For now I seem to be able to use the full 10 Gbit connection speed for some reason. I'll be keeping an eye on it whether it stays that way and if not, look for the root of the problem. – EarthMind Oct 21 '17 at 12:04
  • @EarthMind that's correct - more just making sure your SFP+ modules are in fact SFP+ and not SFPs – Benjamin Dale Oct 22 '17 at 3:59
3

When switching, throughput between two ports is completely independent of the link speed of another interface. With the non-blocking switches of today, it's even independent of the throughput on any other ports.

There are several possible reasons for the low throughput:

  • SFP+ transceivers: they need to support 10GBASE; SFPs/mini GBICs only support 1G rate - check link status in switch
  • interface link configuration: may be limited to 1G speed
  • other interface configuration: rate limiting, maximum frame size mismatch, ...
  • physical link problems: wrong fiber type (-SR = MM, -LR = SM), damaged fiber, dirty ports (FCS and - if indicated - FEC will show errors counting up)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.