9

While studying for Cisco and Microsoft certifications I read a lot about NIC Teaming, Etherchannels and similar stuff.

I know you can do static teaming in MS Windows Server or creating a port-channel in Cisco or use Dynamic protocols such as LACP and PAgP. But they all offer load-balancing using source or destination MAC address or IP address, a combination XORed together or similar things, right? And it seems there's no way to increase your actual throughput if you have a long long file transfer from one server to another (ex. having 4 GigE bonded together to increase it to 4 Gbps).

So I wonder if it is true? Is there a way to use all the bandwidth of bonded interfaces? I heard from few sources that there's something like SLB (Switch-Assisted Load Balancing w/ Fault Tolerance) which supports full load balancing and allows to use full bandwidth (like 4 Gbps from my example between two servers in one transaction).

I've heard so many contradictory things about that, and I am totally confused. Help me guys!

  • Not a bad question, but AFAIK there is no way to increase the throughput of a single data stream by using load balancing. – pauska Jul 24 '13 at 11:37
  • That wasn't an answer! Just a comment :-) wait for the more skilled guy to come up with a more in-depth answer. – pauska Jul 24 '13 at 11:43
  • Of course I will :) Yet I am a little bit skeptic about that being possible, but some some sales guys say that it is and my colleague (server guy) is confused too :) Want to rule that out – Alex Jul 24 '13 at 11:46
  • haha just remembered: Please Do Not Take Sales People's Advice from the OSI lessons a few years ago – Alex Jul 24 '13 at 11:48
8

At layer 2, all load balancing is, at best, done by an XOR or hash of the source and destination MAC, and if you're lucky, it may even read into layer 3 and hash that data too.

At layer 3, however, where we're basically talking about multiple gateways (so, effectively, two physical links with a unique next-hop across each) you can max out the bandwidth across the links IF you're prepared to do per-packet balancing.

Before I go on, per-packet balancing is generally a bad thing due to the fact that it can result in out-of-order packet delivery, this can be especially terrible with TCP connections, but that of course comes down to the implementation and most modern stacks can tolerate this relatively well.

In order to do per-packet balancing, obviously one requirement is that the source and destination IP addresses are not at all on-link to the devices that have the multiple paths since they need to be routed in order for balancing to be possible. Redundancy can be achieved via a routing protocol such as BGP, OSPF, ISIS, RIP, or alternatively, BFD or simple link-state detection.

Finally, there is of course a transport layer solution - protocols like SCTP support connecting to multiple endpoints, and TCP already has drafts in the making that will add options to do similar things. Or... you can just make your application open multiple sockets.

| improve this answer | |
  • Thanks for answering! Yeah, I'm aware of that L3 load-balancing. And, basically, if you want more than one gig at layer two, go with 10gig? Especially if you want to do backups between servers @ layer 2, which are connected to the same switch? – Alex Jul 24 '13 at 12:46
  • Basically, yes - you may be able to do hacky things on the host itself such as making a faux bridge and then rewriting the destination MAC of each frame to make each one go out of a different interface, but this really depends on your operating system, is non-trivial and not scalable. – Olipro Jul 24 '13 at 13:33
  • We've a mix of RHEL and Win 08 R2 – Alex Jul 24 '13 at 13:36
5

First, let's be realistic about your system performance; you have to ensure that your system can consistently spool the bandwidth you're asking about from disk to the NICs in question (see bottom of answer)...

As long as you're willing to use FTP for your file transfers, you can use a client that supports segmenting the file via FTP REST command for downloads, and a server that supports the FTP COMB command for segmented uploads. By using FTP clients that support segmenting, you can open multiple FTP sessions per file transfer and take advantage of the Layer4 LACP hashing supported by many switch vendors... regardless of whether you're bonding 1GE or 10GE links, segmented transfers accelerate your performance across LACP.

CuteFTP for Windows supports REST transfers and Filezilla supports the COMB command. These are just two of several options you could use.


System limits

I don't know what kind of disk you have, but often the single-system limit is disk I/O... there are several points of interest here... from the controller's perspective...

  • SATA (revision 1) max xfer: 1.2Gbps after 8b/10b overhead
  • SATA (revision 2) max xfer: 2.4Gbps after 8b/10b overhead
  • SATA (revision 3) max xfer: 4.8Gbps after 8b/10b overhead
  • 4G Fibre Channel max xfer: 3.56Gbps

Another possible limit is the disk speed / spindles / arms... unless you're using an SSD.

Finally you need NICs with sufficient capacity to the motherboard... check how many lanes are on your PCIe NIC. I'm assuming you're on a LAN and that packet loss / congestion won't be an issue. You might want to turn on jumbos if you find that CPU is a limiting factor for your transfers.

| improve this answer | |
  • I think we use FTP and SCP for transferring. As far as the I/O performance concerned, we are planning to migrate to iSCSI and regardless of what I say, we can't really afford a high-density 10gig switch due to limited budget (which is really needed in that case) so we're trying to improvise as we go. Just planning. – Alex Jul 24 '13 at 13:33
  • @Alex I also work with a limited budget and use iSCSI.. The Dell PowerConnect 8132/8164F switches are about as cheap you can get per port with SFP's and not a absymal portbuffer. We've had great success with them. – pauska Jul 24 '13 at 13:37
  • I'll have a peek at them:) – Alex Jul 24 '13 at 13:39
  • @Alex, I'm not proposing that you use a 10GE switch... even multiple 1GEs bonded together can exceed server capacity... I updated my answer to highlight that I'm suggesting that you use LACP – Mike Pennington Jul 24 '13 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.