Many references online mention you should use 10Gb links (HBA/CNA) when designing your SAN. However, is there a formula or calculation that can provide the amount of bandwidth that is really required when using a SAN-based protocol. i.e can you relate IOPs to actual network traffic.


  • Did any answer help you? If so, you should accept the answer so that the question doesn't keep popping up forever, looking for an answer. Alternatively, you can provide and accept your own answer. – Ron Maupin Dec 25 '18 at 9:42

IOPs depend mainly on throughput capabilities and latencies of your storage hardware and (to a much lesser extent) latencies in your network. In other words, you can't "translate" a certain infrastructure to IOPs (you can make an educated guess though).

However, you need to design your SAN to manage the bandwidth you require. Once you exceed the bandwidth in your infrastructure, IOPs simply stop on that level and latencies skyrocket.

If you have a concrete workload you can try to calculate bandwidth from IOPs but that may or may not match reality.

For maximum iSCSI port throughput, you can multiply the efficiency for TCP over standard Ethernet (94.9%) or using 9k jumbo frames (99.1%) with the nominal Ethernet port rate to get a good figure. Overhead from iSCSI itself is pretty much neglectable (except for very high IOP figures and very low actual bandwidth) and decent switches will be very close to the theoretical maximum.

Native Fibre Channel is a bit harder to grasp due to its very rough nominal speeds (including L0 overhead, unlike Ethernet), but you can get an idea here.

In SANs you need to very specifically look into the (present and future) workloads. Aggregating links may come as scaling path but you need to be aware that an aggregated link cannot transport flows faster than any one of its physical links. In other words: aggregating links increases overall bandwidth but it doesn't increase bandwidth for any one flow.

Four aggregated 10G links support one or more (ideally four) separate 10G flows but they don't support a single 40G flow. That is because each connection is always forwarded through the same physical port pair - depending on the hardware and its configuration, by MAC addresses, IP addresses, or IP and L4 port combinations.

Also, you need to consider that in a SAN, multiple bandwidth-demanding workloads can run at the same time. You may be able to separate them in time windows (e.g. running backups during slow night times) but if there's a potential need to run them in parallel, you need to be prepared. Interconnects can very easily become bottlenecks (which you generally want to avoid in a SAN). Depending on your requirements and your budget, you need to find a viable concept.

So, you need to plan the network very carefully and consider the future demand for workloads within the lifetime of the network. Upgradable switches provide more headroom, aggregating links may have severe limitations. You can start cheap but then you may need to replace hardware before the planned service life ends.

  • Thanks ,but the question is still how do you size the network, when do you go for 40gb over 10gb etc ie for ISCSI let’s say. – RickD Oct 19 '18 at 8:40
  • You need to examine your workloads. When they demand 40G you need your network to provide the speed. – Zac67 Oct 19 '18 at 9:02
  • But you need to design before hand hence looking for how this is typically design/calc from a storage arch standpoint 😃 – RickD Oct 19 '18 at 9:04
  • Yes. You need to take a crystal ball and look at what might be coming at you within the next x years. That is your design base for the network. – Zac67 Oct 19 '18 at 9:13
  • 1
    that's why ability to scale is (or should be) an important aspect of your design, especially if you are unsure about workloads. – Teun Vink Oct 19 '18 at 9:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.