Network Engineering Stack Exchange is a question and answer site for network engineers. It only takes a minute to sign up.
Sign up to join this community
Many references online mention you should use 10Gb links (HBA/CNA) when designing your SAN. However, is there a formula or calculation that can provide the amount of bandwidth that is really required when using a SAN-based protocol. i.e can you relate IOPs to actual network traffic.
Thanks,
IOPs depend mainly on throughput capabilities and latencies of your storage hardware and (to a much lesser extent) latencies in your network. In other words, you can't "translate" a certain infrastructure to IOPs (you can make an educated guess though).
However, you need to design your SAN to manage the bandwidth you require. Once you exceed the bandwidth in your infrastructure, IOPs simply stop on that level and latencies skyrocket.
If you have a concrete workload you can try to calculate bandwidth from IOPs but that may or may not match reality.
For maximum iSCSI port throughput, you can multiply the efficiency for TCP over standard Ethernet (94.9%) or using 9k jumbo frames (99.1%) with the nominal Ethernet port rate to get a good figure. Overhead from iSCSI itself is pretty much neglectable (except for very high IOP figures and very low actual bandwidth) and decent switches will be very close to the theoretical maximum.
Native Fibre Channel is a bit harder to grasp due to its very rough nominal speeds (including L0 overhead, unlike Ethernet), but you can get an idea here.
In SANs you need to very specifically look into the (present and future) workloads. Aggregating links may come as scaling path but you need to be aware that an aggregated link cannot transport flows faster than any one of its physical links. In other words: aggregating links increases overall bandwidth but it doesn't increase bandwidth for any one flow.
Four aggregated 10G links support one or more (ideally four) separate 10G flows but they don't support a single 40G flow. That is because each connection is always forwarded through the same physical port pair - depending on the hardware and its configuration, by MAC addresses, IP addresses, or IP and L4 port combinations.
Also, you need to consider that in a SAN, multiple bandwidth-demanding workloads can run at the same time. You may be able to separate them in time windows (e.g. running backups during slow night times) but if there's a potential need to run them in parallel, you need to be prepared. Interconnects can very easily become bottlenecks (which you generally want to avoid in a SAN). Depending on your requirements and your budget, you need to find a viable concept.
So, you need to plan the network very carefully and consider the future demand for workloads within the lifetime of the network. Upgradable switches provide more headroom, aggregating links may have severe limitations. You can start cheap but then you may need to replace hardware before the planned service life ends.
