3

Looking to purchase some dedicated 10GB switches for our iSCSI setup. Would the following setup allow us to use two switches for redundancy?

Two VMWare ESXi hosts, each with 2 10GB NIC Storage with 2 10GB NIC

If NIC1 on VMWareHost1 connects to switch1 and NIC2 on VMWareHost1 connects to switch2 (and the same setup for VMWareHost2 and Storage) how would the ESXi hosts communicate with the LAN? Do I need to add more NIC to the hosts? Or can I connect the dedicated switch to the LAN Switch and have it go through there? Would the storage server need to communicate with the LAN for any reason?

Please advise, this is my first time setting up iSCSI.

Thank you

3
  • Do you already have a pair of switches for your "regular" LAN traffic? Jul 27, 2021 at 15:29
  • This is more a SF question as it's very much "how to build a storage network", which isn't very much about the "network" (IP) part. Multipathing makes things quite different for a SAN, and you don't necessarily want any routing between SAN LAN segments. (this is something much larger than most will want to document for free in a SE answer. I could swear there's a KB article or two showing a basic iSCSI setup. The last one I built took minutes to build, and hours to explain.)
    – Ricky
    Jul 28, 2021 at 11:30
  • Did any answer help you? if so, you should accept the answer so that the question does not keep popping up forever, looking for an answer. Alternatively, you could post and accept your own answer.
    – Ron Maupin
    Dec 23, 2021 at 21:04

1 Answer 1

1

Two VMWare ESXi hosts, each with 2 10GB NIC Storage with 2 10GB NIC

Where's the storage?

how would the ESXi hosts communicate with the LAN?

The host should use dedicated NICs for LAN and SAN. It is absolutely possible to share NICs, switches and even VLANs across LAN & SAN but that should be considered carefully. Keep in mind that storage is a vital asset in your infrastructure and it should be up and running at all times. Losing storage means losing data and halting production.

Separating VLANs is a requirement for basic security and resilience - even if you protect SAN connections by authentication and encryption it still opens a can of worms in regard of DoS and exposing all vulnerabilities to all users.

Using separate NICs saves your SAN links from any LAN faults - broadcast storms, bottlenecks, congestion and simplifies management. Clean separation is also an absolute life saver for troubleshooting.

Using separate switches minimizes risks from any instabilities that may come from normal LAN operations - think of configuring port or ACL policies. You can also separate administration entirely. Ideally, your SAN switches only have management links into your LAN.

For a fully redundant connection you need at least two NICs (each via a separate switch). Depending on your redundancy requirement, that means a minimum of four NICs for both LAN and SAN dual-path redundancy.

Redundant connections follow the standard "IXI" setup (hosts H1, H2, SAN switches S1, S2, SAN controllers C1, C2):

  H1   H2
   |\ /|
   | X |
   |/ \|
  S1   S2
   |\ /|
   | X |
   |/ \|
  C1   C2

With a tight budget, you could combine one of the SAN links with a LAN link (for failover only), using VLANs. As a minimally redundant setup, you would use two NICs and two switches - one each primarily for LAN and for SAN, failing over to the other one.

However, the host configuration, iSCSI multipathing and failover setup is off-topic here. You should be able to get some help on Server Fault.

Also make sure that each iSCSI path uses its own IP subnet to avoid criss-cross sessions. Most initiators create a session between any of their ports and each target port in the same subnet (n:m mapping) - a reasonable fabric usually performs best with a 1:1 mapping between same-speed ports (see also multipath I/O, ALUA).

2
  • 1
    I think one could make a strong argument against but you won't want that., given the expense, but that's why we don't allow opinion-based questions.
    – Ron Trunk
    Jan 26, 2022 at 13:06
  • @RonTrunk Thanks for pointing that out - I have reworded accordingly. ;-)
    – Zac67
    Jan 26, 2022 at 13:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.