Two VMWare ESXi hosts, each with 2 10GB NIC Storage with 2 10GB NIC
Where's the storage?
how would the ESXi hosts communicate with the LAN?
The host should use dedicated NICs for LAN and SAN. It is absolutely possible to share NICs, switches and even VLANs across LAN & SAN but that should be considered carefully. Keep in mind that storage is a vital asset in your infrastructure and it should be up and running at all times. Losing storage means losing data and halting production.
Separating VLANs is a requirement for basic security and resilience - even if you protect SAN connections by authentication and encryption it still opens a can of worms in regard of DoS and exposing all vulnerabilities to all users.
Using separate NICs saves your SAN links from any LAN faults - broadcast storms, bottlenecks, congestion and simplifies management. Clean separation is also an absolute life saver for troubleshooting.
Using separate switches minimizes risks from any instabilities that may come from normal LAN operations - think of configuring port or ACL policies. You can also separate administration entirely. Ideally, your SAN switches only have management links into your LAN.
For a fully redundant connection you need at least two NICs (each via a separate switch). Depending on your redundancy requirement, that means a minimum of four NICs for both LAN and SAN dual-path redundancy.
Redundant connections follow the standard "IXI" setup (hosts H1, H2, SAN switches S1, S2, SAN controllers C1, C2):
H1 H2
|\ /|
| X |
|/ \|
S1 S2
|\ /|
| X |
|/ \|
C1 C2
With a tight budget, you could combine one of the SAN links with a LAN link (for failover only), using VLANs. As a minimally redundant setup, you would use two NICs and two switches - one each primarily for LAN and for SAN, failing over to the other one.
However, the host configuration, iSCSI multipathing and failover setup is off-topic here. You should be able to get some help on Server Fault.
Also make sure that each iSCSI path uses its own IP subnet to avoid criss-cross sessions. Most initiators create a session between any of their ports and each target port in the same subnet (n:m mapping) - a reasonable fabric usually performs best with a 1:1 mapping between same-speed ports (see also multipath I/O, ALUA).
