I need to set up a modest iSCSI SAN with a few ESX hosts and one storage array. Can I get by using a Catalyst 2960-S or do I need to upgrade? Are there any special tweaks I need to make to the switch in order to handle iSCSI? The switch will be dedicated to the SAN; no other traffic except management will be present.
Given that the Cat2960-S is a desktop/access switch, with very, very small buffers, you would likely experience a lot of output drops. A datacenter switch, such as a 4948E, would be a better choice for an iSCSI application.
To understand the reasoning behind this, you have to remember that an ethernet switch is either transmitting, or not transmitting on a specific port. If traffic arrives on port 1 for port 2, and port 3 is already sending traffic to port 2, the traffic from port 1 has to be buffered until there's a gap in the traffic from 3 to 2. If the buffer fills, additional traffic will be dropped. The term "microburst" is used to reference traffic that, over time, is well below the limit for the interface, but occasionally bursts to cause output drops.
I am not as familiar with the 2960-S platform, but enabling QoS (without extensive tuning, see comments below) on it would probably be a bad idea; that would actually increase the number of output drops. Enabling QoS splits your very small buffers into 4 even smaller buffers, and most traffic will hit only one of them.
I followed Jeremy Stretch's post here a while back, moving our iSCSI SAN off a production switch onto a dedicated 3750G (stack of 2). While I can't speak specifically to the 2960S, the tweaks offered in that post seem to be doing well on the 3750G's. I've got a handful of OutDiscards (less than 100) but for 38 weeks of uptime, that seems pretty good for me.
This is running 4 ESXi hosts connected to 4 JetStor 1Gbps SANs and all links are aggregated.
I'll happily forward a "sh int counters" if you are interested.
(Ha! Just noticed it was Jeremy who asked this...sorry for linking your own blog post!)
I've had experience where customers have chosen to use the lower end switches (3750, 2960) for iSCSI and they are pretty terrible.
The issue gets even worse when you have your storage connected at a higher speed (like 10gbit for the SAN and 1gbit for the end servers).
I would look towards more of the DC class switches, if you want to stick with Catalyst then there is the 4900 series, alternatively you could venture in to the Nexus product line but it really depends on your price constraints.
The Best Practice Guide for HP Lefthand iSCSI Storage stated, "For optimal switch performance, its recommended that the switch have at least 512 KB of buffer cache per port. Consult your switch manufacturer specifications for the total buffer cache. For example, if the switch has 48x 1 Gb ports, the recommendation is to have at least 24 MB of buffer cache dedicated to those ports. If the switch aggregates cache among a group of ports (for example, 1 MB of cache per 8 ports), space your storage modules and servers appropriately to avoid cache oversubscription."
I couldn't tell you what buffer cache the 2960S has per port, as it doesn't seem to be stated on the datasheet.