Network Engineering Stack Exchange is a question and answer site for network engineers. It only takes a minute to sign up.
Sign up to join this community
3560/3750s have small buffers and make good wiring closet switches. However I very often see these switches sitting in DCs. Lots of folks tend to use them as they are generally 1Gb and L3 capable.
Is there a way to prove just how bad they are in DC deployments? I quite often hear people saying they removed their 3750s as soon as they could, but I've yet to hear an actual failure scenario that could be used to let management know to get them out.
FWIW I've had experience with 3750's (3750G, and then later 3750E/3560E) at scale in a TOR setup; initially with L2 port-channels/GLBP (variants of 2x1G and 2x2G and the rare 2x4G for db racks) and then with L3 to the TOR (went with this for 3750E/3560E and 10G to the core). I'm talking thousands of them. We only saw issues with buffers for the most bandwidth intensive services, and at that point we were looking at 10G to the host anyway (and dense pizza boxes with 24-48 SFP+'s).
Whether or not you're going to be able to prove anything to management is really going to depend on the application and you doing your homework on what the requirements of your design and the application are, and knowing exactly what the specifications are of the application, as well as the expected growth velocity of it. Set up a design review with your management chain as well as the primary owners/customers of the network.
Management wants to see data, and if you don't have the resources to fully test the box (come up with a test plan, connect it up to some traffic generation hardware, fully scope it out and stress test it to the design spec, etc) this is going to be hard to do. They're not going to be impressed with anecdotal evidence, and finding this kind of hard data may prove difficult, since I'm sure that folks publishing this kind of thing would violate all kinds of NDA's.
Everyone else that's posted an answer to this has outlined the 3750 platform's "problem areas" pretty nicely: stacking and weird failure modes inherent to it, buffer sizes, etc. There's also this question that outlines the issues with gathering SNMP statistics on output queue drops - the buffers are shared across the ASICs, so any stats you get with this via SNMP are going to be the same for specific port ranges (this could be one sticking point you could bring up with your management chain).
To summarize, I'd say that the 3750/3560 would be "fine" for most deployments, even at somewhat large scales. Avoid stacking them if you can, but I'd say that it's not too horrible to do this in very small and manageable quantities.
It really depends on your deployment scenario. 3560/3750s are great switches, have decent buffers, and they usually work fine for most applications. If your data center sees traffic flows that require larger buffers, you should be able to pull statistics from the switches, like buffer usage and packet drops. Convincing management to drop the switches that are dropping their packets shouldn't be too much of a challenge. I think.
In the early days of the 3750, especially the stacking technology that was released right before 2010 or so, there were a lot of problems with switch failures causing the stack to fail in a not-so-graceful fashion. Combine that with the fact that upgrading a stack was not the most intuitive process (it's improved since then), the 3750 really got a bad reputation that has stuck ever since.
In small data centers, the 3750 stack represents a relatively low-cost option to get the port density without the cost of a chassis-based switch. I myself just installed for a smaller customer a data center solution involving a few Cisco UCS C220 M3 servers with a Netapp FAS2240, and I used a stack of 3750s to provide multi-chassis etherchannel redundancy to each new device as well as all their old servers during the transition. It worked really, really well.
So - does the 3750 have it's issues? Probably the same as any other switch that's been around for this long. The 6500 had it's problems early on in it's lifecycle, and now that it's been out for years and years it's not so bad. I recommend looking at what you're going to be throwing at it, and if the performance metrics hold up, then make sure you monitor their performance with vigilance.
Honestly, the most common way that I've seen the 3750's hit the curb, was when the core switches were upgraded to Nexus 7k's. Usually (but not always) part of that refresh is to move TOR to Nexus 2000 FEXs or Nexus 5000s.
Even though the 3750's don't have the largest buffers, in most people's minds, they work "well enough" in most enterprise DC environments.
Unless you can put a dollar value on the problems caused by having 3560's/3750's in a DC, I doubt you'd be able to convince management to replace them outside of a regular product refresh cycle.
@mellowd is certainly right, these switches are not very usable DC switches, due to very limited buffers they will microburst and drop traffic.
Consider you have 2 * 1GE ingress and 1 * 1GE egress. Worst case scenario is, that egress port starts dropping after the ingress ports have sent at the same time for 2ms. Best case scenario is, that you can handle 8ms burst.
You have 2MB of egress buffer per 4 ports, so 2MB/(1Gbps/8) = 16ms maximum and 16/4 = 4ms minimum. Divide that number by amount of ingress ports wanting to send, and you get the number of how long you can handle it. That is, the more ingress ports (servers) you add, the less microbursting you can handle.
If you must live with 3750/3560, you should read this document to maximize buffer use. And if you're still dropping use LACP on egress, even though your graphs show that average egress demand is very low.
To prove to your managers that the buffers are insufficient monitor/tap/span your current networks switches all downlinks, then you'll have timestamps and packet-sizes going to egress and you can calculate how much over 1Gbps your instantaneous demand is and how much buffer you'll need to handle it.
Performance is certainly an important issue and is well-addressed above but there's also a lot of differentiation based on features and feature sets:
The need for external RPS units is a huge problem in many installations – a 1U switch becomes more expensive in terms of initial costs, lost space and ongoing management. Redundant power should be considered an absolute must in all but the smallest data center environments.
Lots and lots of unnecessary code for end user connectivity is running – more opportunity for defects, security issues and downtime.
DC features (ISSU, DCB, storage, certain on-box scripting elements) aren't – and won't be – on the campus-focused devices. Mechanisms to manage and scale L2 extension in a sane way also (i.e. FabricPath/TRILL, OTV, VXLAN, etc) also tend to be missing from both the present state and roadmaps outside of DC products. The list here is only going to grow – virtualization on-box, support of HW-assist mechanisms, etc.
Scalability – How do you grow the infrastructure? Lots and lots of switches (expensive to manage)? Stacking (operationally difficult, major cabling issues) is a mess. Additionally the flexibility of interface types (fiber vs copper, for example) at density can be challenging.
In general the differences between DC and closet switching are growing. In the Cisco world there are distinct operating systems (NXOS vs IOS) for very good reasons – vastly different requirements yield divergent solutions. The feature velocity for user authentication mechanisms (802.1x) or fancy AV integration aren't needed in the data center while the ability to terminate tons of 10GE isn't needed in the wiring closet. Different tools for different jobs. A Nexus box connecting desktops would also be a less than ideal plan.
I'd also point you to the various design guides (CVD's, etc) that lay out rationale for the types of switches used at various points in the network. There's something to be said for solutions that generally resemble best common practice in the industry, and the switches you're mentioning generally have no place in the DC – apart from management networks or certain special case local connectivity situations.
I've got a customer that has deployed them as a SAN switch stack (using 3750X's) with the SAN connected at 10Gbit and then their ESX hosts connected at Gbit (or multiple Gbit's using LAG) and the amount of output drops is astronomical no matter how you try and tune the buffers.
The same customer has two other 3750 stacks in the same DC for other networks and these are all clean.
TL;DR: It really depends on the type of traffic you are going to be putting through the stack and where your bottlenecks are.
The power supplies / fans within 3560 / 3750 are not hot swappable / once the switch is mounted and the inevitable failure of these devices occurs all servers must be unplugged from the 3560/3750 while it is unmounted and replaced with the RMA.
Also the fan direction on the 3560s / 3750s become a problem with hot aisle / cold aisle and other cooling setups. Mounting the switches where the switch ports face the back of the servers creates a situation where the switch fans blow in the wrong direction. This overheats the switch which makes it more likely to fail / need replacement.
