Cumulus Linux and Whitebox switch hardware?

Question

I'm on a consulting engagement with a new client whose network infrastructure is comprised of slightly non-standard equipment.

During discovery, I inquired about the switching equipment and network topology. I was surprised to find that the main switches were running Debian, in the form of Cumulus Linux, atop Whitebox switch hardware (Edge-Core AS5610-52X).

The infrastructure is relatively small, but needs to run some critical public-facing applications and will also be moving to a VMware vSphere virtualization solution.

I'm in the picture because the environment hasn't been documented by its previous on-staff sysadmins. I'm working on my list of recommendations/risks, and I can't shake the feeling that the Cumulus Linux and Whitebox switch combination isn't a good idea. Granted, I come from a Cisco/HP/Arista background, but is it reasonable to suggest that the relatively low mindshare and vendor presence is a potential operational risk?

Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space? Is it the best choice for an environment that doesn't have the staff with the knowledge to maintain it?

I'd appreciate any thoughts on this.

Answer 1

is it reasonable to suggest that the relatively low mind share and vendor presence is a potential operational risk?

This is always an issue. Products from a small vendor could (and more often than not, do) end up abandoned. As long as they continue to function, there's little to worry about. But when they break, there's nowhere to go but somewhere else.

Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space?

Not in any professional spaces that I'm aware of. HP has "OpenFlow" SDN hardware, but it's all HP software, backed by HP.

Is it the best choice for an environment that doesn't have the staff with the knowledge to maintain it?

That's a universal NO. It's never wise to bank on hardware/software no one in the organization understands. Being able to "bring someone in" may sound like an option, but there's a good chance you'll be calling them more than you think. (I've been on both sides of that coin.)

Answer 2

Disclaimer: I work at Cumulus Networks

To address a few of the high level points/concerns raised:

1.) Is ONIE getting traction?

ONIE is the de facto method for getting software onto whitebox switches. It's been embraced by Dell, HP, Supermicro, Penguin, Facebook and a few others. It's run out of the Open Compute Project (http://www.opencompute.org/) an independent organization working to make datacenter equipment more open, more affordable and more interoperable. When this post was first written HP was mentioned; they didn't have open networking switches. Today HP has open networking switches with ONIE that run Cumulus.

2.) Is it stable? Can I trust it?

Sure. Cumulus is running some of the biggest clouds on the planet, as well as hospitals, police departments, school districts, financial services firms and everything in between. I'm sorry that you had a crash, but with every Cumulus License comes 24x7x365 support based in North Carolina, USA and Sydney, Australia. If you run Cumulus and run into problems they are happy to help!

3.) Coming from a Cisco/Arista background, I don't know what to think of this Linux stuff.

It's different and that can be a challenge for sure. Using FRR to configure BGP or OSPF on Cumulus will look very familiar to those with a Cisco background. To configure IP addresses, bonds (etherchannels), mLAG (vPC) or other L1/L2 features we have a lot of documentation to help you out http://docs.cumulusnetworks.com/display/DOCS/Cumulus+Linux+2.5.4+User+Guide

and again, if you get stuck you can try to search "how to configure an IP on Linux" and it's the same configuration for Cumulus, or call support!

4.) The staff doesn't have the skills they need

That's probably why they called in reinforcements for help on the networking side of the house. Personally, I come from a Cisco background and I think that network engineers and architects have a very important skill. We build networks that are scalable and fault tolerant. That's a skill a lot of server people do not have. But do we need CCIEs to configure VLANs or modify an ACL on day to day operations? I don't think so. Let's have network engineers build networks and systems/ops people operate systems (+ networks).

If there's still a gap or concern we offer full day bootcamps to help get you comfortable, or you can use our virtual edition, Cumulus-Vx (www.cumulusnetworks.com/cumulus-vx) and get a feel for how the software behaves.

If you're interested in hearing more, we'd love to win you back! The things you can do with Cumulus from an automation, configuration, choice and price standpoint make us a great product.

HTH,

Pete Lumbis - CCIE #28677, CCDE 2012::3 [email protected]

Answer 3

is it reasonable to suggest that the relatively low mind share and vendor presence is a potential operational risk?

There is the potential for both risks and rewards with choosing such a solution. Will the company disappear leaving the product abandoned? Possibly. Are there greater chances of bugs/problems with the code that aren't resolved in a timely fashion? Likely.

But on the other hand, you could be on the cutting edge of new features, performance or other benefits. Larger vendors acquire smaller companies all the time to acquire innovative technology and intellectual property they develop. You could benefit from this innovation before it makes a wider impact on the industry. Even if it never does, there are many examples of technologies that may have been better but fade away due to other market considerations.

Additionally, most of these types "commodity" or "merchant" silicone products are delivering high performance devices at much lower costs than the more established players in the market. Each organizations needs to balance their costs vs. their risks and make their own judgement.

Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space?

ONIE really only applies to installing the OS on whitebox switches. As such, it likely isn't going to gain any real traction in more mainstream channels.

However, there are some large Internet based companies that use ONIE or their own version of such software to install the OS on their own network infrastructure, usually along with having the hardware manufactured for them.

Is it the best choice for an environment that doesn't have the staff with the knowledge to maintain it?

Frankly, there is no good choice in an environment that doesn't have the staff with the knowledge to maintain it. Without knowledgeable staff, even the best and most mainstream equipment will fail spectacularly.

This is where I am personally split on solutions such as Cumulus Linux. It is marketed as fitting within potential *nix skill sets that may already exist in an organization.

While this is true, I have my concerns. While most of the good *nix admins I have worked with have at least a decent understanding of networking, they aren't network engineers. Same as most of the good network engineers I know have a decent understanding of *nix. That doesn't mean I want a network engineer designing, implementing, operating, and troubleshooting my *nix environment, nor do I want a *nix admin doing the same for my network.

The flip side is that if you only have the resources to hire one or the other, it is the organization that has to make the decision on which is more important. If they do choose the *nix admin, then perhaps Cumulus Linux may be a better platform for them to operate than having to learn a new OS.