I'm on a consulting engagement with a new client whose network infrastructure is comprised of slightly non-standard equipment.

During discovery, I inquired about the switching equipment and network topology. I was surprised to find that the main switches were running Debian, in the form of Cumulus Linux, atop Whitebox switch hardware (Edge-Core AS5610-52X).

enter image description here

The infrastructure is relatively small, but needs to run some critical public-facing applications and will also be moving to a VMware vSphere virtualization solution.

I'm in the picture because the environment hasn't been documented by its previous on-staff sysadmins. I'm working on my list of recommendations/risks, and I can't shake the feeling that the Cumulus Linux and Whitebox switch combination isn't a good idea. Granted, I come from a Cisco/HP/Arista background, but is it reasonable to suggest that the relatively low mindshare and vendor presence is a potential operational risk?

Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space? Is it the best choice for an environment that doesn't have the staff with the knowledge to maintain it?

I'd appreciate any thoughts on this.

  • 3
    FYI whether Cumulus is "gaining traction" has little to do with it being a good idea (or not). The issue normally boils down to A) what is the local staff comfortable managing, B) does Cumulus do what they need, C) is it stable enough? Jun 2, 2015 at 1:17
  • @MikePennington My assessment is that the staff is not comfortable with networking, Cumulus seems capable, but was clearly more of a religious choice by a Linux-focused sysadmin. Stability is there, but I wouldn't bet the business on it.
    – ewwhite
    Jun 2, 2015 at 1:19
  • Among the key data of your problem, specify on which type of equipment or system the staff of your client feels comfortable, perhaps skilled.
    – dan
    Jun 2, 2015 at 18:45
  • 1
    I would suggest removing "Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space?". It is totally opinion based.
    – Ryan Foley
    Jun 3, 2015 at 0:14
  • 1
    @ZaphodB Oh, I tore it all out and installed Arista switches. Both of the Edge-Core switches suffered flash failures at separate times and couldn't boot.
    – ewwhite
    May 30, 2017 at 15:58

3 Answers 3


is it reasonable to suggest that the relatively low mind share and vendor presence is a potential operational risk?

This is always an issue. Products from a small vendor could (and more often than not, do) end up abandoned. As long as they continue to function, there's little to worry about. But when they break, there's nowhere to go but somewhere else.

Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space?

Not in any professional spaces that I'm aware of. HP has "OpenFlow" SDN hardware, but it's all HP software, backed by HP.

Is it the best choice for an environment that doesn't have the staff with the knowledge to maintain it?

That's a universal NO. It's never wise to bank on hardware/software no one in the organization understands. Being able to "bring someone in" may sound like an option, but there's a good chance you'll be calling them more than you think. (I've been on both sides of that coin.)

  • Thanks. This helps put things in perspective. There's safety in going with well-recommended and endorsed solutions.
    – ewwhite
    Jun 2, 2015 at 15:13

Disclaimer: I work at Cumulus Networks

To address a few of the high level points/concerns raised:

1.) Is ONIE getting traction?

ONIE is the de facto method for getting software onto whitebox switches. It's been embraced by Dell, HP, Supermicro, Penguin, Facebook and a few others. It's run out of the Open Compute Project (http://www.opencompute.org/) an independent organization working to make datacenter equipment more open, more affordable and more interoperable. When this post was first written HP was mentioned; they didn't have open networking switches. Today HP has open networking switches with ONIE that run Cumulus.

2.) Is it stable? Can I trust it?

Sure. Cumulus is running some of the biggest clouds on the planet, as well as hospitals, police departments, school districts, financial services firms and everything in between. I'm sorry that you had a crash, but with every Cumulus License comes 24x7x365 support based in North Carolina, USA and Sydney, Australia. If you run Cumulus and run into problems they are happy to help!

3.) Coming from a Cisco/Arista background, I don't know what to think of this Linux stuff.

It's different and that can be a challenge for sure. Using FRR to configure BGP or OSPF on Cumulus will look very familiar to those with a Cisco background. To configure IP addresses, bonds (etherchannels), mLAG (vPC) or other L1/L2 features we have a lot of documentation to help you out http://docs.cumulusnetworks.com/display/DOCS/Cumulus+Linux+2.5.4+User+Guide

and again, if you get stuck you can try to search "how to configure an IP on Linux" and it's the same configuration for Cumulus, or call support!

4.) The staff doesn't have the skills they need

That's probably why they called in reinforcements for help on the networking side of the house. Personally, I come from a Cisco background and I think that network engineers and architects have a very important skill. We build networks that are scalable and fault tolerant. That's a skill a lot of server people do not have. But do we need CCIEs to configure VLANs or modify an ACL on day to day operations? I don't think so. Let's have network engineers build networks and systems/ops people operate systems (+ networks).

If there's still a gap or concern we offer full day bootcamps to help get you comfortable, or you can use our virtual edition, Cumulus-Vx (www.cumulusnetworks.com/cumulus-vx) and get a feel for how the software behaves.

If you're interested in hearing more, we'd love to win you back! The things you can do with Cumulus from an automation, configuration, choice and price standpoint make us a great product.


Pete Lumbis - CCIE #28677, CCDE 2012::3 [email protected]

  • Thanks for your answer. Unfortunately, the Cumulus/Edge-Core devices continued to be problems. One week after their support lapsed in July, both switches crashed hard: one exhausted disk space, while the other suffered an internal flash storage issue and subsequent filesystem corruption. I still have one unit limping along in production, but am hesitant to renew the support. Earlier attempts to obtain support were met with resistance because of the licensing status.
    – ewwhite
    Oct 22, 2015 at 19:20
  • 3
    @ewwhite - I'd love to get to the bottom of what happened with support, our policy is to help you get to a healthy state and deal with things like licenses after the fact. We'd love to offer you a free 1 year renewal for your switch as well as a free Cumulus Bootcamp seat for an 8 hour training session. If you want to send me an email, [email protected], we can discuss how to solve the issues you are experiencing today, from both a software and hardware standpoint, and what we can do to make things better going forward. Oct 22, 2015 at 20:48
  • Thank you for the followup, Pete. I'll be in touch to try to rectify the situation.
    – ewwhite
    Oct 23, 2015 at 23:50

is it reasonable to suggest that the relatively low mind share and vendor presence is a potential operational risk?

There is the potential for both risks and rewards with choosing such a solution. Will the company disappear leaving the product abandoned? Possibly. Are there greater chances of bugs/problems with the code that aren't resolved in a timely fashion? Likely.

But on the other hand, you could be on the cutting edge of new features, performance or other benefits. Larger vendors acquire smaller companies all the time to acquire innovative technology and intellectual property they develop. You could benefit from this innovation before it makes a wider impact on the industry. Even if it never does, there are many examples of technologies that may have been better but fade away due to other market considerations.

Additionally, most of these types "commodity" or "merchant" silicone products are delivering high performance devices at much lower costs than the more established players in the market. Each organizations needs to balance their costs vs. their risks and make their own judgement.

Is the Open Network Install Environment (ONIE) really gaining traction in the Network Engineering space?

ONIE really only applies to installing the OS on whitebox switches. As such, it likely isn't going to gain any real traction in more mainstream channels.

However, there are some large Internet based companies that use ONIE or their own version of such software to install the OS on their own network infrastructure, usually along with having the hardware manufactured for them.

Is it the best choice for an environment that doesn't have the staff with the knowledge to maintain it?

Frankly, there is no good choice in an environment that doesn't have the staff with the knowledge to maintain it. Without knowledgeable staff, even the best and most mainstream equipment will fail spectacularly.

This is where I am personally split on solutions such as Cumulus Linux. It is marketed as fitting within potential *nix skill sets that may already exist in an organization.

While this is true, I have my concerns. While most of the good *nix admins I have worked with have at least a decent understanding of networking, they aren't network engineers. Same as most of the good network engineers I know have a decent understanding of *nix. That doesn't mean I want a network engineer designing, implementing, operating, and troubleshooting my *nix environment, nor do I want a *nix admin doing the same for my network.

The flip side is that if you only have the resources to hire one or the other, it is the organization that has to make the decision on which is more important. If they do choose the *nix admin, then perhaps Cumulus Linux may be a better platform for them to operate than having to learn a new OS.

  • I visited the site today and took the time to see how the staff felt about the networking equipment. They are onboard with replacing it with HP or Cisco at this point. My monitoring software was unable to correctly define the device, but I discovered one of the Cumulus Linux switches had crashed and been down since November. The out-of-band management was not connected either.
    – ewwhite
    Jun 3, 2015 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.