What is the benefit of a Cisco "top of rack" design?

Question

Aside from the obvious cabling mess you avoid (which I know for us as network engineers is great, but hard to use as an excuse when reasoning with someone who sees no point in it), what do you gain versus copper straight through to a larger core?

Context: - Price is of no concern

• Company had already decided on a dual Nexus core.

  • Option 1: Nexus 7004's, which would be nearly fully populated with 10G SFP+ and aggregate connections to several FEXs at the top of each rack in the DC, as well as aggregate SAN and various server connections

  • Option 2: Nexus 7009 cores that will be approx. 1/3 filled up with various modules to accommodate the aggregation of all fiber connections from all devices.

• This is a colocated data center

• Standard call center/enterprise domain related services hosted on the network

• QoS is a very important bulletpoint to emphasize given that this company is a call center

Problem:

• I am unable to justify going with Cisco's "top of rack" setup despite my wanting less of a cabling mess and a more modular design. I'm unable to do so because you are inserting a point of failure into the network. Doing this increases the latency (even if it's only by a small amount), etc. Not only that, but now that I think about it, since all FEXs rely on the Nexus to operate, you not only increase the chance of a hardware failure bringing down a block of devices, but now a software process that could wig out and cause the FEX to malfunction in some way.

So, before I put the top of rack design in the idea graveyard for this project, can anyone else see a reason not to go with a larger core and no FEXs given the lack of budget limitation?

Answer 1

Keep in mind that a FEX inherently is a method by which to extend the fabric, thus the name. The ability to manage centrally while still having "line cards" distributed throughout the DC is the real reason to use a FEX. Drastically reducing cabling is valuable to anyone, technical or not, and the argument of being able to manage the entire infrastructure at fewer points is time savings, pure and simple.

One of your big doubts is that you're worried about single points of failure. All devices have the ability to be dual-homed, and in certain configurations, you can even establish virtual port channels with the Nexus 2K FEXs themselves.

Take a look at Cisco's documentation. You'll find that you can design a topology that is just as redundant as the "direct-cable" option you're considering, with less fuss.

Answer 2

As for benefits, first cables get sloppy and when you have sloppy, problems will occur. I have seen infrastructure cabling go bad for a number of reason in a data center. Need more cables? Then someone is messing with the cable plant and something could get damaged. Dealing with nearly 400 cables plugged into a device leads to more accidental disconnects than 48. It's just far easier to manage.

Second, this helps to future proof. While there is 10Gbps copper, the distance limitations can be problematic depending on the situation. Additionally, 10G copper tends to draw additional power the longer you go.

Third, FEXes can be more easily replaced. Want to switch from 1Gbps copper to 10Gbps SFP+, just change the FEX. Your core stays the same and the configuration stays largely in place.

I don't see the negatives you provide, and only see benefits for doing so.

Depending on your data center setup, I would go with either two fabric extenders at the top of the rack or one (if servers can shared with neighboring racks). Servers should be connected to two separate extenders. Each fabric extender can be connected with FET's to both Nexus 7k (which should also be connected).

This should reduce your chance of failure. FEXes are an extension of the chassis (read designed for data center w/high MTBF) and more similar to a module in a 1U "body" as opposed to a secondary distribution or access device. They boot they boot the software off the core, so there is no software difference. You can lose a 7k or an extender without loss of service anywhere. Potentially a 7k and a number of extenders without losing service.

You can also then manage this as a single logical unit, allowing things like servers can actually do link aggregation even while connected to two different extenders, both increasing performance and reducing chance of failure.

I can't see how this would increase the latency in any way and it may actually improve it.

When you start using the more advanced features of Nexus, I can only see more benefits.

Ultimately, you need to make the choice for your own needs. But I will say this, if you research how the top internet companies run their data centers, you will find that most of the have some sort of top of rack deployment. They don't choose this because it increases their downtime or decreases performance. They do this because it reduces downtime, increases performance, and greatly increases manageability.

Edit: Consolidating from my comments so I can delete. The comment train is too long at present on this answer to be useful.

Answer 3

1.) Odds are good that if you're in colo space that you'd want to look at the 7010 with front-to-back airflow rather than the 7009 with side-to-side.

2.) One of the obvious points in the ToR vs centralized switching discussion is usually scalability. If your colo footprint is pretty much fixed then it's not much of a concern. If it's slated to grow in any appreciable manner then having the ability to expand the network in a rational manner should be a consideration. That said, I probably would be reluctant to use a 7004 as the concentration point for the FEX units if growth were a concern. The 7K can run to 48 extenders at the moment, and is likely to go higher in the future. If you're going to be at 6 cabinets for the duration, however, it doesn't much matter either way.

3.) The unknown here (at least based on the initial question) is the density of servers in the racks. If it's 6-8 4U's then the FEX is overkill. If it's many dozens of GE links from 1U's or blade pass-throughs then the cabling argument takes on a more serious cast. I've seen certain (dysfunctional) setups with 384+ cables to a single rack. It's not something I want to see again.

All-in-all the main difference between a small 7K hosting a bunch of FEX units and a larger 7K home-running those same connections isn't going to be tremendous at small scale. As has been mentioned above, the FEX just shows up as another line card on the chassis. With a very few exceptions features and functions of FEX ports are going to be equivalent to native ports and will be managed as such.

Also - Contrary to popular suspicion the performance penalty of using a FEX isn't significant if correctly designed. Arguments about latency are measured in microseconds (and the whole design is better addressed with a different platform altogether if this -is- an issue).

Answer 4

There isn't much difference (apart from the cost) between running cables directly to the core, or using fabric extenders in between.

1. If you cable your servers directly to the cores, you'll connect each server with two links, one to each core switch. This way even if one core switch fails, the other keeps the service up.
2. If you fabric extenders on top of each rack, Your servers are connected by two links to two different fabric extenders which are both connected to two core switches. The link between the FEX and Core Switch is a L1 link and the entire setup of fabric extenders behave as a single logical switch. The setup will not introduce additional STP nodes, hence there shouldn't be any more latency than the first option. For loss of connectivity, either both Core switches or both FEXs or their corresponding links should fail. The failure of a single FEX or Core Switch wouldnt affect the service. While the Fabric extenders are a relatively newer idea, the way the work is actually better than option 1.

Since you mentioned that you mentioned that budgets aren't an issue, you might want to size your Nexus 7Ks (and fiber connectivity) to have sufficient capacity to support a future upgrade to 40G or 100G. The FEXs can be installed to suit current requirements. Later on, in case you want to upgrade to 100G you'll just have to replace the FEXs without having to change your Nexus 7ks or the cabling.

Answer 5

Price is usually one of the big drivers for a "top of rack" design, and you've said cost is no object.

We've used it for two other reasons though that I hadn't seen listed yet: modularity or ease of deployment.

If you have a standard "rack" design you can build and test an entire rack (or group of racks) together as one module, or buy them ready-built. Then you just have to connect a few cables at the top, instead of re-connecting all the machines.

The other case with top of rack can make a lot of sense (or top of a few racks, depending on your application) is if you have a standard build configuration to deploy a "cell" of your infrastructure. Sometimes communication within a "cell" is high (for example: web server, app server, db server, image server etc). Not everyone has this sort of configuration, but it can be handy so you can performance characterize a cell, and scaling out means adding more cells instead of growing all your infrastructure (which can cause more performance suprises).

Answer 6

Ultimately, without a budget limitation, it doesn't make sense not to simply go with the 7009 design since there are less devices affected by the failure of a single line of fiber than of an entire fabric extender.

Again, the fabric extender is both an extra hardware and software point of failure within an environment that doesn't have a need for the extra port density beyond what the core device provides.