Aside from the obvious cabling mess you avoid (which I know for us as network engineers is great, but hard to use as an excuse when reasoning with someone who sees no point in it), what do you gain versus copper straight through to a larger core?

Context: - Price is of no concern

  • Company had already decided on a dual Nexus core.

    • Option 1: Nexus 7004's, which would be nearly fully populated with 10G SFP+ and aggregate connections to several FEXs at the top of each rack in the DC, as well as aggregate SAN and various server connections

    • Option 2: Nexus 7009 cores that will be approx. 1/3 filled up with various modules to accommodate the aggregation of all fiber connections from all devices.

  • This is a colocated data center

  • Standard call center/enterprise domain related services hosted on the network

  • QoS is a very important bulletpoint to emphasize given that this company is a call center


  • I am unable to justify going with Cisco's "top of rack" setup despite my wanting less of a cabling mess and a more modular design. I'm unable to do so because you are inserting a point of failure into the network. Doing this increases the latency (even if it's only by a small amount), etc. Not only that, but now that I think about it, since all FEXs rely on the Nexus to operate, you not only increase the chance of a hardware failure bringing down a block of devices, but now a software process that could wig out and cause the FEX to malfunction in some way.

So, before I put the top of rack design in the idea graveyard for this project, can anyone else see a reason not to go with a larger core and no FEXs given the lack of budget limitation?

  • I don't have a great answer, but fault isolation comes to mind. Commented May 26, 2013 at 21:12
  • Yeah - I considered that as well. Ultimately it's fault isolation at the cost of creating the risk of more faults.
    – skrumcd
    Commented May 26, 2013 at 21:24

6 Answers 6


Keep in mind that a FEX inherently is a method by which to extend the fabric, thus the name. The ability to manage centrally while still having "line cards" distributed throughout the DC is the real reason to use a FEX. Drastically reducing cabling is valuable to anyone, technical or not, and the argument of being able to manage the entire infrastructure at fewer points is time savings, pure and simple.

One of your big doubts is that you're worried about single points of failure. All devices have the ability to be dual-homed, and in certain configurations, you can even establish virtual port channels with the Nexus 2K FEXs themselves.

Take a look at Cisco's documentation. You'll find that you can design a topology that is just as redundant as the "direct-cable" option you're considering, with less fuss.

  • We could debate single points of failure all the way down to the pattern of electrons crossing the wire, so I guess I just see adding another one, especially one governed by a single set of devices, of which are probably running the same software and could thus have the same bugs, as unnecessary when not limited by a budget. I guess you helped me answer my own question :)
    – skrumcd
    Commented May 26, 2013 at 22:01
  • Indeed. A line card is just as much a point-of-failure as a FEX. ('tho it's easier to disconnect the FEX)
    – Ricky
    Commented May 27, 2013 at 1:14
  • 1
    Think about where your failure domains lie. Can you tolerate single servers vs. single racks vs. a row of racks failing before becoming overly concerned? That understanding drives the solution to meet your design goals and dictates the level of reliability you need to obtain at each point (server, rack, rows) in the network. Commented May 27, 2013 at 2:02

As for benefits, first cables get sloppy and when you have sloppy, problems will occur. I have seen infrastructure cabling go bad for a number of reason in a data center. Need more cables? Then someone is messing with the cable plant and something could get damaged. Dealing with nearly 400 cables plugged into a device leads to more accidental disconnects than 48. It's just far easier to manage.

Second, this helps to future proof. While there is 10Gbps copper, the distance limitations can be problematic depending on the situation. Additionally, 10G copper tends to draw additional power the longer you go.

Third, FEXes can be more easily replaced. Want to switch from 1Gbps copper to 10Gbps SFP+, just change the FEX. Your core stays the same and the configuration stays largely in place.

I don't see the negatives you provide, and only see benefits for doing so.

Depending on your data center setup, I would go with either two fabric extenders at the top of the rack or one (if servers can shared with neighboring racks). Servers should be connected to two separate extenders. Each fabric extender can be connected with FET's to both Nexus 7k (which should also be connected).

This should reduce your chance of failure. FEXes are an extension of the chassis (read designed for data center w/high MTBF) and more similar to a module in a 1U "body" as opposed to a secondary distribution or access device. They boot they boot the software off the core, so there is no software difference. You can lose a 7k or an extender without loss of service anywhere. Potentially a 7k and a number of extenders without losing service.

You can also then manage this as a single logical unit, allowing things like servers can actually do link aggregation even while connected to two different extenders, both increasing performance and reducing chance of failure.

I can't see how this would increase the latency in any way and it may actually improve it.

When you start using the more advanced features of Nexus, I can only see more benefits.

Ultimately, you need to make the choice for your own needs. But I will say this, if you research how the top internet companies run their data centers, you will find that most of the have some sort of top of rack deployment. They don't choose this because it increases their downtime or decreases performance. They do this because it reduces downtime, increases performance, and greatly increases manageability.

Edit: Consolidating from my comments so I can delete. The comment train is too long at present on this answer to be useful.

  • I think you're under the impression that the FEXs are providing more than they are - We can just as easily multihome the servers as well as the FEXs.
    – skrumcd
    Commented May 26, 2013 at 21:21
  • Yes, but at it's core, we are arguing between the idea of running a cable straight to the core versus running it to FEX devices that could fail.
    – skrumcd
    Commented May 26, 2013 at 21:37
  • So just to be clear - price is of no concern, nor is port density as that will be covered by the remaining 5-6 slots on each nexus. Knowing that, would you still opt to put a fex between the servers and the core?
    – skrumcd
    Commented May 26, 2013 at 21:39
  • 4
    I agree with YLearn, FEX's top of the rack going to back to the core is going to be better in the long run. We have found that we weren't able to fill a full chassis as we can couldn't get any more cables in the hole in the bottom of the rack. I think you are more likely to have an outage due to the mass of cabling in the core network rack than a device failing, It's easy to knock a cable or two when your looking for something or running in new cables also it's easier and quicker to replace a 2m patch lead than it is a 20m cable back to the core should a cable fail.
    – Epaphus
    Commented May 26, 2013 at 23:05
  • 3
    I've gone in to too many places without a proper TOR switch setup. It's error prone, messy, and leads to more outages. Ultimately this isn't an argument - it's opinions based on several people's real world experience. You seem pretty dead set on wanting to connect directly to the 7Ks. That's fine. Will it work? Yes. If you don't touch your cabling much it'll work just fine. Just keep this in mind when it's 2AM and you're running long arus cable between racks and you're like "Why didn't I get TOR switches?" It's not the initial cabling that sucks. It's every single one after that.
    – bigmstone
    Commented May 27, 2013 at 1:30

1.) Odds are good that if you're in colo space that you'd want to look at the 7010 with front-to-back airflow rather than the 7009 with side-to-side.

2.) One of the obvious points in the ToR vs centralized switching discussion is usually scalability. If your colo footprint is pretty much fixed then it's not much of a concern. If it's slated to grow in any appreciable manner then having the ability to expand the network in a rational manner should be a consideration. That said, I probably would be reluctant to use a 7004 as the concentration point for the FEX units if growth were a concern. The 7K can run to 48 extenders at the moment, and is likely to go higher in the future. If you're going to be at 6 cabinets for the duration, however, it doesn't much matter either way.

3.) The unknown here (at least based on the initial question) is the density of servers in the racks. If it's 6-8 4U's then the FEX is overkill. If it's many dozens of GE links from 1U's or blade pass-throughs then the cabling argument takes on a more serious cast. I've seen certain (dysfunctional) setups with 384+ cables to a single rack. It's not something I want to see again.

All-in-all the main difference between a small 7K hosting a bunch of FEX units and a larger 7K home-running those same connections isn't going to be tremendous at small scale. As has been mentioned above, the FEX just shows up as another line card on the chassis. With a very few exceptions features and functions of FEX ports are going to be equivalent to native ports and will be managed as such.

Also - Contrary to popular suspicion the performance penalty of using a FEX isn't significant if correctly designed. Arguments about latency are measured in microseconds (and the whole design is better addressed with a different platform altogether if this -is- an issue).


There isn't much difference (apart from the cost) between running cables directly to the core, or using fabric extenders in between.

  1. If you cable your servers directly to the cores, you'll connect each server with two links, one to each core switch. This way even if one core switch fails, the other keeps the service up.
  2. If you fabric extenders on top of each rack, Your servers are connected by two links to two different fabric extenders which are both connected to two core switches. The link between the FEX and Core Switch is a L1 link and the entire setup of fabric extenders behave as a single logical switch. The setup will not introduce additional STP nodes, hence there shouldn't be any more latency than the first option. For loss of connectivity, either both Core switches or both FEXs or their corresponding links should fail. The failure of a single FEX or Core Switch wouldnt affect the service. While the Fabric extenders are a relatively newer idea, the way the work is actually better than option 1.

Since you mentioned that you mentioned that budgets aren't an issue, you might want to size your Nexus 7Ks (and fiber connectivity) to have sufficient capacity to support a future upgrade to 40G or 100G. The FEXs can be installed to suit current requirements. Later on, in case you want to upgrade to 100G you'll just have to replace the FEXs without having to change your Nexus 7ks or the cabling.

  • The Fabric extender doesnt have it's own software or configuration. It is a plug and play device that downloads the software from the parent Nexus switches. So I don't think you can consider it an additional software point of failure. Commented May 26, 2013 at 22:41
  • 1
    Sure you can. A process that runs on your Nexus parent is doing something to detect a fex, do the flash update when its detected, ensure that the fex is properly recognized as a line card, control/configure the FEXs as such, etc.
    – skrumcd
    Commented May 27, 2013 at 15:39
  • And how is that any different than a chassis solution? They run a process to detect when a module is inserted, check/upgrade the flash on the module (and daughter card if necessary), ensure the module is properly recognized, apply the configuration, etc.
    – YLearn
    Commented Aug 15, 2013 at 17:51

Price is usually one of the big drivers for a "top of rack" design, and you've said cost is no object.

We've used it for two other reasons though that I hadn't seen listed yet: modularity or ease of deployment.

If you have a standard "rack" design you can build and test an entire rack (or group of racks) together as one module, or buy them ready-built. Then you just have to connect a few cables at the top, instead of re-connecting all the machines.

The other case with top of rack can make a lot of sense (or top of a few racks, depending on your application) is if you have a standard build configuration to deploy a "cell" of your infrastructure. Sometimes communication within a "cell" is high (for example: web server, app server, db server, image server etc). Not everyone has this sort of configuration, but it can be handy so you can performance characterize a cell, and scaling out means adding more cells instead of growing all your infrastructure (which can cause more performance suprises).


Ultimately, without a budget limitation, it doesn't make sense not to simply go with the 7009 design since there are less devices affected by the failure of a single line of fiber than of an entire fabric extender.

Again, the fabric extender is both an extra hardware and software point of failure within an environment that doesn't have a need for the extra port density beyond what the core device provides.

  • 1
    I disagree with this stance as this is an old data center mindset. IIRC Google, Microsoft, Netflix, Facebook, and many more disagree with this stance. Feel free to research how they do it, as most have provided some details of their data centers publicly.
    – YLearn
    Commented May 27, 2013 at 2:55
  • 1
    @YLearn, without taking sides here, a design for the big boys isn't necessarily correct for a small data center, though I would give more weight to what they are doing. Commented Jun 10, 2013 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.