Trying to fit centralised firewall into network topology

Question

I'm in the process of over-hauling our network, the issue I keep coming back to is: trying to bring layer 3 to the core, while still having a centralised firewall.

The main issue I have here is that the "mini core" switches I've been looking at always have low in-hardware ACL limits, which even at our current size we could quickly hit. I'm currently about to (hopefully) purchase a pair of EX4300-32Fs for the core, but I've looked at other models, and other options from Juniper and Brocade's ICX range. They all seem to have the same low ACL limits.

This makes perfect sense as the core switches need to be able to maintain the wire-speed routing, so don't want to sacrifice too much through ACL processing. So I can't do all my firewalling on the core switches themselves.

However, we do mostly fully managed servers, and having a centralised (stateful) firewall helps a lot with that management - as we can't have customers talking directly to each other. I'd like to keep it that way if we can, but I feel as though most ISP networks would not do this kind of thing, hence why in most cases it would be straight-forward to do the routing in the core.

For reference, here is the topology I would ideally like to do (but not sure where to fit the FW obviously).

Curent Solution

Right now, we have a router-on-a-stick configuration. This enables us to do NAT, stateful firewalling and VLAN routing all in the one spot. Very simple.

I could continue on with (roughly) the same solution by extending L2 all the way to the "top" of our network - the border routers. But then I lose all the benefits of the wire-speed routing that the core can offer me.

IIRC the core switches can do 464 Gbps of routing while my border routers will be able to offer maybe 10 or 20 Gbps if I'm lucky. This isn't technically a problem right now, but more an issue of growth. I feel as though if we don't design the architecture to leverage core routing capacity now, it's going to be painful to redo everything when we're larger and need to leverage that capacity. I'd rather get it right the first time.

Possible Solutions

Layer 3 to Access

I thought that perhaps I could extend L3 to the access switches, and thus break up the firewall rules into smaller segments that would then fit on the hardware limits of the access switch ACLs. But:

• As far as I know, these would not be stateful ACLs
• L3 to Access, to me, seems much more inflexible. Server moves or VM migrations to other cabinets would be more painful.
• If I'm going to be managing a firewall at the top of each rack (only six of them), I probably want automation anyway. So at that point it's not much of a leap to automate management of firewalls at the host level. Thus avoiding the entire issue.

Bridged/transparent firewalls on each uplink between access/core

This would have to involve multiple firewall boxes, and significantly increase the required hardware. And could end up more expensive than buying larger core routers, even using plain old Linux boxes as firewalls.

Giant core routers

Could buy a larger device that can do the firewalling I need and has a much larger routing capacity. But really don't have the budget for it, and if I'm trying to make a device do something it's not designed for, I'll probably have to go to a much higher spec. than I would otherwise.

No Centralised firewall

Since I'm jumping through hoops, maybe this isn't worth the effort. It's always been a nice thing to have, and at times a selling point for customers who want a "hardware" firewall.

But it seems that having a centralised firewall for your "entire" network is infeasible. I'm wondering, then, how larger ISPs can offer hardware firewall solutions to customers with dedicated servers, when they have hundreds or even thousands of hosts?

Can anyone think of a way of solving this issue? Either something I've missed entirely, or a variation on one of the ideas above?

UPDATE 2014-06-16:

Based on @Ron's suggestion, I stumbled on this article which explains pretty well the issue I'm facing, and a good way to solve the problem.

Unless there's other suggestions, I would say this now is a product recommendation type of problem, so I suppose that's the end of it.

http://it20.info/2011/03/the-93-000-firewall-rules-problem-and-why-cloud-is-not-just-orchestration/

Answer 1

I'd pick one of two choices:

Individual per-tenant Virtual Firewalls

Pros:

• Horizontally scalable
• Spin-up and spin-down
• Relatively immune to future topology/design changes
• Complete customer separation/isolation

Cons:

• Unless you enforce a standard template, you now have n individual firewalls to manage
• You now have n individual firewalls to monitor
• You now have n individual firewalls to patch

Large firewall chassis/cluster with a routing-instance/context per tenant

Deploy a large central firewall (cluster) hanging off to the side of your core, and use an internal and an external routing-instance to route traffic over and back to it (e.g.: default gateway on Internal instance is the firewall, default gateway on the firewall is your external instance on the core and the default for the external instance is your border(s).)

Pros:

• Single box to manage and configure
• Single box to monitor
• Single box to patch
• Customer separation

Cons:

• Day one cost will be higher
• No scaling down
• Depending on configuration inter-customer traffic may start routing across your border routers

Answer 2

What core switches are you running? Policies are usually done at the distribution layer, if you are going with a collapsed Core design, then the core should be able to handle your requirements. Also, are you liking for statefull inspection or just acls. If you have any compliance you need to follow, acls may not be enough.

Personally, I would go with a firewall, perhaps look for one that can be clustered so you can cluster each together and maintain centrally managed rule base, such as a sourcefire firewall.