I'm in the process of over-hauling our network, the issue I keep coming back to is: trying to bring layer 3 to the core, while still having a centralised firewall.
The main issue I have here is that the "mini core" switches I've been looking at always have low in-hardware ACL limits, which even at our current size we could quickly hit. I'm currently about to (hopefully) purchase a pair of EX4300-32Fs for the core, but I've looked at other models, and other options from Juniper and Brocade's ICX range. They all seem to have the same low ACL limits.
This makes perfect sense as the core switches need to be able to maintain the wire-speed routing, so don't want to sacrifice too much through ACL processing. So I can't do all my firewalling on the core switches themselves.
However, we do mostly fully managed servers, and having a centralised (stateful) firewall helps a lot with that management - as we can't have customers talking directly to each other. I'd like to keep it that way if we can, but I feel as though most ISP networks would not do this kind of thing, hence why in most cases it would be straight-forward to do the routing in the core.
For reference, here is the topology I would ideally like to do (but not sure where to fit the FW obviously).
Right now, we have a router-on-a-stick configuration. This enables us to do NAT, stateful firewalling and VLAN routing all in the one spot. Very simple.
I could continue on with (roughly) the same solution by extending L2 all the way to the "top" of our network - the border routers. But then I lose all the benefits of the wire-speed routing that the core can offer me.
IIRC the core switches can do 464 Gbps of routing while my border routers will be able to offer maybe 10 or 20 Gbps if I'm lucky. This isn't technically a problem right now, but more an issue of growth. I feel as though if we don't design the architecture to leverage core routing capacity now, it's going to be painful to redo everything when we're larger and need to leverage that capacity. I'd rather get it right the first time.
Layer 3 to Access
I thought that perhaps I could extend L3 to the access switches, and thus break up the firewall rules into smaller segments that would then fit on the hardware limits of the access switch ACLs. But:
- As far as I know, these would not be stateful ACLs
- L3 to Access, to me, seems much more inflexible. Server moves or VM migrations to other cabinets would be more painful.
- If I'm going to be managing a firewall at the top of each rack (only six of them), I probably want automation anyway. So at that point it's not much of a leap to automate management of firewalls at the host level. Thus avoiding the entire issue.
Bridged/transparent firewalls on each uplink between access/core
This would have to involve multiple firewall boxes, and significantly increase the required hardware. And could end up more expensive than buying larger core routers, even using plain old Linux boxes as firewalls.
Giant core routers
Could buy a larger device that can do the firewalling I need and has a much larger routing capacity. But really don't have the budget for it, and if I'm trying to make a device do something it's not designed for, I'll probably have to go to a much higher spec. than I would otherwise.
No Centralised firewall
Since I'm jumping through hoops, maybe this isn't worth the effort. It's always been a nice thing to have, and at times a selling point for customers who want a "hardware" firewall.
But it seems that having a centralised firewall for your "entire" network is infeasible. I'm wondering, then, how larger ISPs can offer hardware firewall solutions to customers with dedicated servers, when they have hundreds or even thousands of hosts?
Can anyone think of a way of solving this issue? Either something I've missed entirely, or a variation on one of the ideas above?
Based on @Ron's suggestion, I stumbled on this article which explains pretty well the issue I'm facing, and a good way to solve the problem.
Unless there's other suggestions, I would say this now is a product recommendation type of problem, so I suppose that's the end of it.