Layer 3 Switch Routing vs Router on a Stick

Question

CORE: Dell N4032 (6.3.0.18) Access: Dell N2048 (6.3.1.13)

We are in the process of setting up a serviced office that will serve multiple businesses. Each "customer" will naturally be on their own VLAN, and each VLAN will require access to the Internet and some other VLANs.

There will be some open VLANs like a "Public Print" VLAN and so forth. There will also be "Server" VLANs which potentially all other VLANs will need to access (or some may not, i.e. filtered)

We have core L3 switches, 20 L2 Access Switches and two HA firewalls.

The question is whether we do the routing for the VLANs on the L3 switches themselves, or via the Firewall.

My thoughts - if we route via the firewall (i.e. Router on a Stick), all inter-VLAN traffic is going to have to go via our Firewall device, which will naturally put on a load on it's overall throughput (especially with DPI enabled). The pro being it's a hell of a lot easier to configure the firewalling between the VLANs this way.

If we route via the L3 switch, we benefit from the full throughput of the firewall for the WAN (which will be a 1Gbps connection), but the con being having to use "Policy Based Routing" on the 2 Core Switches which is generally a pain to administer.

Is there a best practice in a situation like this? What should I consider to make a decision?

UPDATE

The issue using the ACLs on the switch is as follows.

• There is a maximum of 100 ACLs
• We will have "Customer VLANs" that will require access to the Internet, to a VOIP VLAN, a Server VLAN and Print VLAN.
• They however should not have access to the other "Customer VLANs".

Problem is all "Customers" will be within one /16 - i.e.

[Parent subnet being 10.100.0.0/16]
Customer 1 - 10.100.0.0/24
Customer 2 - 10.100.1.0/24
Customer 3 - 10.100.2.0/24

I need to apply the ACLs on the VLAN so when traffic hits that VLAN it is blocked from accessing other customer VLANs.

Currently I have something like this:

1000 deny 10.100.0.0 0.0.255.255 192.168.50.0 0.255.255.255 
1010 deny 10.100.0.0 0.0.255.255 192.168.60.0 0.255.255.255  
1020 permit any any 

This is then applied to all Customer VLAN interfaces. This solves the issue of blocking access to Server VLANs and other private VLANs, however, this doesn't prevent these VLANs from then accessing other customer VLANs (within the 10.100.0.0/16).

Problem is we could end up having 50 plus customers, which would mean an inconceivable number of ACLs (given each ACL can only have 10 entries, and each ACL would need an ACE for every other customer). This hits the max number of ACLs the switch supports almost instantly.

I just can't see another way of doing this at this stage without using our firewall as a "Router on a Stick". Any advice is greatly appreciated.

Update #2

Strange a I attempted this today with an ACL on a Customer VLAN as below (applied "IN" on the interface)

1000 deny 10.100.0.0 0.0.255.255 192.168.50.0 0.255.255.255 
1010 deny 10.100.0.0 0.0.255.255 192.168.60.0 0.255.255.255 
1015 deny 10.100.0.0 0.0.255.255 10.100.0.0 0.0.255.255 
1020 permit any any 

UPDATE #3

When applying suggested ACL below - it prevents access between devices on the same VLAN. The ACL is L3:

ip access-list Block_Customers
 1000 deny ip any 10.100.0.0 0.0.255.255
 1020 permit any any
!

However, then, two devices on this VLAN, with IPs that can speak to each other i.e. 10.100.1.0/24, can no longer ping one another. If I remove the ACL, they once again can ping each other.

This is wrong right, my traffic on one VLAN is L2, why is this L3 ACL applying?

UPDATE #4

If I apply the ACL as suggested and...

• Two devices on the same VLAN on different switches, they cannot ping in each other (without the ACL they can ping so the trunks must be working). They can ping the Internet.

• Two devices on the same VLAN, on the same switch, they can ping each other, but can no longer ping the SVI IP Address on the core switch on that VLAN. They can however ping the Internet, so traffic can reach that IP.

Answer 1

My suggestion is to keep it simple. You seem to be trying to add more complexity to a scenario that could be simplified. This is so far what you are doing, please correct me if I'm wrong:

• Isolating customers by using VLANs.
• Removing that isolation by enabling routing between VLANs.
• Asking what will be the best way to isolate them based on your available resources.

My point is to not route traffic between VLANs by default.

Suggested solutions:

• Use VRFs on the L3 cores: A VRF per VLAN. This way you could keep the isolation, but provide access to common resources. You might even want to consider doing NAT between VLANs to enable (controlled) communications between them. Why NAT? Because you might also want to use the same IP addressing space on all customer's VLANs - making new deployments easier for you (they're repeatable, standardized)
• Instead of VRFs, a physical router per VLAN and repeat the above.

Answer 2

L3 and even some L2 services (such as DHCP) are not just a pain to administrate on any given switch directly; they're typically dangerously clumsy.

I can't recommend administrating the vast majority of switches directly beyond very small implementations. Instead, you might consider abstracting their interfaces away via an SDN solution such as OpenDaylight (there are many others - this is just an example). An SDN system will use "drivers" to render your standard and friendly configurations into configurations that the given switch will understand. This can vastly help you to standardize your control plane in a safe and highly effective manner.

In addition to that, CM platforms like SaltStack or Puppet have switch support that may work well for you. This won't give you as complete an experience that an SDN platform would, but it's a step in the right direction.

A router on a stick is to be avoided, though. In small stacks it's fine, but yours sounds too big for those shoes.

Answer 3

It appears that you are using some type of Cisco layer-3 switch. It is extremely easy to deny traffic from a network to all the other networks contained within a supernet. If all the customer network are in 10.100.0.0/16, you can have a single ACL that will block one customer from the other customers' networks. Simply apply it inbound to the layer-3 switch on the layer-3 interfaces (SVIs). For example:

ip access-list extended Block_Customers
 deny ip any 10.100.0.0 0.0.255.255
 permit any any
!
interface Vlan10
 ip address 10.100.10.1 255.255.255.0
 ip access-group Block_Customers in
!
interface Vlan11
 ip address 10.100.11.1 255.255.255.0
 ip access-group Block_Customers in
!
interface Vlan12
 ip address 10.100.12.1 255.255.255.0
 ip access-group Block_Customers in
!
interface Vlan13
 ip address 10.100.13.1 255.255.255.0
 ip access-group Block_Customers in
!

and so on.

This will prevent any traffic destined for the 10.100.0.0/16 network from entering the router (layer-3, not layer-2) part of the switch from the customer networks. This only affects layer-3, and it has absolutely no affect on any traffic within the network, even if the network extends across multiple switches (a bad practice, by the way).