Struggling with interVLAN routing + multiple VPNs + default routes on multi-layer switch

Question

I have a fairly simply setup, using a PCEngines firewall running OPNSense and an EdgeCore ECS4620-28P L3 switch. The EdgeCore is doing InterVLAN routing and that works just fine, but I cannot get internet access on all VLANs simultaneously.

I want to have the following:

• 5 separate VLANs on the switch
• InterVLAN routing between certain VLANs e.g. 30 (desktop workstations) -> 50 (rdp machines)
• Internet access for each VLAN running out a separate VPN each.
• If I cannot have a VLAN per VPN, then even just getting all VLANs out one VPN would be great.

---

Network:

---

There are:

• 5 VLANs: 10-50
• Each VLAN is defined in the OPNSense firewall and I am routing each VLAN out a separate VPN using NAT
• Each VLAN in the OPNSense firewall has an interface address for that VLAN e.g. 10.0.20.1, 10.0.10.1, etc...
• Each VLAN is also defined (with ports tagged) on the EdgeCore, each VLAN has an interface
• The default route for each PC is the EdgeCore VLAN interface for that VLAN e.g. 10.0.20.200, 10.0.10.200, etc...

If I set the default route on the PCs to the OPNSense VLAN Interface e.g. 10.0.10.1, it routes to the internet but not between VLANs (because the L3 switch interface is on 10.0.10.200). However, if I set the default route to 10.0.10.200, then I can access PCs on other VLANs but not the internet (unless I set the default route for one VLAN as shown below)

I'm pretty sure I'm wrong but right now I want the path to be:

• PC#n (10.0.x.x/24) -> EdgeCore VLAN Interface (10.0.x.200) -> OPNSense VLAN Interface (10.0.x.1) -> Internet

But that just seems wrong because then there are essentially two interfaces for the same VLAN...

---

Here are my NAT settings inside OPNSense:

---

Here's the rub, I can traceroute from inside the EdgeCore at 10.0.20.200 -> 10.0.20.1 -> 8.8.8.8 when the static route below is defined:

As can any other VLAN provided it's next hope (i.e. 10.0.x.1) is defined in this manner. Now I know this is the default route, so I know that I cannot have one default route per VLAN at least not as defined above.

There is this post asking the same question about default routes per VLAN and the suggested fix was either policy-based routing or VRF-lite. The EdgeCore makes no mention of VRF in the manual and the only policy-based routing I can see is in reference to BGP.

Do I need to setup BGP? Or is the answer here in that the firewall needs a route back to my clients?

There is also this post which states needing an extra VLAN that takes care of routing between the switch and router, however I can't seem to make that work either.

---

Here is the ping response from one of the clients when configured with the EdgeCore as default route, with no default route set in the EdgeCore:

IP: 10.0.10.40/24 Default: 10.0.10.200 (EdgeCore VLAN Interface)

NOTE: It will just hang like that, there is no response of "Destination Unreachable" or anything at all in fact.

Here is the ping from one of the clients when configured in the .20 subnet with a default route set in the EdgeCore (0.0.0.0 -> 10.0.20.1):

Those DUP packets are another sign that the seemingly double VLAN interface is just not correct.

Answer 1

Basically, you need to make up your mind which device your clients should use as default gateway: the OPNsense - then inter-VLAN routing goes that way, too - or the L3 switch, that in turn uses the OPNsense as default gateway.

If you use the OPNsense for inter-VLAN routing, you set up firewall rules for which connections are permitted. Note that the OPNsense is stateful by default, meaning that connections across it consume internal resources.

Using the L3 switch as client-side default gateway has the advantage of (likely) higher-speed internal routing, but you'll need ACLs to permit the wanted and deny unwanted traffic. ACLs are generally stateless and require definition for both directions. Stateless also means that the switch doesn't care how many connections run across it, in contrast to the OPNsense.

Also, the L3 switch sends all traffic to the same gateway, regardless of source VLAN - make sure you define your firewall rules with that in mind. Make also sure the OPNsense accepts traffic from directly connected subnets from a different VLAN, that's likely not default. Alternatively - and recommended -, you can remove the trunk from the OPNsense-switch link and use an access port with a single VLAN (and remove the VLANs from the firewall interface).

In order to route out of different interfaces, depending on source IP address/subnet you'll need to set up policy-based routing (PBR).