I have a fairly simply setup, using a PCEngines firewall running OPNSense and an EdgeCore ECS4620-28P L3 switch. The EdgeCore is doing InterVLAN routing and that works just fine, but I cannot get internet access on all VLANs simultaneously.
I want to have the following:
- 5 separate VLANs on the switch
- InterVLAN routing between certain VLANs e.g. 30 (desktop workstations) -> 50 (rdp machines)
- Internet access for each VLAN running out a separate VPN each.
- If I cannot have a VLAN per VPN, then even just getting all VLANs out one VPN would be great.
- 5 VLANs: 10-50
- Each VLAN is defined in the OPNSense firewall and I am routing each VLAN out a separate VPN using NAT
- Each VLAN in the OPNSense firewall has an interface address for that VLAN e.g. 10.0.20.1, 10.0.10.1, etc...
- Each VLAN is also defined (with ports tagged) on the EdgeCore, each VLAN has an interface
- The default route for each PC is the EdgeCore VLAN interface for that VLAN e.g. 10.0.20.200, 10.0.10.200, etc...
If I set the default route on the PCs to the OPNSense VLAN Interface e.g. 10.0.10.1, it routes to the internet but not between VLANs (because the L3 switch interface is on 10.0.10.200). However, if I set the default route to 10.0.10.200, then I can access PCs on other VLANs but not the internet (unless I set the default route for one VLAN as shown below)
I'm pretty sure I'm wrong but right now I want the path to be:
- PC#n (10.0.x.x/24) -> EdgeCore VLAN Interface (10.0.x.200) -> OPNSense VLAN Interface (10.0.x.1) -> Internet
But that just seems wrong because then there are essentially two interfaces for the same VLAN...
Here are my NAT settings inside OPNSense:
Here's the rub, I can traceroute from inside the EdgeCore at 10.0.20.200 -> 10.0.20.1 -> 184.108.40.206 when the static route below is defined:
As can any other VLAN provided it's next hope (i.e. 10.0.x.1) is defined in this manner. Now I know this is the default route, so I know that I cannot have one default route per VLAN at least not as defined above.
There is this post asking the same question about default routes per VLAN and the suggested fix was either policy-based routing or VRF-lite. The EdgeCore makes no mention of VRF in the manual and the only policy-based routing I can see is in reference to BGP.
Do I need to setup BGP? Or is the answer here in that the firewall needs a route back to my clients?
There is also this post which states needing an extra VLAN that takes care of routing between the switch and router, however I can't seem to make that work either.
Here is the ping response from one of the clients when configured with the EdgeCore as default route, with no default route set in the EdgeCore:
IP: 10.0.10.40/24 Default: 10.0.10.200 (EdgeCore VLAN Interface)
NOTE: It will just hang like that, there is no response of "Destination Unreachable" or anything at all in fact.
Here is the ping from one of the clients when configured in the .20 subnet with a default route set in the EdgeCore (0.0.0.0 -> 10.0.20.1):
DUP packets are another sign that the seemingly double VLAN interface is just not correct.