0

I have a fairly simply setup, using a PCEngines firewall running OPNSense and an EdgeCore ECS4620-28P L3 switch. The EdgeCore is doing InterVLAN routing and that works just fine, but I cannot get internet access on all VLANs simultaneously.

I want to have the following:

  • 5 separate VLANs on the switch
  • InterVLAN routing between certain VLANs e.g. 30 (desktop workstations) -> 50 (rdp machines)
  • Internet access for each VLAN running out a separate VPN each.
  • If I cannot have a VLAN per VPN, then even just getting all VLANs out one VPN would be great.

Network:

enter image description here


There are:

  • 5 VLANs: 10-50
  • Each VLAN is defined in the OPNSense firewall and I am routing each VLAN out a separate VPN using NAT
  • Each VLAN in the OPNSense firewall has an interface address for that VLAN e.g. 10.0.20.1, 10.0.10.1, etc...
  • Each VLAN is also defined (with ports tagged) on the EdgeCore, each VLAN has an interface
  • The default route for each PC is the EdgeCore VLAN interface for that VLAN e.g. 10.0.20.200, 10.0.10.200, etc...

If I set the default route on the PCs to the OPNSense VLAN Interface e.g. 10.0.10.1, it routes to the internet but not between VLANs (because the L3 switch interface is on 10.0.10.200). However, if I set the default route to 10.0.10.200, then I can access PCs on other VLANs but not the internet (unless I set the default route for one VLAN as shown below)

I'm pretty sure I'm wrong but right now I want the path to be:

  • PC#n (10.0.x.x/24) -> EdgeCore VLAN Interface (10.0.x.200) -> OPNSense VLAN Interface (10.0.x.1) -> Internet

But that just seems wrong because then there are essentially two interfaces for the same VLAN...


Here are my NAT settings inside OPNSense:

enter image description here


Here's the rub, I can traceroute from inside the EdgeCore at 10.0.20.200 -> 10.0.20.1 -> 8.8.8.8 when the static route below is defined:

enter image description here

As can any other VLAN provided it's next hope (i.e. 10.0.x.1) is defined in this manner. Now I know this is the default route, so I know that I cannot have one default route per VLAN at least not as defined above.

There is this post asking the same question about default routes per VLAN and the suggested fix was either policy-based routing or VRF-lite. The EdgeCore makes no mention of VRF in the manual and the only policy-based routing I can see is in reference to BGP.

Do I need to setup BGP? Or is the answer here in that the firewall needs a route back to my clients?

There is also this post which states needing an extra VLAN that takes care of routing between the switch and router, however I can't seem to make that work either.


Here is the ping response from one of the clients when configured with the EdgeCore as default route, with no default route set in the EdgeCore:

IP: 10.0.10.40/24 Default: 10.0.10.200 (EdgeCore VLAN Interface)

enter image description here

NOTE: It will just hang like that, there is no response of "Destination Unreachable" or anything at all in fact.

Here is the ping from one of the clients when configured in the .20 subnet with a default route set in the EdgeCore (0.0.0.0 -> 10.0.20.1):

enter image description here

Those DUP packets are another sign that the seemingly double VLAN interface is just not correct.

4
  • Each VLAN will have its own network and gateway address in the layer-3 switch, and the default gateway in a host is the gateway address of its VLAN network. The layer-3 switch should use a routed link to your firewall (OPNSense is off-topic here as the manufacturer does not offer optional, paid support). You can either set up static routes in the firewall, or you can run a routing protocol, e.g. OSPF, between the layer-3 switch and the firewall. Do not trunk the VLANs between switch and firewall, that should be a routed link.
    – Ron Maupin
    Jul 18 at 19:23
  • Hey @RonMaupin yes I understand as much that the default route on the clients should be to the interface on the switch, how else would it route? (I asked myself) haha but that's why I was so confused when it didn't then route through to the .1 interface address, I thought it might pick it up through dynamic routing. Ah ha, a routed link using OSPF or another routing protocol. Thank you! A direction was all I needed, I will dig further and learn more :)
    – BitShift
    Jul 18 at 19:28
  • 2
    If you are using a layer-3 switch as the LAN router, you do not trunk the VLANs beyond the layer-3 switch, the layer-3 switch terminates the VLANs. You set up a routed link between the switch and firewall, and you route the traffic to the WAN to the firewall..
    – Ron Maupin
    Jul 18 at 19:31
  • ahhh ok that makes sense... gonna do PBR with OSPF. Thanks again you're a legend!
    – BitShift
    Jul 19 at 13:26
3

Basically, you need to make up your mind which device your clients should use as default gateway: the OPNsense - then inter-VLAN routing goes that way, too - or the L3 switch, that in turn uses the OPNsense as default gateway.

If you use the OPNsense for inter-VLAN routing, you set up firewall rules for which connections are permitted. Note that the OPNsense is stateful by default, meaning that connections across it consume internal resources.

Using the L3 switch as client-side default gateway has the advantage of (likely) higher-speed internal routing, but you'll need ACLs to permit the wanted and deny unwanted traffic. ACLs are generally stateless and require definition for both directions. Stateless also means that the switch doesn't care how many connections run across it, in contrast to the OPNsense.

Also, the L3 switch sends all traffic to the same gateway, regardless of source VLAN - make sure you define your firewall rules with that in mind. Make also sure the OPNsense accepts traffic from directly connected subnets from a different VLAN, that's likely not default. Alternatively - and recommended -, you can remove the trunk from the OPNsense-switch link and use an access port with a single VLAN (and remove the VLANs from the firewall interface).

In order to route out of different interfaces, depending on source IP address/subnet you'll need to set up policy-based routing (PBR).

1
  • Awesome thanks @Zac67. The current situation is actually a migration from using OPNSense as an interVLAN router, hence the cruft of leftover VLANs on that device haha... Yes ACLs till the cows come home that's for sure!! L3 switch for routing absolutely and I'm gonna use OSPF with PBR in place of the trunk..
    – BitShift
    Jul 19 at 13:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.