5

Were setting up a pfsense box as a virtual machine inside a VMWare ESXi 6.0 environment (inside a VXRail hyper-converged Box).

The target configuration is that to access any machine within this box, you need to go through the Firewall. Traffic between machines within the same box also need to go through the firewall. Since they are different servers e.g. Web Server, Database Server, the VMs are setup to be in different VLANs.

As such, we've setup pfSense with two interfaces. One is the WAN that will be used by the "Outside World" to communicate to servers within the ESXI environment and the other is a Trunk that should then connect to all the VLANs protected by the pfSense box.

ESXi:

We've followed the guide here to setup a Distributed vSwitch in VxRail. we've setup a distributed port group? of VLAN Type VLAN Trunking and with VLAN IDs 0-200:

enter image description here

To our understanding, this Port Group is what we will attach to pfsense Trunk so that it is able to "read" all the VLAN tagged traffic?

Trunk on PfSense:

To create a trunk on pfsense is basically adding the NIC to the pfsense VM. The NIC should be the Port Group we've created above.

enter image description here

VLAN on pfSense:

After that we create a VLAN on pfSense and add a VLAN ID. This VLAN is sitting on the Trunk we've created above. Example below:

enter image description here

We then add an interface based on this VLAN and give it an IP of 192.168.152.1

enter image description here

enter image description here

Protected Machine:

We then create a machine that will be protected by the firewall. So first, we add an NIC to it. The NIC is based on a host network that has a VLAN tag e.g. 152 as seen below.

enter image description here

enter image description here

We then assign the protected machine an IP of 192.168.152.10 with a default gateway of 192.168.152.1.

Problem Statement:

Issue is, after doing all this, the protected machine cannot ping its default gateway. The default gateway cannot ping that machine. It's like there is no communication between them at all. We've added a firewall rule to allow all traffic on Interface152 and logged everything but we cannot see any traffic being accepted or rejected.

What could we have missed? The biggest confusion we have is on the VXRail ESXI setup but any correction on pfSense setup is also welcome.

Improve this question
8
  • You can avoid using the virtual VLAN trunk into the VM. Instead, create a port group for each VLAN and add a NIC for each port group. Works a charm.
    – Zac67 ♦
    Jun 14, 2017 at 17:59
  • If you mean add an NIC to the firewall for each Port Group, then that we've tried and it works. Issue is, its not sustainable because it means every time you have a new VLAN you have to restart the firewall in order to add the NIC. We really really want to avoid that.. Any other ideas?
    – lukik
    Jun 15, 2017 at 0:55
  • If you know which VLANs are bound to show up you could add them in advance. ;-) I haven't used a dSwitch yet but it should be a driver issue.
    – Zac67 ♦
    Jun 15, 2017 at 7:26
  • That's possible but we are a growing setup so would really really like to avoid adding an NIC every time we have a change in VLANs. Else, we raised the issue with EMC last night as its a VXRail box and they said its not up their alley so they've escalated to VMWare... Waiting for the VMWare team to guide us. I will definitely post the solution here when we do get it.
    – lukik
    Jun 15, 2017 at 8:48
  • Is the trunk port on the exsi connected to trunk switch port ? you also need to confirm that the protected vm is connected to a trunk port that allows vlan 152, or at least an access port configure for vlan 152. The switch-port configuration seems to be the missing link.
    – Oshin
    Jul 18, 2017 at 17:04

3 Answers 3

Reset to default
1

If you add a distributed portgroup with VLAN 4095, that port group will keep 802.1Q frame tags set by pfSense virtual machine.

This configuration is known in vSphere parlance as VGT or Virtual Guest Tagging, see KB 1003806 and KB 1004252.

Improve this answer
1

I've done a bit of research/testing as I was trying to set up decent packet capturing: as documented, VGT (port group with VLAN ID 4095) forwards all VLANs from a vSwitch (and probably dSwitch as well) to the VM.

However (poorly documented): in Windows, both VMXNET3 and E1000 will - by default - receive all frames from all VLANs but both will also remove all Q tags. Frames being sent out remain untagged, so they'll go to the untagged VLAN on the physical switch.

When you configure a VLAN ID in Advanced Settings on the vNIC, the vNIC starts to filter, only receives frames for that VID (untagged), and tags all outbound frames with this VID. This is exactly the same for VMXNET3 and E1000.

If you install the Intel PROSet utilities with the E1000 vNIC you can set up virtual vNIC instances inside the guest that each provide connectivity to a single VLAN (just like on a physical machine).

Therefore, accessing the full trunk and seeing the 802.1Q tags in packet capture doesn't seem possible in any way in Windows. My impression is that this is a limitation of the virtual vNIC hardware/driver/OS and not of the vSwitch.

Linux and FreeBSD on the other hand have native VLAN support, using subinterfaces for each VID. My guess is that with VGT you could actually access the VLAN trunk and see the tags in captured frames. I'll try this some time and keep you posted.

Improve this answer
1

I looked over your configuration, and it seems proper on the pfsense side, that's exactly what I'm doing in my own lab with pfsense, and I have several VMs in production with that type of VLAN trunks.

The issue seems to be that you're using host networking for your VM. I'd configure a new port group on the dSwitch with the settings VLAN and VLAN ID 152, and assign the NIC to that.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.