I'm scratching my head on what could cause the following behavior:
Scenario A: Laptop -> virtualizedpfsense -> switch -> physicalpfsense -> modem
speedtest.com gives me 84mbps download (which is expected: bottleneck from physicalpfsense's fast ethernet port.) (Interesting observation was that upload failed, I theorize(wild guess) maybe it's due to double NATing, but don't have enough experience to know.
Scenario B: Laptop -> virtualizedpfsense -> modem
speedtest.com and other websites are not reachable, google.com and other low bandwidth websites are reachable but slow to load. The virtualizedpfsense's web interface's traffic monitor showed max of 1mbps on WAN connection. Google.com's speed test eventually loaded and started at 0.05mb/sec then went to 0.01mb/sec, at which point I closed it.
For Clarity Sake: My Question is why is this weird behavior happening? Weird behavior is virtualizedpfsense gets 84mbps in Scenario A then <1mbps in Scenario B. Desired behavior is >84mbps in scenario B.
Additional Info: physicalpfsense is a laptop with a 2nd Ethernet port added on a PCI slot, it's 192.168.1.1 (Tried and true setup.) (The virtualizedpfsense was set to 192.168.9.1)
virtualizedpfsense is a Atom PC with 4 NICs running CentOS7 minimal with, virtualization enabled in bios, VT-d is not supported by hardware. yum install libvirt and openssh are the only packages installed. (virt-manager is installed on another PC for remote GUI management via statically configured Eth4.) Eth1 and Eth2 are configured so that IP addresses won't be assigned on KVM host, pfsense 2.4.1 x64 is running as a KVM guest, Eth1 and Eth2 are set to paravirtualized passthrough through the virt-manager GUI, so they map to virtualpfsense vEth1(WAN) and vETH2(LAN) (I've read that paravirtualization makes it so you don't need hardware support like VT-d. Paravirtualization is supposed to be near baremetal performance and Scenario A seems to confirm this.) (Also might be worth noting that I was able to have host have 1 IP and guest have another IP, on the same physical interface, that caused problems so I did "nano /etc/sysconfig/network-scripts/ifcfg-enp2s0" BOOTPROTO=none To make the host stop getting IP addresses on Eth 1 and Eth2. Both PFsense Firewall/Light Routing devices have near default bare-bones configurations. Just WAN via DHCP and NAT on LAN side.
Troubleshooting doneInitial Troubleshooting:
- Wiggled and Swapped out Patch Cables to check if the problem was physical
- Switched back and forth between Scenario A and B 3 times with reboots in between just to see if it was a fluke.
Troubleshooting I'll do later this weekPlanned Troubleshooting/Follow up:
- I remember reading that guest OS's need to be designed for use as paravirtualized guests for paravirtualization to be effective. Not sure if pfsense is or not, maybe it's not and Scenario A is a fluke? Still weridweird behavior.
- Will try with pfsense 2.3.5 x64 (older stable version, to see if it's a bug in new code)
- Will test if similar behavior with VyOS (ultimate goal is VyOS and pfsense virtualized on atom box)