1

I'm scratching my head on what could cause the following behavior:

Scenario A: Laptop -> virtualizedpfsense -> switch -> physicalpfsense -> modem

speedtest.com gives me 84mbps download (which is expected: bottleneck from physicalpfsense's fast ethernet port.) (Interesting observation was that upload failed, I theorize(wild guess) maybe it's due to double NATing, but don't have enough experience to know.

Scenario B: Laptop -> virtualizedpfsense -> modem

speedtest.com and other websites are not reachable, google.com and other low bandwidth websites are reachable but slow to load. The virtualizedpfsense's web interface's traffic monitor showed max of 1mbps on WAN connection. Google.com's speed test eventually loaded and started at 0.05mb/sec then went to 0.01mb/sec, at which point I closed it.

For Clarity Sake: My Question is why is this weird behavior happening? Weird behavior is virtualizedpfsense gets 84mbps in Scenario A then <1mbps in Scenario B. Desired behavior is >84mbps in scenario B.

Additional Info: physicalpfsense is a laptop with a 2nd Ethernet port added on a PCI slot, it's 192.168.1.1 (Tried and true setup.) (The virtualizedpfsense was set to 192.168.9.1)

virtualizedpfsense is a Atom PC with 4 NICs running CentOS7 minimal with, virtualization enabled in bios, VT-d is not supported by hardware. yum install libvirt and openssh are the only packages installed. (virt-manager is installed on another PC for remote GUI management via statically configured Eth4.) Eth1 and Eth2 are configured so that IP addresses won't be assigned on KVM host, pfsense 2.4.1 x64 is running as a KVM guest, Eth1 and Eth2 are set to paravirtualized passthrough through the virt-manager GUI, so they map to virtualpfsense vEth1(WAN) and vETH2(LAN) (I've read that paravirtualization makes it so you don't need hardware support like VT-d. Paravirtualization is supposed to be near baremetal performance and Scenario A seems to confirm this.) (Also might be worth noting that I was able to have host have 1 IP and guest have another IP, on the same physical interface, that caused problems so I did "nano /etc/sysconfig/network-scripts/ifcfg-enp2s0" BOOTPROTO=none To make the host stop getting IP addresses on Eth 1 and Eth2. Both PFsense Firewall/Light Routing devices have near default bare-bones configurations. Just WAN via DHCP and NAT on LAN side.

Initial Troubleshooting:

  • Wiggled and Swapped out Patch Cables to check if the problem was physical
  • Switched back and forth between Scenario A and B 3 times with reboots in between just to see if it was a fluke.

Planned Troubleshooting/Follow up:

  • I remember reading that guest OS's need to be designed for use as paravirtualized guests for paravirtualization to be effective. Not sure if pfsense is or not, maybe it's not and Scenario A is a fluke? Still weird behavior.
  • Will test if similar behavior with VyOS (ultimate goal is VyOS and pfsense virtualized on atom box)
Improve this question
6
  • You need to edit your question to include the router configurations.
    – Ron Maupin ♦
    Commented Dec 27, 2017 at 9:19
  • I see you plan to use VyOs. For information I do use VyOs on top of KVM (on Debian) and get very good performance (up to 40Gbs). I also have this setup on some ATOM box for small customers and no performance issue (but for 1gbs, not 40 of course). BUT I did have very poor performance with Realtek-based NIC. What is the NIC chip?
    – JFL
    Commented Dec 27, 2017 at 10:18
  • Have you checked all the "obvious" things like duplex mismatch, port errors, etc?
    – Ron Trunk
    Commented Dec 27, 2017 at 18:21
  • I added configuration to question. Both boxes have bare bones default configurations, Eth1 = WAN via DHCP, Eth2 = LAN side configured for NAT. I'll check the NIC chip when I get home, good idea always hear bad things about Realtek, but could never map it to concrete reasoning. Are there Linux commands I can use to check port errors/duplex mismatch. Duplex mismatch is within the realm of possibility I know Gig Usually does duplex correctly, but pfsense bridge mode I know does asynchronous/half-duplex, I'm not sure if paravirtualization does half or full but a good path to follow up thanks.
    – neoakris
    Commented Dec 27, 2017 at 18:51
  • 1
    Also check QEMU cpu usage on the hypervisor and if there's offloading features activated on the NIC in pfsense and if yes deactivate them.
    – JFL
    Commented Dec 27, 2017 at 20:55

1 Answer 1

Reset to default
4

JFL was correct it was a NIC checksum offloading error. My guess is that when my virtual router was behind my physical router it somehow made it so it didn't need to do checksumming.

Recap of Problem:

  • Scenario A: virtual behind physical resulted in 88mbps (pfsense 2.4.2 as a KVMguest on CentOS 7 with default settings on the pfsense install.) (bottlenecked by physical pfsense laptop's PCI addon card.)

  • Scenario B: virtual replacing physical resulted in 0.01mbps

Solution:

  • I was able to get 80mbps in Scenario B by running the following on the KVMhost. 
ethtool --offload  enp1s0  rx off  tx off 
ethtool --offload  enp2s0  rx off  tx off

Notes:

  • Sadly that only resulted in 80mbps. (and doesn't persist between reboots, I'm sure I could figure out a way to make it persist between reboots but the performance makes this learning project not worth it.)
  • I also tried pfsense web GUI: System -> Advanced -> Networking -> Disable hardware checksum offload
  • And tried pfsense console: ee /boot/loader.conf (added the following entries to make pfsense aware that is running as a paravirtualized guest)

virtio_load="YES"
virtio_pci_load="YES"
if_vtnet_load="YES"
virtio_blk_load="YES"

  • In the end with all 3 optimizations, the speed was still just 80mbps. Apparently, FreeBSD doesn't have good virtio drivers, but VyOS (linux based) does have good virtio drivers and thus can get good performance, but at this point, I might as well run bare metal since I can't virtualize them both at acceptable performance levels. I guess in the future I could try this experiment again with a box that has vt-d support so I could do pci passthrough of the NICs, or maybe I can buy a ssd bigger than 16GB so I could see if pfsense plays well with HyperV's paravirtualization solutions.
Improve this answer
1
  • I don't have time to dive deeper into this but if anyone's interested. The network box I bought had an Atom PC that didn't support VT-d which immediately caused some performance issues, and it was brand new hardware so driver support may have been an issue. (I think KVM does best with VT-d hardware virtualization support, and probably benefits from good drivers.) I wonder if I would have fared better with Debian and Xen VM, Xen is what AWS uses and it doesn't need hardware virtualization support to get good performance, and it being a more mature product may handle drivers better?
    – neoakris
    Commented Mar 20, 2018 at 3:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.