pfSense - Trunking VLANs on LAGG vs. individual links

Question

We are a small business. I am running pfSense on a decommissioned developer's PC. We have also a Cisco SG 300 L3 switch, capable of VLANs and LAGG (link aggregation).

I am going to do RoaS (router on a stick) with 3 - 4 VLANs and a firewall between them.

The PC that is being used as a router has 2 integrated NICs and 4 available PCIe slots. All links are 1Gb.

I want to avoid congestion so I am planning to buy 4 additional PCIe 1Gb network cards and use either:

1. LAGG (link aggregation) on those 4 NICs and trunk the VLANs.
2. Put different VLANs directly to the 4 different NICs on the router.

What are the pros and cons of each approach?

If I go the LAGG way, which mode is best - LACP or LOADBALLANCE?

Thanks.

Answer 1

There is no clear answer - it depends on your workloads.

If all VLANs carry approximately the same traffic, using a separate connection per VLAN spreads the load evenly.

If load across the VLANs is uneven, a LAG with VLAN trunking might work better. Note that with a LAG, the transmitting device is responsible with traffic distribution: the SG300 distributes by L2 or L3 hash, so all traffic between the same IP endpoints into the pfSense always uses the same port.

On the pfSense (= traffic flowing into the SG300), LACP mode is a bit obscure. The docs state for communication between two single hosts it will only use one single port at a time because the client will only talk to one MAC address at a time which doesn't really say much. If traffic distribution is based on L3 addressing, it'll balance by IP endpoints. If MAC addresses it'll balance by L2 endpoint - which may or may not be a problem (e.g. with a router port on the other end).

"LoadBalance" mode might be better: Outbound traffic is load balanced based on all active ports in the LAGG using a hash computed using several factors, such as the source and destination IP address, MAC address, and VLAN tag. It requires a static LAG on the switch and is likely to distribute traffic in a better way.

Additionally, you might want to look for a quad-port NIC, saving a lot of space in your machine.