How to solve high ping "spikes" to pfSense when ipSec connections are active?

Question

Background information:

We have latency/freeze problems with our pfSense. To be able to answer the question i will first give some general background information:

• We run pfSense V2.6.0 on following hardware:
  • Supermicro mainboard
  • Intel(R) Pentium(R) CPU N3710 / AES-NI CPU Crypto: Yes (inactive) / QAT Crypto: No
  • 4Gb memory
• CPU load never exceed 50% usage(spikes) / Iddle 5-8%
• Memory never exceeds 15% usage
• Swap usage always 0%
• We have around 45 active ipSec connections

Our problem:

We have an internal PBX (3CX) which we use with their (smartphone) app when we are out of the office. We noticed hick ups/freezes in the conversations so we started searching for the cause.

We noticed that pfSense has sometimes really high ping latencies. When i ping pfSense from my desktop the latency is in general 1ms. Sometimes we see spikes from 2000+ms (see image) These spikes correspond with the freezings during a phone call. Note that the spike occur randomly.

After some testing we noticed that disabling all ipSec connections fixes these spikes. The ipSec connections are only used for really small amounts of data which are send periodically between internal an customer sytems. Most of the time there is no data send over the connections.

We also monitored our incoming and outgoing bandwidth and its never even close to our DSL limits.

ipSec configurations

The most ipSec connections use the same configuration. For the investigation we disabled all different (ca. 5) configurations but still have the issue. The "default" ipSec configuration we use:

<phase1>
        <ikeid>*********</ikeid>
        <iketype>ikev2</iketype>
        <interface>wan</interface>
        <remote-gateway>*********</remote-gateway>
        <protocol>inet</protocol>
        <myid_type>myaddress</myid_type>
        <myid_data></myid_data>
        <peerid_type>fqdn</peerid_type>
        <peerid_data>*********</peerid_data>
        <encryption>
            <item>
                <encryption-algorithm>
                    <name>aes</name>
                    <keylen>128</keylen>
                </encryption-algorithm>
                <hash-algorithm>sha256</hash-algorithm>
                <dhgroup>14</dhgroup>
            </item>
        </encryption>
        <lifetime>28800</lifetime>
        <pre-shared-key>*********</pre-shared-key>
        <private-key></private-key>
        <certref></certref>
        <caref></caref>
        <authentication_method>pre_shared_key</authentication_method>
        <descr>*********</descr>
        <nat_traversal>on</nat_traversal>
        <mobike>off</mobike>
        <closeaction></closeaction>
        <startaction>none</startaction>
    </phase1>
    <phase2>
        <ikeid>*********</ikeid>
        <uniqid>*********</uniqid>
        <mode>tunnel</mode>
        <reqid>39</reqid>
        <localid>
            <type>network</type>
            <address>*********</address>
            <netbits>16</netbits>
        </localid>
        <remoteid>
            <type>network</type>
            <address>*********</address>
            <netbits>24</netbits>
        </remoteid>
        <protocol>esp</protocol>
        <encryption-algorithm-option>
            <name>aes</name>
            <keylen>128</keylen>
        </encryption-algorithm-option>
        <encryption-algorithm-option>
            <name>aes128gcm</name>
            <keylen>128</keylen>
        </encryption-algorithm-option>
        <hash-algorithm-option>hmac_sha256</hash-algorithm-option>
        <pfsgroup>14</pfsgroup>
        <lifetime>3600</lifetime>
        <pinghost>*********</pinghost>
        <descr>*********</descr>
    </phase2>

Question: How can we solve these high pings/freezes?

Answer 1

Watch the output of 'top -aSH' in the pfsense command line to see what process is using a lot of CPU at the time of the spike. If it is reliable on a certain amount of time (every 30 seconds or every 5 minutes, etc.) then it might be a cron entry on the system or some scheduled task or feature in PFSense. You would need to check with PFSense to make sure your system is updated completely and see if there are any open bugs related to recurring CPU load spikes.

I don't know about that number of IPsec sessions, that is a lot for a small platform but if the CPU is generally under low load, it is probably ok to keep that hardware unless you can determine that the timing of the spikes coincides with actual bandwidth use. You may have 45 tunnels in use but they might not be passing hardly any traffic most of the time, resulting in very low CPU load. If some user is doing something every few minutes that causes a traffic spike and corresponding CPU spike then the issue is related to hardware being underpowered.

Also, I would agree that you should be using AES-NI hardware acceleration. Apparently you have to activate it manually, or at least check to make sure it is enabled.

https://forum.netgate.com/topic/124357/aes-in-inactive/3

It may make a huge improvement in performance under actual load.

Answer 2

We tried if stronger hardware solved our issue. We cloned our pfSense to hardware with a Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz. The 100% CPU spikes where still there but the ping response time peeks where max 600ms instead of the max 3000ms. This means stronger hardware doesn't solve the 100% CPU spikes but made the issue less noticeable during e.g. phonecalls.

As the stronger hardware was not the solution (even if it took of some pressure to solve this issue) we monitored our pfSense intensively. We noticed that pfctl is the cause of the the high pings as it spikes CPU usage exactly at the time when the high ping occurs.

We found a bugreport on the pfSense redmine (https://redmine.pfsense.org/issues/12045) which fits to our findings. This line Yes, we are aware, but 2.6.0 will get the fix when we do a full sync with FreeBSD sources next, which wasn't an option for 2.5.2 (so we picked only the necessary commits there). showed us that we are probably not the first/only one. This Reddit topic is also related: https://www.reddit.com/r/PFSENSE/comments/nz8fmw/high_cpu_usage_sbinpfctl_ss/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

As they mention that the issue will be fixed with a full FreeBSD sync let us think that its a bug/issue in FreeBSD. We also found some other topics on the web which pointed us in this direction.

pfSense 2.7 is the first version with a more recent FreeBSD version so we decided to set up a pfSense clone and upgraded this one to pfSense 2.7-dev. Since the update we don't experience any spikes anymore.

pfSense v2.7 runs stable (so far) and solved our issue so we decided to stay on v2.7-dev until the final release of v2.7