0

Background information:

We have latency/freeze problems with our pfSense. To be able to answer the question i will first give some general background information:

  • We run pfSense V2.6.0 on following hardware:
    • Supermicro mainboard
    • Intel(R) Pentium(R) CPU N3710 / AES-NI CPU Crypto: Yes (inactive) / QAT Crypto: No
    • 4Gb memory
  • CPU load never exceed 50% usage(spikes) / Iddle 5-8%
  • Memory never exceeds 15% usage
  • Swap usage always 0%
  • We have around 45 active ipSec connections

Our problem:

We have an internal PBX (3CX) which we use with their (smartphone) app when we are out of the office. We noticed hick ups/freezes in the conversations so we started searching for the cause.

We noticed that pfSense has sometimes really high ping latencies. When i ping pfSense from my desktop the latency is in general 1ms. Sometimes we see spikes from 2000+ms (see image) These spikes correspond with the freezings during a phone call. Note that the spike occur randomly.

After some testing we noticed that disabling all ipSec connections fixes these spikes. The ipSec connections are only used for really small amounts of data which are send periodically between internal an customer sytems. Most of the time there is no data send over the connections.

We also monitored our incoming and outgoing bandwidth and its never even close to our DSL limits.

enter image description here

ipSec configurations

The most ipSec connections use the same configuration. For the investigation we disabled all different (ca. 5) configurations but still have the issue. The "default" ipSec configuration we use:

<phase1>
        <ikeid>*********</ikeid>
        <iketype>ikev2</iketype>
        <interface>wan</interface>
        <remote-gateway>*********</remote-gateway>
        <protocol>inet</protocol>
        <myid_type>myaddress</myid_type>
        <myid_data></myid_data>
        <peerid_type>fqdn</peerid_type>
        <peerid_data>*********</peerid_data>
        <encryption>
            <item>
                <encryption-algorithm>
                    <name>aes</name>
                    <keylen>128</keylen>
                </encryption-algorithm>
                <hash-algorithm>sha256</hash-algorithm>
                <dhgroup>14</dhgroup>
            </item>
        </encryption>
        <lifetime>28800</lifetime>
        <pre-shared-key>*********</pre-shared-key>
        <private-key></private-key>
        <certref></certref>
        <caref></caref>
        <authentication_method>pre_shared_key</authentication_method>
        <descr>*********</descr>
        <nat_traversal>on</nat_traversal>
        <mobike>off</mobike>
        <closeaction></closeaction>
        <startaction>none</startaction>
    </phase1>
    <phase2>
        <ikeid>*********</ikeid>
        <uniqid>*********</uniqid>
        <mode>tunnel</mode>
        <reqid>39</reqid>
        <localid>
            <type>network</type>
            <address>*********</address>
            <netbits>16</netbits>
        </localid>
        <remoteid>
            <type>network</type>
            <address>*********</address>
            <netbits>24</netbits>
        </remoteid>
        <protocol>esp</protocol>
        <encryption-algorithm-option>
            <name>aes</name>
            <keylen>128</keylen>
        </encryption-algorithm-option>
        <encryption-algorithm-option>
            <name>aes128gcm</name>
            <keylen>128</keylen>
        </encryption-algorithm-option>
        <hash-algorithm-option>hmac_sha256</hash-algorithm-option>
        <pfsgroup>14</pfsgroup>
        <lifetime>3600</lifetime>
        <pinghost>*********</pinghost>
        <descr>*********</descr>
    </phase2>

Question: How can we solve these high pings/freezes?

3
  • What do you expect from a 1.6/2.5 GHz Atom CPU? As it seems, the hardware limits the throughput and you'll need to properly scale your equipment. However, your hardware is off topic here and we can't really help you with that. Additionally, you should make sure that the AES-NI feature is actively used as it considerably speeds up IPsec processing.
    – Zac67
    Commented Nov 15, 2022 at 10:09
  • Thanks, i will take a look at the AES-NI feature and enable it. Near that, are you sure its about hardware limitations? I mean, the CPU is never above 50% which are only spikes, mostly it idles at 5-8%. Btw. its not an Atom CPU but Pentium with 4 cores. I think its should be more enough but i might be wrong...
    – CodeNinja
    Commented Nov 15, 2022 at 13:04
  • 1
    Depending on the OS, 25% CPU load may already represent a full core, maxing out a single thread. The N3710 is an Airmont/Braswell Atom - OoO but still: ark.intel.com/content/www/us/en/ark/products/91830/….
    – Zac67
    Commented Nov 15, 2022 at 18:02

2 Answers 2

3

Watch the output of 'top -aSH' in the pfsense command line to see what process is using a lot of CPU at the time of the spike. If it is reliable on a certain amount of time (every 30 seconds or every 5 minutes, etc.) then it might be a cron entry on the system or some scheduled task or feature in PFSense. You would need to check with PFSense to make sure your system is updated completely and see if there are any open bugs related to recurring CPU load spikes.

I don't know about that number of IPsec sessions, that is a lot for a small platform but if the CPU is generally under low load, it is probably ok to keep that hardware unless you can determine that the timing of the spikes coincides with actual bandwidth use. You may have 45 tunnels in use but they might not be passing hardly any traffic most of the time, resulting in very low CPU load. If some user is doing something every few minutes that causes a traffic spike and corresponding CPU spike then the issue is related to hardware being underpowered.

Also, I would agree that you should be using AES-NI hardware acceleration. Apparently you have to activate it manually, or at least check to make sure it is enabled.

https://forum.netgate.com/topic/124357/aes-in-inactive/3

It may make a huge improvement in performance under actual load.

1
  • Thanks for the info and tips. I updated my question; the spikes occur randomly and you are right, we have lots of connections but most of the time there is no traffic. I activated AES-NI hardware accelleration now but unfortunaltely it did not solve the issue. I also read that using AES-GCM is prefered above AES(-cbc?) as this works the best with AES-NI accelleration. We are concidering to update all of the connections to this. pfSense is up-to-date. For now i could not fix the issue with your tips but they are still helpfull and usefull. Thanks!
    – CodeNinja
    Commented Nov 16, 2022 at 11:38
1

We tried if stronger hardware solved our issue. We cloned our pfSense to hardware with a Intel(R) Xeon(R) CPU E3-1220 v6 @ 3.00GHz. The 100% CPU spikes where still there but the ping response time peeks where max 600ms instead of the max 3000ms. This means stronger hardware doesn't solve the 100% CPU spikes but made the issue less noticeable during e.g. phonecalls.

As the stronger hardware was not the solution (even if it took of some pressure to solve this issue) we monitored our pfSense intensively. We noticed that pfctl is the cause of the the high pings as it spikes CPU usage exactly at the time when the high ping occurs.

We found a bugreport on the pfSense redmine (https://redmine.pfsense.org/issues/12045) which fits to our findings. This line Yes, we are aware, but 2.6.0 will get the fix when we do a full sync with FreeBSD sources next, which wasn't an option for 2.5.2 (so we picked only the necessary commits there). showed us that we are probably not the first/only one. This Reddit topic is also related: https://www.reddit.com/r/PFSENSE/comments/nz8fmw/high_cpu_usage_sbinpfctl_ss/?utm_source=share&utm_medium=ios_app&utm_name=iossmf

As they mention that the issue will be fixed with a full FreeBSD sync let us think that its a bug/issue in FreeBSD. We also found some other topics on the web which pointed us in this direction.

pfSense 2.7 is the first version with a more recent FreeBSD version so we decided to set up a pfSense clone and upgraded this one to pfSense 2.7-dev. Since the update we don't experience any spikes anymore.

pfSense v2.7 runs stable (so far) and solved our issue so we decided to stay on v2.7-dev until the final release of v2.7

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.