Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    PfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM

    Scheduled Pinned Locked Moved IPsec
    30 Posts 10 Posters 6.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • RMBR
      RMB
      last edited by

      Any news on this bug?
      The problem is still there in version 2.4.2.
      I have to disable AES-NI to prevent a kernel panic during load through an IPSec tunnel.

      1 Reply Last reply Reply Quote 0
      • S
        SisterOfMercy
        last edited by

        Nope, but feel free to comment on the redmine bug repo:
        https://redmine.pfsense.org/issues/8070

        Or find someone with a support contract that can complain.  ::)

        Hi, I'm Lance Boyle, and people often wonder if I'm real.

        1 Reply Last reply Reply Quote 0
        • T
          Tacoma
          last edited by

          Having what I believe is this issue since moving to 2.4.x
          Here is a picture of the console with the Kernel crash.
          No log available.
          Reverted back to version 2.3.x and the problem has not occurred as of yet.

          2018-03-08_072957.png
          2018-03-08_072957.png_thumb

          1 Reply Last reply Reply Quote 0
          • T
            Tacoma
            last edited by

            One clarification on my application, using a supermicro motherboard with pfsense installed directly to hard drive.  No VM Software involved.

            1 Reply Last reply Reply Quote 0
            • T
              Tacoma
              last edited by

              Anyone know if this release has a fix for this issue?

              2.4.3-DEVELOPMENT (amd64)
              built on Tue Mar 13 10:14:21 CDT 2018
              FreeBSD 11.1-RELEASE-p7

              I see this is a patched version of FreeBSD, and there was a reference to ipsec fixes in the release notes, but it wasn't clear if this fixed this same issue.

              1 Reply Last reply Reply Quote 0
              • G
                GyroK
                last edited by

                @Tacoma:

                Anyone know if this release has a fix for this issue?

                2.4.3-DEVELOPMENT (amd64)
                built on Tue Mar 13 10:14:21 CDT 2018
                FreeBSD 11.1-RELEASE-p7

                I see this is a patched version of FreeBSD, and there was a reference to ipsec fixes in the release notes, but it wasn't clear if this fixed this same issue.

                Unfortunately, this bug is still valid with the following SW version:

                2.4.3-RELEASE (amd64)
                built on Mon Mar 26 18:02:04 CDT 2018
                FreeBSD 11.1-RELEASE-p7

                GCM mode cannot be used on the machines with AES-NI.

                Regards,
                GyroK

                1 Reply Last reply Reply Quote 0
                • jimpJ
                  jimp Rebel Alliance Developer Netgate
                  last edited by

                  To claim it's unusable in general is untrue. The crash must be specific to a certain combination of hardware, traffic load, and/or pattern of traffic.

                  Loads of people are using AES-NI and AES-GCM without crashing, including just about every Netgate employee from our home firewalls.

                  Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

                  Need help fast? Netgate Global Support!

                  Do not Chat/PM for help!

                  L msrachelchenM 2 Replies Last reply Reply Quote 0
                  • T
                    timthetortoise
                    last edited by

                    Can confirm this is occurring for me on two different systems.
                    Both are running on ESXi 6.5, one on DL380 G8, the other on DL380 G9.
                    NIC type is vmxnet3, open-vm-tools installed on both.
                    Phase 1: AES128-GCM / 128 / SHA1 / DH2
                    Phase 2: AES128-GCM / AES-XCBC / no PFS

                    Hard crash with a reboot within 5 minutes of initiating continuous iperf run, sometimes one side, sometimes both.

                    Switching to any non-AES-NI algorithms kills throughput, but doesn't hard crash.

                    My```
                    dmesg | grep -i aes

                    Features2=0xffba2203 <sse3,pclmulqdq,ssse3,cx16,pcid,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand,hv>aesni0: <aes-cbc,aes-xts,aes-gcm,aes-icm>on motherboard</aes-cbc,aes-xts,aes-gcm,aes-icm></sse3,pclmulqdq,ssse3,cx16,pcid,sse4.1,sse4.2,x2apic,popcnt,tscdlt,aesni,xsave,osxsave,avx,f16c,rdrand,hv>

                    
I'll do some more testing this weekend when there's not as much production traffic flowing but for right now I'm knocked back down to plain AES.

It does indeed make pfSense unusable for installations requiring decent IPSec interconnect speeds. Considering this issue I'll likely move to VyOS for my concentrators.

Has anyone attempted to use the patch from the previous FreeBSD thread posted?

Edit: both running 2.4.3-Release
                    1 Reply Last reply Reply Quote 0
                    • L
                      lkolbe @jimp
                      last edited by

                      @jimp we're experiencing the same problem. One client, using AES256-gcm, reliably crashes the SG-8860 w/pfsense 2.4.3 when using e.g. speedtest.net (during the upload phase), another can't bring it down at all. Switching back to plain AES with SHA512 seems to fix it for now. All clients are Macbook Pros.
                      Kind regards,
                      Lukas

                      1 Reply Last reply Reply Quote 0
                      • D
                        dave.opc
                        last edited by

                        is AES256-GCM better than AES?

                        1 Reply Last reply Reply Quote 0
                        • DerelictD
                          Derelict LAYER 8 Netgate
                          last edited by

                          AES-GCM is an authenticated cipher so you can eliminate the hashing step. If max IPsec performance is what you seek, AES-GCM is the way to go.

                          Chattanooga, Tennessee, USA
                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                          1 Reply Last reply Reply Quote 0
                          • D
                            dave.opc
                            last edited by

                            I got 2 sites connected via IPSEC using AES. Both have 100Mb connection to internet. And IPSEC uses the whole 100Mb bandwidth on file transfers. So what is the limitation for AES, compared to AES-GCM?

                            1 Reply Last reply Reply Quote 0
                            • DerelictD
                              Derelict LAYER 8 Netgate
                              last edited by

                              AES-GCM will consume fewer CPU cycles to accomplish the same task.

                              Chattanooga, Tennessee, USA
                              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                              Do Not Chat For Help! NO_WAN_EGRESS(TM)

                              1 Reply Last reply Reply Quote 0
                              • D
                                dave.opc
                                last edited by

                                So basically, if you got powerful enough CPU/PC it dosn't matter which algorithm to use?

                                1 Reply Last reply Reply Quote 0
                                • DerelictD
                                  Derelict LAYER 8 Netgate
                                  last edited by

                                  AES-GCM will use fewer CPU cycles to accomplish the same task.

                                  Fewer cycles means more cycles available for other tasks.

                                  You can waste them if you so desire.

                                  Chattanooga, Tennessee, USA
                                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                  1 Reply Last reply Reply Quote 0
                                  • L
                                    lkolbe
                                    last edited by

                                    ... as long as you can live with the occasional hard crash (at least on an SG-8860/Atom C7258 using AES-NI). I haven't yet found out what kind of traffic pattern causes this crash; switching to AES-CBC w/SHA512 removed the crashes reliably for us.

                                    1 Reply Last reply Reply Quote 0
                                    • msrachelchenM
                                      msrachelchen @jimp
                                      last edited by

                                      @jimp said in PfSense 2.4.1 - ikev2 IPSEC tunnel under load crashes whole firewall VM:

                                      To claim it's unusable in general is untrue. The crash must be specific to a certain combination of hardware, traffic load, and/or pattern of traffic.

                                      Loads of people are using AES-NI and AES-GCM without crashing, including just about every Netgate employee from our home firewalls.

                                      If that's the case, what are their hardware configurations? It seems to me that this issue is pretty common among users, and there isn't a pattern in hardware I can see.

                                      1 Reply Last reply Reply Quote 0
                                      • msrachelchenM
                                        msrachelchen
                                        last edited by

                                        Same issue with AES-GCM+AES-NI crashing the system:

                                        Version	2.4.3-RELEASE-p1 (amd64) 
built on Thu May 10 15:02:52 CDT 2018 
FreeBSD 11.1-RELEASE-p10 
	 
CPU Type	Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz
Current: 3600 MHz, Max: 3601 MHz
4 CPUs: 1 package(s) x 4 core(s)
AES-NI CPU Crypto: Yes (active)
                                        1 Reply Last reply Reply Quote 0
                                        • DerelictD
                                          Derelict LAYER 8 Netgate
                                          last edited by Derelict

                                          All kinds, but mostly Netgate devices such as the SG-2440, SG-4860, SG-3100.

                                          Intel(R) Atom(TM) CPU C2558 @ 2.40GHz
                                          4 CPUs: 1 package(s) x 4 core(s)
                                          AES-NI CPU Crypto: Yes (active)

                                          Uptime 122 Days 21 Hours 45 Minutes 01 Seconds

                                          AES_GCM_16
                                          MODP_2048
                                          IPComp: none

                                          I used TRex a few weeks ago to run terabytes and terabytes through AES-GCM IPsec trying to make it crash. Hundreds of megabits per second for days on end. Could not duplicate.

                                          Chattanooga, Tennessee, USA
                                          A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                                          DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                                          Do Not Chat For Help! NO_WAN_EGRESS(TM)

                                          msrachelchenM 1 Reply Last reply Reply Quote 0
                                          • msrachelchenM
                                            msrachelchen @Derelict
                                            last edited by msrachelchen

                                            @derelict a few things were mentioned in the thread

                                            1. 2.3 was unaffected yet 2.4 was crashing
                                            2. issues appeared on VM and bare metal
                                            3. there was a bug in the kernel
                                            4. this "issue" first appeared 10 months ago
                                            5. usr @RMB is using Netgate product yet he was experiencing the issue

                                            Maybe I suggest:

                                            1. Let us know if that particular patch was merged
                                            2. Can you try running AES-GCM with any EC (say nist ecp384)
                                            3. What do you suggest for us the users to get to the bottom of this?
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.