Navigation

    Netgate Discussion Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search

    My IPSEC service hangs

    IPsec
    14
    74
    4505
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • F
      Flukester last edited by

      Just a update since changes I made we now been up for 7days... too early to say yet for sure, but things looking good

      david11717 1 Reply Last reply Reply Quote 0
      • david11717
        david11717 @Flukester last edited by

        @flukester I made the same changes you detailed above and the issue still happens for me. At this point I've stopped playing with it and my cron job fixes the issue within about 60 seconds of it happening. It's kind of ridiculous, honestly.

        F 1 Reply Last reply Reply Quote 0
        • R
          romczak last edited by romczak

          I have the same problem with the firewall, so I purchased the Enterprise support. They pointed me to the link https://redmine.pfsense.org/issues/13014 and said to follow what is there... without any explanation... Working with Palo Alto and Cisco daily I was quite stunned to get typical online forum response.
          I guess you got what you pay for, but If anyone is considering buying the subscription I DO NOT recommend.

          Right now I am running on the above script except the service restarts don't fix IPSec, so I replaced it with system reboot. I am using it for IPSec only so when it goes down I already have outage.

          G 1 Reply Last reply Reply Quote 0
          • G
            gassyantelope @romczak last edited by gassyantelope

            @romczak Yeah, it's been pretty concerning how this whole situation has been handled. It's one thing if there's problems when using pfSense community edition, since that's always been free and open source and you can't expect everything to be fixed ASAP. It's another thing when Netgate sells devices, support, their paid edition of pfSense, and market them as being stable and robust, yet they can't even do IPsec reliably.

            I'd expect such common/core functionality to work properly when buying their devices and/or their "enterprise" pfSense edition. We're now going on 8+ months since the issue was submitted on Redmine, but there were other similar issues submitted over a year or two ago. It's a really bad look for them to take people's money for a product that lacks reliable IPsec functionality, something that I've never seen any other firewall software/company struggle with. It's just as bad that you can pay for support, but they don't seem to do much more than point you to the public issue tracker (Redmine, which is free for everyone) or the community forum.

            The good thing is that the issue on Redmine is finally getting some responses from some developers. Hopefully that means we'll have a real fix soon. Though, it still makes me worry about how future problems may be handled. If there's another issue with core functionality in the future, is it going to be another 8+ month wait to get that fixed as well?

            R 1 Reply Last reply Reply Quote 0
            • T
              Topogigio last edited by

              I'm leaving pfSense, this is the only solution. The product (considering it from all points of views) simply is now too bad to count on it for production environments and real life.

              M 1 Reply Last reply Reply Quote 0
              • M
                michmoor @Topogigio last edited by

                @topogigio
                I feel your frustration. I will say this. Ive had network-breaking bugs in high-profile vendors on products that do simple tasks - i.e. Switching or Firewalling.
                If you are in the Palo Alto space then i will tell you the 10.X release has been troublesome and that's putting it lightly. In the end, after 5 months we got a specially developed hotfix. Juniper supports switch stacking (VC) and its fun when multicast stops working in a crucial deployment. To this day my S1 ticket still has no solution from the development team. This is a pretty important function - switching - that apparently is not being taken seriously enough even though their internal PRs have numerous customers impacted.

                Just trying to illustrate that having all of these prodcuts from high profile vendors, eac have issues but at least with Netgate i would argue that the Pros outweigh the Cons.

                Firewall: NetGate 6100/7100U, Palo Alto
                Routing: Juniper MX204 , Arista 7050X3
                Switching: Juniper EX/QFX. Arista 7050SX
                Wireless: Unifi, Aruba IAP

                T 1 Reply Last reply Reply Quote 0
                • T
                  Topogigio @michmoor last edited by

                  @michmoor also Fortinet asks for a lot of money and has a lot of bugs (7.2.x version is and hell) and a bad support service.

                  Still I think that so many months without a basic function as IPSEC working is not acceptable. This is not some secondary feature.

                  M 1 Reply Last reply Reply Quote 0
                  • M
                    michmoor @Topogigio last edited by

                    @topogigio said in My IPSEC service hangs:

                    Still I think that so many months without a basic function as IPSEC working is not acceptable. This is not some secondary feature.

                    I dont think IPsec is a secondary feature. I dont think any feature supported by a vendor is secondary. If you pay for TAC than you are within your right to hammer them every single day until a resolution is found otherwise you will have to move to another vendor.
                    Im only saying that Netgate is not alone in this. Leave Netgate then you may find that vendor X has some game breaking bug and the cycle repeats.
                    Ultimately its down to the business. If the risk is worth it then you have to find another solution.

                    Firewall: NetGate 6100/7100U, Palo Alto
                    Routing: Juniper MX204 , Arista 7050X3
                    Switching: Juniper EX/QFX. Arista 7050SX
                    Wireless: Unifi, Aruba IAP

                    1 Reply Last reply Reply Quote 0
                    • R
                      romczak @gassyantelope last edited by

                      @gassyantelope Bug is one thing, but the response of the support is unacceptable. And I selected the Enterprise level subscription.
                      This was the response to the ticket I've submitted:
                      """
                      It looks like you are hitting the bug: https://redmine.pfsense.org/issues/13014

                      Please refer to that page for updates.

                      Thanks,
                      """

                      M 1 Reply Last reply Reply Quote 0
                      • M
                        michmoor @romczak last edited by

                        @romczak yeah… that’s not a good way of responding to a customer issue. Can’t defend that.

                        Firewall: NetGate 6100/7100U, Palo Alto
                        Routing: Juniper MX204 , Arista 7050X3
                        Switching: Juniper EX/QFX. Arista 7050SX
                        Wireless: Unifi, Aruba IAP

                        1 Reply Last reply Reply Quote 0
                        • J
                          jdemmer last edited by jdemmer

                          Any updates on this? Running 4 other pfsense instances in Azure 2.4.5-RELEASE with zero IPSEC service issues. My 5th instance is running 22.05-RELEASE pfsense + as this seems to be the only version of pfsense available in the azure marketplace. I wanted to just deploy regular pfsense 2.4.5 but it is no longer and option. So I am stuck with 22.05 pfSense+ and the IPSEC service hangs exactly as others have in this post and the only fix is to restart the VM. Not seeing much progress @redmine

                          R 1 Reply Last reply Reply Quote 0
                          • R
                            romczak @jdemmer last edited by

                            @jdemmer From my observation it looks like the problem occurs when the interesting traffic is unable to bring the VPN up.
                            I have disabled all keepalives, and since we don't have any traffic towards remotes, the IPSec service stopped failing. It has been up for a month already.

                            J 1 Reply Last reply Reply Quote 0
                            • J
                              jdemmer @romczak last edited by

                              @romczak Thanks for the update, I am going to disable all keepalives and see how it goes

                              R 1 Reply Last reply Reply Quote 0
                              • R
                                romczak @jdemmer last edited by

                                @jdemmer Just to clarify. If the VPN is up, the keepalives are not a problem. We had issues only when the VPN was not able to be established due to external connectivity issues.

                                1 Reply Last reply Reply Quote 0
                                • F
                                  Flukester @david11717 last edited by

                                  @david11717 said in My IPSEC service hangs:

                                  @flukester I made the same changes you detailed above and the issue still happens for me. At this point I've stopped playing with it and my cron job fixes the issue within about 60 seconds of it happening. It's kind of ridiculous, honestly.

                                  Thanks, still happening for us on 23.01-RELEASE if we leave a VPN trying to connect for any period of time. (easy to forget you were working on one and leave it)

                                  Can you detail how to install your script ?, very much a beginner

                                  Cheers mate

                                  G 1 Reply Last reply Reply Quote 0
                                  • G
                                    glreed735 @Flukester last edited by

                                    @flukester - Someone else on the board created the script to address this, so credit is due to that person, but here's how you can set it up on your router.

                                    • First, enable ssh access in System > Advanaced > Enable Secure shell

                                      I use putty for my ssh client, available for general download on the Internet

                                    • Use putty to login to your router with your web admin credentials and select '8'.

                                    • Next you have to use an editor to put the script in place. I use 'vi'.

                                    vi /usr/local/bin/pfsense_charon_fix
                                    Insert the script contents
                                    #!/bin/sh

                                    queueLength=$(netstat -Lan | grep charon.vici | cut -c 7)

                                    if [ $((queueLength)) -gt 0 ]; then

                                                /usr/bin/killall -9 charon
                                                /usr/local/sbin/pfSsh.php playback restartipsec; sleep 20; /usr/local/sbin/pfSsh.php playback restartipsec
                                                echo "Process ran @ `date`" >>/var/log/ipsec_restart.log
                                    

                                    else

                                    fi

                                    Save the file.

                                    • Now, set execute permissions on this file with this command
                                      chmod 0755 /usr/local/bin/pfsense_charon_fix

                                    • Next step is to add this routine to the cron scheduler, using vi again to edit a file
                                      vi /etc/crontab

                                    Add this line to bottom of the file
                                    */1 * * * * root /usr/local/bin/pfsense_charon_fix

                                    You'll need to leave a blank line after that entry too.

                                    Save this file.

                                    Logout and reboot the router.
                                    I'd recommend going back to System > Advanced and turning off ssh access until you need it again.

                                    This has been working really well for me for quite a while. It's not perfect, but it's better than status quo.

                                    F 1 Reply Last reply Reply Quote 2
                                    • F
                                      Flukester @glreed735 last edited by

                                      @glreed735 many thanks for the detailed guide to setting up, very much appreciated.

                                      This 'QueueLength', if it's over 0 is IPsec in 'meltdown mode' and essentially broken. Reason for asking is we have around 70 IPsec VPNs and don't want everything falling into some IPsec restart loop.

                                      G 1 Reply Last reply Reply Quote 0
                                      • G
                                        glreed735 @Flukester last edited by

                                        @flukester - That's correct. The script does a hard reset of the IPSEC service only, the other router functions are unaffected. Since I put this in place, my router has been up for over 152 days, I've got about 300 VPNs on this instance. The script has been triggered 41 times to date. The only disruption is VPNs getting reestablished, if it happens after hours no one notices, but it can occur at any time.

                                        F 1 Reply Last reply Reply Quote 0
                                        • F
                                          Flukester @glreed735 last edited by Flukester

                                          @glreed735 thanks

                                          Having issue with this, created the file fine in right place and set permissions on a test PF with a active VPN

                                          As soon as I reboot even without CRON job though the test VPN is fully up p2 no longer receives traffic or can send

                                          #!/bin/sh
                                          queueLength=$(netstat -Lan | grep charon.vici | cut -c 7)
                                          if [ $((queueLength)) -gt 0 ]; then
                                          /usr/bin/killall -9 charon
                                          /usr/local/sbin/pfSsh.php playback restartipsec; sleep 20; /usr/local/sbin/pfSsh.php playback restartipsec
                                          echo "Process ran @ date" >>/var/log/ipsec_restart.log
                                          else
                                          fi

                                          If I delete the file and reboot everything works again.

                                          edit: on a 2nd test, even deleting file after reboot having same issue, very strange, ended up just replacing root volume from snapshot in AWS to fix

                                          G 1 Reply Last reply Reply Quote 1
                                          • G
                                            glreed735 @Flukester last edited by

                                            @flukester The existence of a script will have no effect on operation if it's not called. It's just another file on disk. In theory, garbling he crontab might some operational consequences, but not on VPN connections. Executing the script should result in no action for the overwhelming majority of the time, it's set to run every minute with that entry in crontab. Only when the queue gets backed up will it try the hard reset of the IPSEC service.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post