Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    IPSec Down after Upgrade to 2.3

    Scheduled Pinned Locked Moved IPsec
    72 Posts 30 Posters 40.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • jnorellJ
      jnorell
      last edited by

      We've now been up for over a week with these settings (set in System > Advanced > System Tunables):

      net.inet.raw.maxdgram 131072
      net.inet.raw.recvspace 1048576
      net.raw.recvspace 1048576
      net.raw.sendspace 1048576

      Edit:  up over 2 weeks now, still no problem

      1 Reply Last reply Reply Quote 0
      • A
        augustinermonch
        last edited by

        Hi I'm new here and have a problem with my PFSense and the IPsec connection .

        The environment :
        Location A pfsense 2.3.1_1
        Location B pfsense 2.3.1_1

        Connected via IPSec " SitetoSite "

        I tried all the tips from this thread. Unfortunately without success.

        Like
        changeing net.inet.raw.maxdgram  131072 
        net.inet.raw.recvspace  1048576 
        net.raw.recvspace  1048576 
        net.raw.sendspace  1048576

        The problem is when I try to access Site B about RMTC works without problems .
        However, if I want to print a print job from B to site A drops the connection and restarts.

        Does somebody has any idea ?

        I'm a bit desperate .

        Thank you very much

        I Forget to say that it works perfect before i updatet my pfsense …

        1 Reply Last reply Reply Quote 0
        • A
          augustinermonch
          last edited by

          Hi it´s me again, i tryed to use OPENVPN instead of IPSEC
          I have the same Problem and my PFSENSE reboot new after 2 min.

          Does anyone know this situation ?

          1 Reply Last reply Reply Quote 0
          • K
            Kamyk
            last edited by

            Hello everybody!

            I have read that thread but unfortunately I have the same issue. We use PfSense 2.3.1 with OpenBGPD+IPsec to Amazon AWS.

            We have set that:

            
            net.inet.raw.maxdgram="131072"
            net.inet.raw.recvspace="131072"
            net.raw.recvspace=65535
            net.raw.sendspace=65535
            
            

            Our IPsec disconnect every couple hours. When I check IPsec status - looks ok, but I can not transfer any packets. I don't have to reboot Firewalls but only stop OpenBGPD and IPsec. Start again and all is working again ok for next couple of hours.

            Do you have any idea what I can check more? I didn't check that fix from GitHub. But do you think it could be it?

            Thank you for any help or answer.

            Best,
            Kamyk

            1 Reply Last reply Reply Quote 0
            • O
              olobley
              last edited by

              I run a couple of pfsense boxes to link my house to a few neighbors (so hardly mission critical).
              Since the upgrade to 2.3, then 2.3.1, then 2.3.1p1, my IPsec tunnels haven't worked.
              I don't run OpenBGP (at least I don't think I do) and I tried applying the System Tuneables that jnorell suggested.
              I also tried purging all my VPN configurations, and recreating them. Still no love :(
              What's odd (at least to me), is that all the tunnels come up in the web interface, but they don't pass traffic.
              It's not the end of the world, as I moved to OpenVPN in the interim, however I'd prefer to get back to IPsec.
              Thanks in advance

              1 Reply Last reply Reply Quote 0
              • J
                JorgeOliveira
                last edited by

                @Kamyk:

                I have read that thread but unfortunately I have the same issue. We use PfSense 2.3.1 with OpenBGPD+IPsec to Amazon AWS.

                Known issue: https://redmine.pfsense.org/issues/6223

                My views have absolutely no warranty express or implied. Always do your own research.

                1 Reply Last reply Reply Quote 0
                • jnorellJ
                  jnorell
                  last edited by

                  @olobley:

                  Since the upgrade to 2.3, then 2.3.1, then 2.3.1p1, my IPsec tunnels haven't worked.
                  …
                  What's odd (at least to me), is that all the tunnels come up in the web interface, but they don't pass traffic.

                  It sounds like you have a different problem (try enabling cisco extentions in ipsec advanced settings), this one is indicated by 'No buffer space available' errors in the logs.

                  1 Reply Last reply Reply Quote 0
                  • jnorellJ
                    jnorell
                    last edited by

                    Today, after almost 29 days uptime, we're getting 'error sending to PF_KEY socket: No buffer space available' again .. I'm bumping settings up some more:

                    
                    net.inet.raw.maxdgram = 131072
                    net.inet.raw.recvspace = 1048576
                    net.raw.recvspace = 1048576
                    net.raw.sendspace = 2097152
                    
                    
                    1 Reply Last reply Reply Quote 0
                    • M
                      mraymond
                      last edited by

                      Hello, new to this forum. Just throwing my hat in the ring for this issue as well. Plagued by "error sending to PF_KEY socket: No buffer space available".
                      I'm using three IPsec tunnels. One to AWS (with BGP), one to Azure, one to a mikrotik router at a remote office.

                      Is there a way to effectively restart IPsec and flush that buffer without rebooting?
                      Restarting the service via the GUI, or manually killing charon and starter and restarting ipsec via terminal does not do it.

                      EDIT: Of course I should mention this problem started happening after upgrading from 2.2.(6?) to 2.3.1_1
                      I have increased
                      net.inet.raw.maxdgram
                      net.inet.raw.recvspace
                      net.raw.recvspace
                      net.raw.sendspace

                      to recommended values, but have not rebooted since. I will reboot late tonight.

                      1 Reply Last reply Reply Quote 0
                      • O
                        obrienmd
                        last edited by

                        Same issue with upgrade to 2.3.1_5, any idea if this will be resolved in 2.3.2 or 2.4.x (FreeBSD 11, right?)

                        1 Reply Last reply Reply Quote 0
                        • C
                          cmb
                          last edited by

                          Not something we're going to have time for in 2.3.2 (release next week), hopefully it's either resolved already in FreeBSD 11, so 2.4 will be fine, or someone can track down the root cause and get it fixed (my last day here is in two weeks).

                          2.4 snapshots should be out soon. Help testing then would be appreciated.

                          1 Reply Last reply Reply Quote 0
                          • T
                            Thnadim
                            last edited by

                            i face the same problem when i try to establish a new ipsec site to site vpn between 2 branches with a pfsense with a firmware 2.2.6. I solved that by adding
                            on the phase 1 proposal (authentication ) the real ip of my peer as it was behind the a nat

                            My identifier ===== choose Ip address ======= then put your real ip address

                            and on the Peer Identifier you should put the private ip of the other side if he do the same

                            Peer identifier ======== ip address =========then put your private ip address

                            1 Reply Last reply Reply Quote 0
                            • O
                              obrienmd
                              last edited by

                              @cmb:

                              Not something we're going to have time for in 2.3.2 (release next week), hopefully it's either resolved already in FreeBSD 11, so 2.4 will be fine, or someone can track down the root cause and get it fixed (my last day here is in two weeks).

                              2.4 snapshots should be out soon. Help testing then would be appreciated.

                              Sure thing, will test as soon as 2.4 snapshots are available! Good luck on your next adventure, and thanks for all the hard work on pfSense :)

                              1 Reply Last reply Reply Quote 0
                              • O
                                owczi
                                last edited by

                                Is anybody aware of any progress on this? Bumping the buffer sizes only extends the issue from a few hours to about two days but that is it.

                                Also is there any news regarding the root cause? I am struggling to understand the interaction between IPSec (+GRE) and OpenBGPd. Surely the same would happen with any TCP-based application, or is it something that OpenBGPd specifically repeatedly calls on the sockets that causes IPSec to eventually die?

                                I run a number of tunnels with IPSec + GRE + BGP (pfSense to pfSense and pfSense to Cisco) and since 2.1 they were never really stable. All the way up to 2.3 I had to monitor the GRE tunnels and bounce them after any IPSec re-key or tunnel flap because OpenBGPd was seeing them as invalid next hops. This went away in 2.3, but now IPSec is basically unusable. Doesn't matter why, it makes for an incomplete product. Nobody really runs static routing over non-trivial topologies, and with non-functional BGP, IPSec is only usable for mobile clients. I'm going to give BIRD a try - and migrating the whole network to OSPF is not really an option here, although I will consider it.

                                Failing that, after many years with pfSense I am going to start looking for alternatives. pfSense is a fantastic platform, and thanks for all the hard work guys, but constant IPSec issues have just about killed it for me.

                                1 Reply Last reply Reply Quote 0
                                • O
                                  owczi
                                  last edited by

                                  …OK, some progress.

                                  Having looked up the PF_KEY rcvbuf error got me a change and a setting introduced in StrongSWAN 5.3.0 where the event socket buffer can be tuned.

                                  Once all IPSec tunnels were dead, I stopped ipsec, stopped openbgpd, then I opened /etc/inc/vpn.inc, searched for the charon { plugins { section and added the following:

                                  ....
                                  
                                                  kernel-pfkey {
                                  
                                                          events_buffer_size = 1048576
                                  
                                                  }
                                  
                                  

                                  Started ipsec via GUI which re-generated the configs, started openbgpd. Guess what - tunnels came back up, I can see SADs and SPDs again, and some of the BGP sessions are up again (those to Cisco, funny enough). I have now rebooted all pfSense instances and will see how long they will last.

                                  Thanks,
                                  owczi

                                  1 Reply Last reply Reply Quote 0
                                  • O
                                    obrienmd
                                    last edited by

                                    @owczi:

                                    …OK, some progress.

                                    ...

                                    Started ipsec via GUI which re-generated the configs, started openbgpd. Guess what - tunnels came back up, I can see SADs and SPDs again, and some of the BGP sessions are up again (those to Cisco, funny enough). I have now rebooted all pfSense instances and will see how long they will last.

                                    Thanks,
                                    owczi

                                    Still up?

                                    1 Reply Last reply Reply Quote 0
                                    • O
                                      owczi
                                      last edited by

                                      @obrienmd:

                                      Still up?

                                      Nope - shat itself after about 24 hours. HOWEVER, I don't have to reboot to get the tunnels and BGP sessions back up. The setting I added to charon config may not have anything to do with it. I will keep trying various combinations to get a sensible answer: on some of the pfSense instances I did not have to restart IPSec at all, only bgpd, but it could have been that they had BGP down because of the other peers, and an ipsec restart is still required. I have no time to investigate right now.

                                      Basically:```
                                      /usr/local/etc/rc.d/bgpd.sh stop; ipsec stop; sleep 1; ipsec start; sleep 2; /usr/local/etc/rc.d/bgpd.sh start

                                      
                                      I need to write a monitoring script that will do this when all tunnels go down. For now I will just make it a cron job every few hours, maybe even every hour. offset so it doesn't happen on all instances at the same time. This will at least keep me going.
                                      1 Reply Last reply Reply Quote 0
                                      • O
                                        owczi
                                        last edited by

                                        EDIT: full path for ipsec - required when invoked from cron; do not reset ipsec / bgpd if there are no connections.
                                        EDIT2: fixed to correctly pick up connections when nothing is up and check for buffer errors
                                        Crude as can be, but will do the job… I run this every 5 minutes via a cron job:

                                        
                                        #!/bin/sh
                                        estabcount=0
                                        p2count=0
                                        totalcount=0
                                        buferr=0
                                        
                                        bounceall() {
                                        /usr/local/etc/rc.d/bgpd.sh stop
                                        sleep 1
                                        $ipsecpath stop
                                        sleep 1
                                        $ipsecpath start
                                        sleep 3
                                        /usr/local/etc/rc.d/bgpd.sh start
                                        }
                                        
                                        ipsecpath=/usr/local/sbin/ipsec
                                        
                                        echo "=== started at `date` ==="
                                        
                                        for con in `$ipsecpath status | grep "\[" | sed 's/\[.*//g' | sort | uniq` ; do 
                                        echo $con
                                        estab=0
                                        p2=0
                                        
                                        $ipsecpath status $con | grep ESTAB >/dev/null 2>&1 && estab=1
                                        $ipsecpath status $con | grep INSTALLED >/dev/null 2>&1 && p2=1
                                        
                                        [ $estab -eq 1 ] && { 
                                        	echo $con p1 up
                                        	estabcount=$(( $estabcount + 1 ))
                                        	[ $p2 -eq 0 ] && {
                                        	 	echo $con p2 down, restarting
                                                        echo stopping $con...
                                        		$ipsecpath down $con >/dev/null 2>&1
                                        		sleep 1
                                                        echo starting $con...
                                        		$ipsecpath up $con | grep error | grep "buffer space" >/dev/null 2>&1  && { echo "PF_KEY buffer error while starting $con"; buferr=$(( $buferr + 1 )); }
                                        	}
                                        
                                        }
                                        [ $estab -eq 0 ] && { echo $con p1 down; }
                                        [ $p2 -eq 1 ] && { echo $con p2 up; p2count=$(( $p2count + 1 )); }
                                        totalcount=$(( $totalcount + 1 ))
                                        done
                                        
                                        echo
                                        echo ===
                                        echo estab $estabcount / $totalcount
                                        echo p2 $p2count / $totalcount
                                        echo buf_err $buferr / $totalcount
                                        echo ===
                                        echo
                                        
                                        [ $totalcount -gt 0 ] && [ $buferr -gt 0 ] && {
                                        echo $buferr connections show buffer space errors - bouncing openbgpd and ipsec
                                        bounceall
                                        exit
                                        }
                                        
                                        [ $totalcount -gt 0 ] && [ $estabcount -eq 0 ] && {
                                        echo no connections have p1 up - bouncing openbgpd and ipsec
                                        bounceall
                                        exit
                                        }
                                        
                                        [ $totalcount -gt 0 ] && [ $estabcount -eq $totalcount ] && [ $p2count -eq 0 ] && {
                                        echo all connections have p1 up but no connections have p2 up - bouncing openbgpd and ipsec
                                        bounceall
                                        exit
                                        }
                                        
                                        

                                        It will bounce all tunnels which have phase 2 down, and if no tunnels have p1 it will bounce ipsec and bgpd. We'll see how long this will last.

                                        1 Reply Last reply Reply Quote 0
                                        • O
                                          obrienmd
                                          last edited by

                                          How has this worked for you?

                                          1 Reply Last reply Reply Quote 0
                                          • O
                                            owczi
                                            last edited by

                                            I have just edited the post with the latest version of the script. It didn't work when everything was down, and didn't correctly react to the situation where all P1 is up but no P2 is up. With these modifications This has been working fine for me for the past two weeks or so. It still crashes, but at least it recovers now.

                                            So: The charon setting I added does not fix the problem, but allows you to recover by stopping bgpd, stopping ipsec, starting ipsec and then starting bgpd again - which did not seem possible before. Better than nothing…

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.