CARP/HA IPSec on Backup Node - connection not found?



  • Hi,

    we have a CARP/HA Setup with 2 nodes.
    On the primary node all switches for synching are checked.
    We have 3 IPSec configurations, the first one is deactivated and the second and third are active.
    I can see all three connections in the user interface of both nodes... in sync.
    When a failoiver to the backup node happens, the second IPSec connection does not come up.
    In the logfiles there is an entry:

    Oct 29 15:59:58	charon		14[CFG] received stroke: initiate 'con2000'
    Oct 29 15:59:58	charon		05[CFG] no IKE_SA named 'con2000' found
    

    For me it looks like the configuration 'con2000' does not exists on the second node...
    Any ideas?


  • Netgate

    The first thing I would try is a simple edit/save of that connection on the primary. Be sure there are no XMLRPC errors there afterward.

    If that doesn't work you might try a simple edit/save on the secondary but that should not be necessary.

    If you want, run ipsec statusall con2000 on the secondary before you do anything and see what it says. Safe for Diagnostics > Command Prompt.



  • @derelict said in CARP/HA IPSec on Backup Node - connection not found?:

    ipsec statusall con2000

    Hi,

    now, I have disabled all other connections for debugging.
    There are no XML RPC errors after saving a connection.

    Oct 30 10:30:37	php-fpm	88966	/rc.filter_synchronize: Filter sync successfully completed with http://192.168.XXX.XXX:80.
    Oct 30 10:30:37	check_reload_status		Reloading filter
    Oct 30 10:30:37	check_reload_status		Reloading filter
    Oct 30 10:30:37	php-fpm	88966	/rc.filter_synchronize: XMLRPC sync successfully completed with http://192.168.XXX.XXX:80.
    Oct 30 10:30:36	php-fpm	88966	/rc.filter_synchronize: Beginning XMLRPC sync to http://192.168.XXX.XXX:80.
    

    If we switch to the secondary with "CARP Maintenance Mode" I have to connect manually with the UI, otherwise there is no connection attempt.
    "ipsec status all" shows that the connection is up on the primary, on the secondary it shows "no connection":

    ...
    Connections:
    Security Associations (0 up, 0 connecting):
      no match
    

    It seems that for the secondary this connection does not exist... but in the UI the connection is correct displayed.
    Where can I check the synced ipsec configuration files via shell/command prompt?

    Thx


  • Netgate

    Look in /var/etc/ipsec/ipsec.conf

    You might want to force a config sync in Status > Filter Reload on the primary (Yes, that's a strange place for that button.)

    If we switch to the secondary with "CARP Maintenance Mode" I have to connect manually with the UI, otherwise there is no connection attempt.
    "ipsec status all" shows that the connection is up on the primary, on the secondary it shows "no connection":

    Not sure what you're saying here. There is no automatic change to the other node's webui. I generally connect to both nodes on their interface addresses when working on a cluster, not the CARP VIP. Your connection state should be on both but the nginx process on one doesn't have a session on the other.



  • Hi.

    thanks for your hint - I have checked the files on both nodes:
    They differ totally! "conn2000" is missing.

    Master:

    ls -l
    total 16
    -rw-r--r--   1 root  wheel  1299 Nov  6 14:18 ipsec.conf
    

    Backup:

    ls -l
    total 16
    -rw-r--r--   1 root  wheel  732 Oct 16 13:35 ipsec.conf
    

    If I hit the "Force config sync" on the primary or if anything in the configuration of the primary is changed, the file is not synced or updated to the second node. It just changes on the primary.
    There are no sync errors:

    XMLRPC sync successfully completed with XXXX
    

    With UI I mean when I look into the User Interface of the IPSec configuration (VPN->IPSec), the configurations here are identical on both nodes! ... but not in the real IPSec config files (/var/etc/ipsec/ipsec.conf). This means the pfsense XML configuration of the second is not in sync with the ipsec config file.
    Btw, we still use 2.3.5-RELEASE-p2.

    Usually we use OpenVPN and we bind all our VPNs on the CARP-VIP (Server and Clients). They are only up on the primary and are down on the backup. They switch with the CARP-VIP... if the VIP is on the secondary all the connections come up automatically.
    Does this not work with IPSec connections?

    Thx for your help!


  • Netgate

    Then I would edit/save the con2000 VPN connection on the primary again. There must have been a syncing issue at one time.

    If that does not work I would delete it and recreate it on the primary.

    Or backup and restore just the IPSEC section on the primary.

    2.3.5 is EOL. As is the underlying FreeBSD 10.3. You'd do well to get to 2.4.4



  • Hi,

    after backup and restore the IPSec config there where no sync errors...
    But after a rolling restart of both nodes the ipsec configuration is synced again.
    => solved
    I am wondering why a reboot helped... it is not a Windows system ;-)

    Thanks for your hint of eol 2.3.5... I have looked here:
    https://www.netgate.com/docs/pfsense/releases/index.html#current-upcoming-supported-releases
    It is still listed as supported in the documentation.


  • Netgate

    @sepp_huber said in CARP/HA IPSec on Backup Node - connection not found?:

    It is still listed as supported in the documentation.

    Not for long. Thanks. ;)


  • Netgate

    @sepp_huber said in CARP/HA IPSec on Backup Node - connection not found?:

    I am wondering why a reboot helped... it is not a Windows system ;-)

    A restart completely rebuilds all configuration files from the config.xml file.



  • @derelict said in CARP/HA IPSec on Backup Node - connection not found?:

    A restart completely rebuilds all configuration files from the config.xml file.

    OK...
    After changing something on the primary it is still not synced to the secondary.
    Anyway, I can live with this workaround => reboot of the secondary after changing the ipsec configuration.

    I will update to 2.4.X in the next weeks.


  • Netgate

    I would continue to work the problem. That is certainly not normal.

    Is IPsec checked in the XMLRPC sync settings on the primary?



  • As already stated in my inital post here... yes everything in the sync settings is checked AND there are no sync errors in the logs. Is there a debugging/logging feature for the XMLRPC sync?


  • Netgate

    The system logs tell you what it is complaining about.



  • No errors on master

    Nov 12 10:52:22	php-fpm		/rc.filter_synchronize: Filter sync successfully completed with http://192.168.XXX.XXX:80.
    Nov 12 10:52:21	php-fpm		/rc.filter_synchronize: XMLRPC sync successfully completed with http://192.168.XXX.XXX:80.
    Nov 12 10:52:21	php-fpm		/rc.filter_synchronize: Beginning XMLRPC sync to http://192.168.XXX.XXX:80.
    

    Filter reload sync output extract on master

    ...
    Pre-caching IPSec Port...
    Creating filter rule IPSec Port ...
    Creating filter rules IPSec Port ...
    Setting up pass/block rules
    Setting up pass/block rules IPSec Port
    Creating rule IPSec Port
    Pre-caching IPSec Port...
    Creating filter rule IPSec Port ...
    Creating filter rules IPSec Port ...
    Setting up pass/block rules
    Setting up pass/block rules IPSec Port
    Creating rule IPSec Port
    ...
    Signaling CARP reload signal...
    Syncing CARP data to http://192.168.XXX.XXX
    XMLRPC sync successfully completed with http://192.168.XXX.XXX:80
    

    But on the backup I found:

    Nov 12 10:57:06	php-fpm	39865	/xmlrpc.php: The command '/usr/sbin/pw groupadd -n 'admins' -g '2000' -M '0' 2>&1' returned exit code '65', the output was 'pw: group name `admins' already exists'
    


  • Hi,

    after upgrading to 2.4.4, the synchronization still does not work.
    The ipsec file on the backup node is outdated.
    The message "/xmlrpc.php: The command '/usr/sbin/pw groupadd -n 'admins'..." does not appear anymore on the backup node.


  • Netgate

    I have no idea what the issue is there. Never appeared on any system I have been involved with before. XMLRPC generally works or it doesn't. It isn't selective as to what configs it syncs or what it doesn't.



  • I think the XMLRPCSync is working, there are no errors.
    All changes are visible in the ipsec user interface of the second node and if I make a "diff" of the configuration-backup XML of both nodes... all ipsec changes are included in the configuration of the second node.
    There must be an additional step after the XMLRPCSync which transfers the changes to "/var/etc/ipsec/ipsec.conf" and that fails or is not executed...
    Because after a reboot the file is "in-sync" with the configuration.

    Is it possible to change the debuglevel somewhere or add log output to the php source code?