Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Pfsense with HA closing sessions when apply any rule.

    Scheduled Pinned Locked Moved HA/CARP/VIPs
    9 Posts 4 Posters 1.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • D
      diogo_falcomer
      last edited by

      Hi all!

      Here in my enterprise, we have a pfSense configuration with two HA pfsense, using CARP, virtual IPs, IPv4 and IPv6 configured and pfBlocker.

      After i upgraded to 2.4.3-Release, everytime i apply any rule all sessions are closed ( like SSH connections receive a timeout ).

      For that reason, i`ve upgraded to 2.4.3-RELEASE-p1. And then, it stops load firewall rules (solved applying this correction: https://redmine.pfsense.org/issues/8518).

      But then, the problem with sessions (PFSYNC) continue. And now apper this two erros too:

      0_1528983810048_290f9f42-ab5d-4e96-997a-59500efa2a13-image.png

      Both firewalls have the same Installed Packages. Same versions. SYNC interfaces ping each other.

      SYNC interface Rules:

      0_1528984922566_28a33431-1c56-4cc6-8ad9-633ced722711-image.png

      System Logs when i apply a firewall rule:

      Jun 14 08:29:07	php-cgi		notify_monitor.php: Message sent to lista-seguranca.centralit@listas.icmbio.gov.br OK
      Jun 14 08:29:02	php-fpm	327	[pfBlockerNG] XMLRPC sync to [ 192.168.1.2:{port} ] completed successfully.
      Jun 14 08:29:02	php-fpm	327	/rc.start_packages: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.merge_installedpackages_section).
      Jun 14 08:29:01	php-fpm	327	/rc.start_packages: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:29:01	php-fpm	327	/rc.start_packages: New alert found: A communications error occurred while attempting to call XMLRPC method merge_installedpackages_section:
      Jun 14 08:29:01	php-fpm	327	/rc.start_packages: A communications error occurred while attempting to call XMLRPC method merge_installedpackages_section:
      Jun 14 08:29:00	php-fpm	70969	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.restore_config_section).
      Jun 14 08:28:57	php-cgi		notify_monitor.php: Message sent to lista-seguranca.centralit@listas.icmbio.gov.br OK
      Jun 14 08:28:57	php-fpm	70969	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:28:57	php-fpm	70969	/rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:
      Jun 14 08:28:57	php-fpm	70969	/rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method restore_config_section:
      Jun 14 08:28:01	php-fpm	327	/rc.start_packages: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:28:01	php-fpm	327	[pfBlockerNG] XMLRPC sync is starting.
      Jun 14 08:28:01	check_reload_status		Reloading filter
      Jun 14 08:28:01	check_reload_status		Syncing firewall
      Jun 14 08:27:59	check_reload_status		Reloading filter
      Jun 14 08:27:59	php-fpm	327	/rc.start_packages: The command '/sbin/ifconfig 'bce0.521' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
      Jun 14 08:27:59	php-fpm	86843	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.restore_config_section).
      Jun 14 08:27:59	nrpe	40186	Allowing connections from: 10.197.32.238
      Jun 14 08:27:59	nrpe	40186	Listening for connections on port 0
      Jun 14 08:27:59	nrpe	40186	Warning: Daemon is configured to accept command arguments from clients!
      Jun 14 08:27:59	nrpe	40186	Server listening on 10.197.21.3 port 5666.
      Jun 14 08:27:59	nrpe	40186	Starting up daemon
      Jun 14 08:27:59	php-fpm	327	[pfBlockerNG] Starting cron process.
      Jun 14 08:27:59	nrpe	60867	Daemon shutdown
      Jun 14 08:27:59	nrpe	60867	Cannot remove pidfile '/var/run/nrpe2.pid' - check your privileges.
      Jun 14 08:27:59	nrpe	60867	Caught SIGTERM - shutting down...
      Jun 14 08:27:59	php-fpm	327	/rc.start_packages: Restarting/Starting all packages.
      Jun 14 08:27:58	check_reload_status		Starting packages
      Jun 14 08:27:58	check_reload_status		Reloading filter
      Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: XMLRPC versioncheck: 18.0 -- 18.0
      Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.host_firmware_version).
      Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: XMLRPC versioncheck: 18.0 -- 18.0
      Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.host_firmware_version).
      Jun 14 08:27:56	pkg-static		pfSense-pkg-System_Patches-1.1.7 installed
      Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:27:56	php		/etc/rc.packages: Successfully installed package: System Patches.
      Jun 14 08:27:55	check_reload_status		Syncing firewall
      Jun 14 08:27:55	check_reload_status		Syncing firewall
      Jun 14 08:27:54	php		/etc/rc.packages: Beginning package installation for System Patches .
      Jun 14 08:27:45	php-fpm	79998	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.restore_config_section).
      Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: XMLRPC versioncheck: 18.0 -- 18.0
      Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.host_firmware_version).
      Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
      Jun 14 08:27:41	check_reload_status		Syncing firewall
      

      If anyone can help me to solve this problem, Thanks!!

      1 Reply Last reply Reply Quote 0
      • K
        kien nguyen
        last edited by

        Hi all, i have similar error with pfsense version 2.4.4. Do you resolve???

        1 Reply Last reply Reply Quote 0
        • R
          Rocco83
          last edited by Rocco83

          I have the same issue too.
          Running 2.4.4p1 on XG-7100.

          Apr 29 00:57:32 pf1-lipi php-fpm[347]: /rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method restore_config_section:
          Apr 29 00:57:32 pf1-lipi php-fpm[347]: /rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:
          Apr 29 00:57:32 pf1-lipi php-fpm[347]: /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.11.8.232:443/xmlrpc.php.
          Apr 29 00:57:34 pf1-lipi php-fpm[5542]: /rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method host_firmware_version:
          Apr 29 00:57:34 pf1-lipi php-fpm[5542]: /rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method host_firmware_version:
          Apr 29 00:57:34 pf1-lipi php-fpm[5542]: /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.11.8.232:443/xmlrpc.php.
          
          1 Reply Last reply Reply Quote 0
          • R
            Rocco83
            last edited by

            As by
            https://redmine.pfsense.org/issues/9489#change-40418
            "You have a configuration error, probably a down gateway triggering state killing. Keep the discussion on the forum."

            In my case, in this specific configuration, the above comment is true (this is a test configuration).

            IMO this is anyway a bug, as a gateway down should not trigger the reset of sessions.

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by Derelict

              If you tell it to reset states and it resets states, where is the bug?

              499ca131-11a0-4f14-a50f-05bfacc9475b-image.png

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 1
              • R
                Rocco83
                last edited by

                Correct, i had the setting enabled. Thanks again for spotting it out.

                I would suggest at least to have a warning message about this checkbox, stating that:
                "If one gateway is down, or goes down during the config sync, pfsense may report a failed synchronization task"

                And i see two improvements:

                1. i would expect to have pfsense being smart enough to, in case of connection reset, try it once again and do not report the error. This is pfsense against pfsense.
                2. the gateway didn't goes down at reload time, the gateway were down also before the config sync. Meaning, the text is not consistent. Clearly, the check on gateway restart (i suppose now), but a down gateway that is down before and after a config sync should not trigger the state killing IMO.

                What's your view about the 3 points above?

                Thanks anyway for spotting it out and for the support!

                1 Reply Last reply Reply Quote 0
                • DerelictD
                  Derelict LAYER 8 Netgate
                  last edited by

                  Can only prevent so much foot-shooting. Many of these checkboxes have many, many ramifications. Users should know what they are doing and understand the ramifications before checking them. The default here is unchecked because that is generally the desired setting.

                  The link to the book that describes this is in my signature.

                  The documentation would be the place for that. if you believe something there is unclear there is a feedback function at the bottom of each page.

                  Chattanooga, Tennessee, USA
                  A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                  DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                  Do Not Chat For Help! NO_WAN_EGRESS(TM)

                  R 1 Reply Last reply Reply Quote 0
                  • R
                    Rocco83 @Derelict
                    last edited by

                    @Derelict

                    I can fully understand i am pissing you off, and i am pretty sure (as i write code sometimes too) that your position likely will not change.
                    But please understand that i am writing the below post with the best willing.

                    The best reference i have found is:
                    https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html#gateway-monitoring
                    There is a note

                    When this is triggered, the entire state table is cleared. This is necessary because it is not possible to kill all states for the failing WAN and the LAN-side states associated with the failing WAN. Removing states on the WAN side alone is ineffective, the LAN-side states must be cleared as well.
                    

                    and
                    https://docs.netgate.com/pfsense/en/latest/book/multiwan/multi-wan-terminology-and-concepts.html#multi-wan-state-killing-forced-switch
                    Where is written

                    This is an optional behavior, enabled by default. For information on changing this setting, see Gateway Monitoring.
                    

                    At a first sight, it collide with your statement:

                    The default here is unchecked because that is generally the desired setting.

                    Also, IMO this does not change what i have reported as improvements.

                    And i see two improvements:

                    1. i would expect to have pfsense being smart enough to, in case of connection reset, try it once again and do not report the error. This is pfsense against pfsense.
                    2. the gateway didn't goes down at reload time, the gateway were down also before the config sync. Meaning, the text is not consistent. Clearly, the check on gateway restart (i suppose now), but a down gateway that is down before and after a config sync should not trigger the state killing IMO.

                    What do you think about the above two point?

                    For the message in the checkbox,

                    "If one gateway is down, or goes down during the config sync, pfsense may report a failed synchronization task"

                    i can agree that the manual is the correct reference but i feel that this checkbox is poorly documented.
                    Schermata 2019-04-30 alle 22.50.15.png
                    Just check the below section, this is much more extensive.
                    And the manual section (again https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html#gateway-monitoring) is even shorter :)

                    So my suggestion would be to change the text of the checkbox to:

                    The monitoring process will flush all states when a gateway goes down if this box is checked. This may be useful with multiple gateway and backup link.
                    Please note that all states will be flushed, not only the one related to the specific gateway.
                    This is equivalent to Diagnostic -> States -> Reset States -> check "Reset the firewall state table" -> Reset 
                    

                    Thank you very much for reading it all up to here,
                    Daniel

                    1 Reply Last reply Reply Quote 0
                    • DerelictD
                      Derelict LAYER 8 Netgate
                      last edited by

                      Again, the proper forum for documentation feedback is the give feedback link on the page.

                      Chattanooga, Tennessee, USA
                      A comprehensive network diagram is worth 10,000 words and 15 conference calls.
                      DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
                      Do Not Chat For Help! NO_WAN_EGRESS(TM)

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post
                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.