Pfsense with HA closing sessions when apply any rule.

[diogo_falcomer]

Hi all!

Here in my enterprise, we have a pfSense configuration with two HA pfsense, using CARP, virtual IPs, IPv4 and IPv6 configured and pfBlocker.

After i upgraded to 2.4.3-Release, everytime i apply any rule all sessions are closed ( like SSH connections receive a timeout ).

For that reason, i`ve upgraded to 2.4.3-RELEASE-p1. And then, it stops load firewall rules (solved applying this correction: https://redmine.pfsense.org/issues/8518).

But then, the problem with sessions (PFSYNC) continue. And now apper this two erros too:

Both firewalls have the same Installed Packages. Same versions. SYNC interfaces ping each other.

SYNC interface Rules:

System Logs when i apply a firewall rule:

Jun 14 08:29:07	php-cgi		notify_monitor.php: Message sent to lista-seguranca.centralit@listas.icmbio.gov.br OK
Jun 14 08:29:02	php-fpm	327	[pfBlockerNG] XMLRPC sync to [ 192.168.1.2:{port} ] completed successfully.
Jun 14 08:29:02	php-fpm	327	/rc.start_packages: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.merge_installedpackages_section).
Jun 14 08:29:01	php-fpm	327	/rc.start_packages: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:29:01	php-fpm	327	/rc.start_packages: New alert found: A communications error occurred while attempting to call XMLRPC method merge_installedpackages_section:
Jun 14 08:29:01	php-fpm	327	/rc.start_packages: A communications error occurred while attempting to call XMLRPC method merge_installedpackages_section:
Jun 14 08:29:00	php-fpm	70969	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.restore_config_section).
Jun 14 08:28:57	php-cgi		notify_monitor.php: Message sent to lista-seguranca.centralit@listas.icmbio.gov.br OK
Jun 14 08:28:57	php-fpm	70969	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:28:57	php-fpm	70969	/rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:
Jun 14 08:28:57	php-fpm	70969	/rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method restore_config_section:
Jun 14 08:28:01	php-fpm	327	/rc.start_packages: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:28:01	php-fpm	327	[pfBlockerNG] XMLRPC sync is starting.
Jun 14 08:28:01	check_reload_status		Reloading filter
Jun 14 08:28:01	check_reload_status		Syncing firewall
Jun 14 08:27:59	check_reload_status		Reloading filter
Jun 14 08:27:59	php-fpm	327	/rc.start_packages: The command '/sbin/ifconfig 'bce0.521' delete '10.10.10.1'' returned exit code '1', the output was 'ifconfig: ioctl (SIOCDIFADDR): Can't assign requested address'
Jun 14 08:27:59	php-fpm	86843	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.restore_config_section).
Jun 14 08:27:59	nrpe	40186	Allowing connections from: 10.197.32.238
Jun 14 08:27:59	nrpe	40186	Listening for connections on port 0
Jun 14 08:27:59	nrpe	40186	Warning: Daemon is configured to accept command arguments from clients!
Jun 14 08:27:59	nrpe	40186	Server listening on 10.197.21.3 port 5666.
Jun 14 08:27:59	nrpe	40186	Starting up daemon
Jun 14 08:27:59	php-fpm	327	[pfBlockerNG] Starting cron process.
Jun 14 08:27:59	nrpe	60867	Daemon shutdown
Jun 14 08:27:59	nrpe	60867	Cannot remove pidfile '/var/run/nrpe2.pid' - check your privileges.
Jun 14 08:27:59	nrpe	60867	Caught SIGTERM - shutting down...
Jun 14 08:27:59	php-fpm	327	/rc.start_packages: Restarting/Starting all packages.
Jun 14 08:27:58	check_reload_status		Starting packages
Jun 14 08:27:58	check_reload_status		Reloading filter
Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: XMLRPC versioncheck: 18.0 -- 18.0
Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.host_firmware_version).
Jun 14 08:27:56	php-fpm	70969	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: XMLRPC versioncheck: 18.0 -- 18.0
Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.host_firmware_version).
Jun 14 08:27:56	pkg-static		pfSense-pkg-System_Patches-1.1.7 installed
Jun 14 08:27:56	php-fpm	86843	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:27:56	php		/etc/rc.packages: Successfully installed package: System Patches.
Jun 14 08:27:55	check_reload_status		Syncing firewall
Jun 14 08:27:55	check_reload_status		Syncing firewall
Jun 14 08:27:54	php		/etc/rc.packages: Beginning package installation for System Patches .
Jun 14 08:27:45	php-fpm	79998	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.restore_config_section).
Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: XMLRPC versioncheck: 18.0 -- 18.0
Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: XMLRPC reload data success with https://192.168.1.2:443/xmlrpc.php (pfsense.host_firmware_version).
Jun 14 08:27:42	php-fpm	79998	/rc.filter_synchronize: Beginning XMLRPC sync data to https://192.168.1.2:443/xmlrpc.php.
Jun 14 08:27:41	check_reload_status		Syncing firewall

If anyone can help me to solve this problem, Thanks!!

[kien nguyen]

Hi all, i have similar error with pfsense version 2.4.4. Do you resolve???

[Rocco83]

I have the same issue too.
Running 2.4.4p1 on XG-7100.

Apr 29 00:57:32 pf1-lipi php-fpm[347]: /rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method restore_config_section:
Apr 29 00:57:32 pf1-lipi php-fpm[347]: /rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:
Apr 29 00:57:32 pf1-lipi php-fpm[347]: /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.11.8.232:443/xmlrpc.php.
Apr 29 00:57:34 pf1-lipi php-fpm[5542]: /rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method host_firmware_version:
Apr 29 00:57:34 pf1-lipi php-fpm[5542]: /rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method host_firmware_version:
Apr 29 00:57:34 pf1-lipi php-fpm[5542]: /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.11.8.232:443/xmlrpc.php.

[Rocco83]

As by
https://redmine.pfsense.org/issues/9489#change-40418
"You have a configuration error, probably a down gateway triggering state killing. Keep the discussion on the forum."

In my case, in this specific configuration, the above comment is true (this is a test configuration).

IMO this is anyway a bug, as a gateway down should not trigger the reset of sessions.

[Derelict]

If you tell it to reset states and it resets states, where is the bug?

[Rocco83]

Correct, i had the setting enabled. Thanks again for spotting it out.

I would suggest at least to have a warning message about this checkbox, stating that:
"If one gateway is down, or goes down during the config sync, pfsense may report a failed synchronization task"

And i see two improvements:

1. i would expect to have pfsense being smart enough to, in case of connection reset, try it once again and do not report the error. This is pfsense against pfsense.
2. the gateway didn't goes down at reload time, the gateway were down also before the config sync. Meaning, the text is not consistent. Clearly, the check on gateway restart (i suppose now), but a down gateway that is down before and after a config sync should not trigger the state killing IMO.

What's your view about the 3 points above?

Thanks anyway for spotting it out and for the support!

[Derelict]

Can only prevent so much foot-shooting. Many of these checkboxes have many, many ramifications. Users should know what they are doing and understand the ramifications before checking them. The default here is unchecked because that is generally the desired setting.

The link to the book that describes this is in my signature.

The documentation would be the place for that. if you believe something there is unclear there is a feedback function at the bottom of each page.

[Rocco83]

@Derelict

I can fully understand i am pissing you off, and i am pretty sure (as i write code sometimes too) that your position likely will not change.
But please understand that i am writing the below post with the best willing.

The best reference i have found is:
https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html#gateway-monitoring
There is a note

When this is triggered, the entire state table is cleared. This is necessary because it is not possible to kill all states for the failing WAN and the LAN-side states associated with the failing WAN. Removing states on the WAN side alone is ineffective, the LAN-side states must be cleared as well.

and
https://docs.netgate.com/pfsense/en/latest/book/multiwan/multi-wan-terminology-and-concepts.html#multi-wan-state-killing-forced-switch
Where is written

This is an optional behavior, enabled by default. For information on changing this setting, see Gateway Monitoring.

At a first sight, it collide with your statement:

The default here is unchecked because that is generally the desired setting.

Also, IMO this does not change what i have reported as improvements.

And i see two improvements:

1. i would expect to have pfsense being smart enough to, in case of connection reset, try it once again and do not report the error. This is pfsense against pfsense.
2. the gateway didn't goes down at reload time, the gateway were down also before the config sync. Meaning, the text is not consistent. Clearly, the check on gateway restart (i suppose now), but a down gateway that is down before and after a config sync should not trigger the state killing IMO.

What do you think about the above two point?

For the message in the checkbox,

"If one gateway is down, or goes down during the config sync, pfsense may report a failed synchronization task"

i can agree that the manual is the correct reference but i feel that this checkbox is poorly documented.

Just check the below section, this is much more extensive.
And the manual section (again https://docs.netgate.com/pfsense/en/latest/book/config/advanced-misc.html#gateway-monitoring) is even shorter :)

So my suggestion would be to change the text of the checkbox to:

The monitoring process will flush all states when a gateway goes down if this box is checked. This may be useful with multiple gateway and backup link.
Please note that all states will be flushed, not only the one related to the specific gateway.
This is equivalent to Diagnostic -> States -> Reset States -> check "Reset the firewall state table" -> Reset 

Thank you very much for reading it all up to here,
Daniel

[Derelict]

Again, the proper forum for documentation feedback is the give feedback link on the page.