High Avail. Sync Doesn't Work - version 2.3.3 and 2.3.3-p1

[ErsanY]

Hi,

I am not very good with unix,linux or freebsd. I can only run some simple commands.
I have attached the logs as screenshots.

I have a pair of virtual pfSenses, one is master the other one is backup. The pfSense version is 2.3.3
I have a second pair of physical pfSenses one of them is master and the other one is backup. The pfSense version is 2.3.3-RELEASE-p1

When I do a change on the master pfSense, it tries to sync the backup pfSense but it fails. And it creates the Warning message on the Dashboard.
The warning message is this:
Settings Sync

A communications error occurred while attempting XMLRPC sync with username admin https://10.x.x.x:443. @ 2017-03-31 15:58:52
A communications error occurred while attempting XMLRPC sync with username admin https://10.x.x.x:443. @ 2017-03-31 16:00:34
A communications error occurred while attempting XMLRPC sync with username admin https://10.x.x.x:443. @ 2017-03-31 16:02:16
A communications error occurred while attempting XMLRPC sync with username admin https://10.x.x.x:443. @ 2017-03-31 16:04:52

While this is happening somehow the backup pfSense stops responding and freezes. When I try to login again Web Configurator it doesn't bring the login page.

But I still can login to pfSense via putty CLI. I restart the Web Configurator by selecting the option 11. And I still doesn't get the login page.

When I reboot the backup pfSense, then I am able to login and I trigger the High Availability Sync via saving it again, on the master pfSense, and it synchronizes, until the problem happens again a few days later. So the Synchronization works for a few days.
Obviously I can't keep rebooting the backup pfSenses once or twice a week. There should be a proper solution. But I have no idea what it is.
I should also say that the master pfSenses are always fine. They never freeze, nothing happens to them. They work just perfectly fine.
The physical pfSenses have access directly to internet. Both master and backup. But there are very few connections made through them. They are new and I can't use them as the main firewalls until I fix this problem.
The virtual pfSenses doesn't have access to Internet, they just sit and actually there is no traffic running on them. No HAProxy, no CPU no memory usage.

I have done some research on internet and it may be a problem about PHP-FPM. Also I can see it on the logs. I don't know how to identify the problem and what to do to fix the problem permanently.

If anybody can help on this I will really appreciate, because since 1 month I couldn't do much to fix it. It's becoming a nightmare recently.
If I have missed anything please write and I will explain/attach as quick as possible.

Kind Regards

[sthames42]

As it happens, I am implementing the same thing with the same version. Right now, I'm using a Virtual Box network lab but the configuration is exactly like my production environment. Everything is working great for me.

Start by reviewing this: https://doc.pfsense.org/index.php/Configuring_pfSense_Hardware_Redundancy_(CARP) and this: https://doc.pfsense.org/index.php/CARP_Configuration_Sync_Troubleshooting.

Also, post a screenshot of your interface assignments (Interfaces->(assign)) and High Availability Sync configuration.

I had a similar problem as you because the interfaces were not assigned in the same order. The screenshots will indicate if that might be your problem.

[Derelict]

Make sure the secondary can access the internet and can resolve names while it is CARP BACKUP.

You can clear that nginx issue (which is really a php-fpm issue) using console/ssh options 16 followed by 11. You do not have to reboot.

[ErsanY]

Thanks a lot guys. I wasn't aware of option 16 on pfSense CLI. I have used it and it seem to fix the Synchronization problem on both virtual and physical pfSenses. I indeed appreciate your helpful comments.

On the other hand, I started to get another error. I have created an IP alias on the master virtual pfSense. And then I have seen that it appeared in the backup pfSense. Which is the desired behavior. But then I received the following error message on the master pfSense, even though the synchronization worked without any visible problem. Then I have deleted that IP alias and the master pfSense has generated the same error one more time. Here are they:

An error code was received while attempting Filter sync with username admin https://10.x.x.x:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload @ 2017-04-03 18:11:00
An error code was received while attempting Filter sync with username admin https://10.x.x.x:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload @ 2017-04-03 18:21:52

Should I worry about this error messages? How can I stop it?

Kind Regards.

[sthames42]

The synchronization process is bidirectional only for the states (pfsync). If you are providing an XMLRPC configuration on the backup then you might get this message.
AFAIK, you need to provide the XMLRPC sync config only on the master. The backup should have no XMLRPC config pointing back to the master.
Other than that, make sure you're allowing all traffic on the SYNC interface.
As I said, I'm working on the same thing and it's all working for me.

[Derelict]

An IP Alias VIP will not sync unless it is riding on a CARP VIP because the same IP Alias active on both nodes at the same time will create an IP address conflict.