Master not sending XML RPC sync data to Backup node

[michlschmid]

I noticed that my backup node is suddenly out of XML sync although it's activated on master.
State sync is working fine.

The sync is configured to work on an dedicated PfSync VLAN where the state sync as well as the XML sync should go.

I doublechecked the following:

• Both nodes have the same protocol (HTTPS) and ports (443)
• Traffic on PfSync is both allowed, also no related log entries on both stating sth. else
• PfSync enabled on both
• XML RPC Sync enabled on Master only pointing to -> Backup
• XML RPC Sync settings on Backup are empty
• Both nodes have the same admin user and password installed

Generally no changes were made to the HA settings compared to the last working state.

I have no alerts in my WebGUI and my General Log on master says nothing critical:
{{{
<27>Jun 8 09:44:13 php-fpm[29604]: /system_hasync.php: waiting for pfsync…
<27>Jun 8 09:44:44 php-fpm[29604]: /system_hasync.php: pfsync done in 30 seconds.
<27>Jun 8 09:44:44 php-fpm[29604]: /system_hasync.php: Configuring CARP settings finalize…
}}}

TCPDumped the traffic on the PfSync Interface on Master and I can only see "state" Packets
leaving the interface towards backup.

I suppose the XML data never hits the wire.

Any ideas on this?
Where to go next for debugging?
Which process triggers the sync on master?

Thanks for any hint!

My Version: 2.3.1p1

[nodau]

same here with 2.3.1 p5

[michlschmid]

Hi bahsig,

my problem was that I tried to start a 3rd party script (ElasticSearch Beats binary in my case) via the shellcmd package.
As Beats is not a service by default it ran as a program and didn't provide an exit code to shellcmd.
So in the end shellcmd waited to infinity for Beats exit code.
Due to the daisy chaining of shellcmd in the PfSense / FreeBSD boot process it blocked the machine from booting. ;-)

Once I killed the binary from the console / SSH booting finished and PfSense worked as expected with syncing, etc.

Sadly there is no alert or system stat that shows you the FWs "boot state". ;-(

Hope that helps.

• Michl