CARP - Backup crashes while Suricata XMLRPC-Sync



  • Hi,

    I've a CARP setup running for years. Now the WebConfigurator at backup crashes when I save some changes in Suricata at Master. Suricata service is running at backup and the sync seems to complete successfully.

    This is the corresponding summary of master system.log:

    
    Mar 25 15:27:32 firewall1 check_reload_status: Syncing firewall
    Mar 25 15:27:32 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync is starting.
    Mar 25 15:27:32 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync sending auto-SID conf files to https://10.166.48.3:1111.
    Mar 25 15:27:32 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync sending auto-SID conf files to https://10.166.48.3:1111.
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync sending auto-SID conf files to https://10.166.48.3:1111.
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync auto-SID conf files success with https://10.166.48.3:1111 (pfsense.exec_php).
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] Beginning package configuration XMLRPC sync to https://10.166.48.3:1111.
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] Package configuration XMLRPC sync successfully completed with https://10.166.48.3:1111.
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync sending reload configuration cmd set as a file to https://10.166.48.3:1111.
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync reload configuration success with https://10.166.48.3:1111 (pfsense.exec_php).
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync sending https://10.166.48.3:1111 cmd to execute configuration reload.
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync reload configuration success with https://10.166.48.3:1111 (pfsense.exec_php).
    Mar 25 15:27:33 firewall1 php-fpm[79198]: /suricata/suricata_global.php: [suricata] XMLRPC sync completed.
    Mar 25 15:27:34 firewall1 php-fpm[97953]: /rc.filter_synchronize: Beginning XMLRPC sync to https://10.166.48.3:1111.
    Mar 25 15:27:34 firewall1 php-fpm[97953]: /rc.filter_synchronize: An error code was received while attempting XMLRPC sync with username admin https://10.166.48.3:1111 - Code 2: Invalid return payload: enable debugging to examine incoming payload
    Mar 25 15:27:34 firewall1 php-fpm[97953]: /rc.filter_synchronize: New alert found: An error code was received while attempting XMLRPC sync with username admin https://10.166.48.3:1111 - Code 2: Invalid return payload: enable debugging to examine incoming payload
    Mar 25 15:27:35 firewall1 php-fpm[97953]: /rc.filter_synchronize: Message sent to admin@media-management.at OK
    Mar 25 15:27:35 firewall1 php-fpm[97953]: /rc.filter_synchronize: Beginning XMLRPC sync to https://10.166.48.3:1111.
    Mar 25 15:27:35 firewall1 check_reload_status: Reloading check_reload_status because it exited from an error!
    Mar 25 15:27:35 firewall1 kernel: pid 94827 (check_reload_status), uid 0: exited on signal 10 (core dumped)
    Mar 25 15:27:35 firewall1 check_reload_status: check_reload_status is starting.
    Mar 25 15:27:35 firewall1 check_reload_status: fcgipath from environment /var/run/php-fpm.socket
    
    

    Backup system.log:

    
    Mar 25 15:27:33 firewall2 check_reload_status: Syncing firewall
    Mar 25 15:27:33 firewall2 php: suricata_sync_cmds.php: [suricata] XMLRPC pkg sync: Update of downloaded rule sets requested...
    Mar 25 15:27:33 firewall2 php: suricata_sync_cmds.php: [Suricata] Emerging Threats Open rules are up to date...
    Mar 25 15:27:34 firewall2 php: suricata_sync_cmds.php: [Suricata] Snort VRT rules are up to date...
    Mar 25 15:27:34 firewall2 php: suricata_sync_cmds.php: [Suricata] The Rules update has finished.
    Mar 25 15:27:34 firewall2 check_reload_status: Syncing firewall
    Mar 25 15:27:34 firewall2 php: suricata_sync_cmds.php: [suricata] XMLRPC pkg sync: Generating suricata.yaml file using Master Host settings...
    Mar 25 15:27:34 firewall2 php: suricata_sync_cmds.php: [Suricata] Updating rules configuration for: DMZ ...
    Mar 25 15:27:35 firewall2 kernel: pid 84359 (lighttpd), uid 0: exited on signal 11 (core dumped)
    Mar 25 15:27:38 firewall2 php: suricata_sync_cmds.php: [Suricata] Enabling any flowbit-required rules for: DMZ...
    Mar 25 15:27:38 firewall2 php: suricata_sync_cmds.php: [Suricata] Building new sid-msg.map file for DMZ...
    Mar 25 15:27:39 firewall2 php: suricata_sync_cmds.php: [suricata] XMLRPC pkg sync: Checking Suricata status...
    Mar 25 15:27:39 firewall2 php: suricata_sync_cmds.php: [suricata] XMLRPC pkg sync: Suricata is running...
    Mar 25 15:27:39 firewall2 php: suricata_sync_cmds.php: [suricata] XMLRPC pkg sync process on this host is complete...
    
    

    After that the backup WebGUI isn't accessible any more till I restart it from the console and the master reports an XMLRPC sync failure.
    The last item I've changed in Suricata setup before this happens was the Snort VRT rule name to "snortrules-snapshot-2976.tar.gz", since the old didn't work any more.

    All other XMLRPC syncs work without any issues.

    Any idea how to resolve this?



  • There are currently issues with the XMLRPC Sync on both Suricata and Snort.  The bug is on the list to be fixed, but I just haven't gotten to it yet.  Spent the time and energy initially on getting the packages converted to Bootstrap for pfSense 2.3 and then implementing inline IPS mode for Suricata.  I will try and get the XMLRPC problem fixed in the next release of the package.

    Bill



  • Good news. Thanks for the info.



  • Looks like it's crashing lighttpd. That's probably (yet another) lighttpd bug (there are reasons it's gone in 2.3) which is reportedly fixed in the latest lighttpd. If you run 'pkg install lighttpd' on the secondary, then reboot, that'll likely fix if the only issue is lighttpd crashing.



  • Yes, only the webconfigurator crashes. When I restart it from shell everything is nice again and the sync seems to be applied.
    I will try the lighttpd update when I'm back at work on Tuesday. Thanks.



  • Yes, with the latest lighttpd version the issue is gone and the sync finishes quickly.
    Thank you very much.



  • I have a similar issue but not exactly identical.

    Two boxes with identical hardware. Intel Pro 1000 NICs. Dedicated interface for pfsync. One interface for LAN, and one for WAN. The WAN interfaces of the pfsense boxes are connected to an HP 3500yl which is itself connected via fiber to our ISP.

    The LAN interfaces of the pfsense boxes are connected to the same blade of an HP 7506.

    I have setup the primary to sync to the backup via the HA menu for just pfsync, firewall rules, and virtual IPs. When I create a CARP IP for the WAN interface and click SAVE, everything works fine. The backup firewall gets the VIP, the master remains the master, and life is good.

    But when I create the LAN virtual IP on the master and click SAVE, the backup firewall quits responding. No attempts to hit its LAN IP for mgmt will work. It requires a hardware reboot.

    Do we think that this is something to do with this bug or is it the HP 7506 not passing some sort of traffic between the two boxes that needs to be passed?

    Thanks,
    coach



  • @coachmark2:

    Do we think that this is something to do with this bug or is it the HP 7506 not passing some sort of traffic between the two boxes that needs to be passed?

    That has nothing to do with this. Worst case, this causes the config sync to fail to happen, so nothing changes on the secondary. Also not likely it has anything to do with your network. Rough guess, maybe you're setting the CARP IP to the same IP as the LAN IP of the secondary, or something similar where the presence of the VIP breaks the IP of the secondary. Start a new thread describing what you're doing and what you're seeing.


Log in to reply