PfSense 2.2.6 XMLRPC causes lighttpd to crash / Snort / Squidguard
-
Hello,
iam using Pfsense a long time. After Upgrading from 2.2.5 to 2.2.6, i have now an issue with XMLPC Sync causing lighttpd to crash on the backup system.
Log says: kernel: pid 24197 (lighttpd), uid 0: exited on signal 11 (core dumped) on Backupsystem
Some packets..rule sets are synced without errors..some not and causing a lighttpd crash…
Packages like snort and Squidguard that sends large XMLRPC packets will crash the backup lighttpd constantly...
Despite the thread in the forum this was NOT an issue with 2.2.5?!?
I was thinking that there was an issue with the updateroutine from 2.2.5 to 2.2.6, so i installed complete new to 2.2.6..
Serversystem:Dell R310 with 4 Intel IGBE and 2 Broadcom BCE (4GB RAM)But there were new problems:
1. after new install of 2.2.6 an error directly after install occurs: to less mbuffers... and the machine stopped on the console..after hardreset on the machine.. the install was resuming..and the console Screen was showing..
2. the new 2.2.6 version was showing an very high amount of used mbuffs...so i was trying to set the mbuffers higher in system tunables..but that was ineffictive and not working... i had to create the local file /boot/loader.conf.local and edit this file..
3. after this the mbufs are OK, BUT compared to the 2.2.5 the usage is much higher ..without network traffic (i was doing this first on the backup system)...
4. So i installed the following packages:
Lightsquid Network Management 2.43
snort Security 3.2.9.1
squid3 Services 0.4.7 (with clamav activated)
squidGuard Network Management 1.9.18So i tried to increase memory for php and lighttpd with:
/etc/inc/config.inc ("memory_limit","128M"); to 500M
This was better, now Squidguards syncs were ok and not crashing the backup... but snort unfortunly was crashing the backup...
So log was php-fpm[97304]: /snort/snort_interfaces_edit.php: New alert found: A communications error occurred while attempting Snort XMLRPC sync with https://xxx.xxx.xxx.xxx:443. Failed to transfer file: enablesid-sample.conf
But this feature was turned off on both systems… I created the files:
disablesid-sample.conf
enablesid-sample.conf
modifysid-sample.conin the package dir of snort on both systems /usr/pbi/snort-amd64/local/etc/snort/snort_xxxxxxx (interfacenumber) and the error in the log was gone...
But after a sync of snort changes from master to backup, i still get this XMLRPC Error on the Mastersystem:
[ An error code was received while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload]
The Backupsystemlog shows:
php: snort_sync_cmds.php: [snort] XMLRPC pkg sync process on this host is complete…
Feb 15 12:18:47 php: snort_sync_cmds.php: [Snort] Building new sid-msg.map file for WAN…
Feb 15 12:18:46 php: snort_sync_cmds.php: [Snort] Enabling any flowbit-required rules for: WAN…
Feb 15 12:18:42 php: snort_sync_cmds.php: [Snort] Updating rules configuration for: WAN …
Feb 15 12:18:42 php: snort_sync_cmds.php: [snort] XMLRPC pkg sync: Generating snort.conf file using Master Host settings…Now iam off...here...in the meantime i had reinstalled also the master system, so both system are complete fresh new installs...
Perhaps someone have a tip?
Kind Regards..
Andreas -
Hi,
as I remember there where changes in 2.2.6 for the XMLRPC Sync code. I belive that Snort and Suricate rely on the same Sync code that is broken now.
As I understood from Bill, the sync code was writte by somebody else and he is now focusing on the transition to bootstrap with the GUIs for Snort and Suricata.
Let's hope it will be fixed on pfSense 2.3 and the new bootstrap based GUIs.Regards,
Emanuel
-
@somosane:
Hi,
as I remember there where changes in 2.2.6 for the XMLRPC Sync code. I belive that Snort and Suricate rely on the same Sync code that is broken now.
As I understood from Bill, the sync code was writte by somebody else and he is now focusing on the transition to bootstrap with the GUIs for Snort and Suricata.
Let's hope it will be fixed on pfSense 2.3 and the new bootstrap based GUIs.Regards,
Emanuel
This is correct. There is a problem with the XMLRPC Sync code and the new web server in 2.2.6 and up. I have not yet taken time to look as I have been occupied converting the Snort package to Bootstrap in preparation for the pfSense 2.3 release. For now I recommend disabling the sync option in Snort (and Suricata) until the issue is sorted out. I can troubleshoot it and attempt to fix after I complete the Bootstrap conversion.
Bill
-
The sync code didn't change at all between 2.2.5 and 2.2.6. The fix in 2.2.6 related to config sync was in upgrading the lighttpd version to a newer release that fixed a problem in it that impacted a small portion of users. Unfortunately there's yet another, different, lighttpd issue in the version in 2.2.6 that's impacting a different small subset of users.
A couple people have indicated a 'pkg install lighttpd' to get an updated version of it fixes the crashes they were seeing.
It's a non-issue in 2.3 because lighttpd is gone, because of problems like this that they keep having.
-
Hi Bill,
–-------------
This is correct. There is a problem with the XMLRPC Sync code and the new web server in 2.2.6 and up. I have not yet taken time to look as I have been occupied converting the Snort package to Bootstrap in preparation for the pfSense 2.3 release. For now I recommend disabling the sync option in Snort (and Suricata) until the issue is sorted out. I can troubleshoot it and attempt to fix after I complete the Bootstrap conversion.Bill------------
From My View, Focus on the new Version, i can live without the sync, i just cut&paste the rules to the backup...
If u have sparetime ;D after that ... it would be really great if it would be possible to include the snort snmp plugin http://www.cysol.co.jp/contrib/snortsnmp/README.SNMP ..
That would enable snort to be monitored from nagios via snmp when something is blocked.....etc..
Anyway i have to say a BIG THANK YOU to you and all Developers for your great work!
Best wishes,
Andreas
-
Hello sorry to add the issue here but I'm also seeing the following XMLRPC sync errors
Failed to transfer file: modifysid-sample.conf @ 2016-05-14 11:54:35
Failed to transfer file: emerging-compromised-ips.txt @ 2016-05-14 11:55:50
as well as the Snort XMLRRPC sync errors
Failed to transfer file: emerging-compromised-ips.txt @ 2016-05-14 11:55:50
plus this error (below) which does not make sense since I have made sure to use the same password for the admin user on both systems - checked several times on this.
A communications error occurred while attempting XMLRPC sync with username admin
thanks