XMLRPC Sync no longer working (HTTP/1.0 413 Request Entity Too Large)
We are on 2.1-BETA0 (i386) built on Tue Sep 18 20:20:31 EDT 2012 and our H/A sync has stopped working. The following error shows up in the logs:
php: : Beginning XMLRPC sync to https://x.x.x.xx:443
php: : An error code was received while attempting XMLRPC sync with username admin https://x.x.x.xx:443 - Code 5: Didn't receive 200 OK from remote server. (HTTP/1.0 413 Request Entity Too Large)
If we clear the error, restart the firewall, then un-check / re-check HA Sync, the firewalls sync up again.
Later in the day when we start to add stuff - it breaks again.
How large is your config.xml? Are there any particular sections that are very large?
I don't think anything has changed in the sync code in quite a while (At least since June) so I don't think it would be prone to breaking spontaneously, unless something inside the config is causing it…
I ran into the same thing on a pair of boxes running 2.1-BETA0 (i386) - 20120913 nano; I figured out that the disk was full, so it was rejecting the uploaded xml. Have you checked your disk space when you have received the error?
So this popped up again today, and it is specifically the /tmp drive that is full:
$ df Filesystem 512-blocks Used Avail Capacity Mounted on /dev/ufs/pfsense0 3780028 573580 2904046 16% / devfs 2 2 0 100% /dev /dev/md0 78812 78752 -6244 109% /tmp /dev/md1 118492 36220 72796 33% /var /dev/ufs/cf 101055 3743 89228 4% /cf devfs 2 2 0 100% /var/dhcpd/dev
I'm wondering if there is an easy way to parition the extra 60G on the ssd for /tmp; I would do this without hesitation on a linux box, but I'm not as familiar with partitioning and mounting in BSD. Also, here is a listing of the /tmp file sizes:
$ du -sh /tmp/* 38M /tmp/PHP_errors.log 2.0k /tmp/apinger.status 72k /tmp/config.cache 0B /tmp/config.lock 2.0k /tmp/dhcpd.sh 0B /tmp/filter.lock 0B /tmp/ipsecdns.lock 42k /tmp/lighttpdcompress 4.0k /tmp/mnt 2.0k /tmp/ovpns1_router 0B /tmp/ovpns1up 2.0k /tmp/pfSense_version 2.0k /tmp/pfctl_si_out 6.0k /tmp/pfctl_ss_out 0B /tmp/php-fastcgi.socket-0 0B /tmp/php-fastcgi.socket-1 0B /tmp/php_errors.txt 2.0k /tmp/re0_defaultgw 0B /tmp/re0_defaultgwv6 0B /tmp/resolvconf.lock 2.0k /tmp/rules.boot 16k /tmp/rules.debug 16k /tmp/rules.debug.old 2.0k /tmp/rules.limits 0B /tmp/shm1000.lock 40k /tmp/system-processor.rrd-4year.png 40k /tmp/system-processor.rrd-8hour.png 38k /tmp/system-processor.rrd-day.png 38k /tmp/system-processor.rrd-month.png 38k /tmp/system-processor.rrd-quarter.png 36k /tmp/system-processor.rrd-week.png 34k /tmp/system-processor.rrd-year.png 0B /tmp/tmpHOSTS 2.0k /tmp/uploadbar 0B /tmp/xmlrpc.lock
I'm wondering if there is an easy way to parition the extra 60G on the ssd for /tmp;
You are apparently running the nano-BSD based variant of pfSense where /tmp is a "ram disk" (/tmp is mounted on /dev/md0, a "memory disk") in order to minimise the writes to the hard drive.
Your du output suggests your /tmp has filled up due to lots of reports in /tmp/PHP_errors.log. That would be worth investigating.
Yes.. I had to scp it off of there and look at it. The PHP_errors.log was 38M of this one line:
PHP Warning: stristr(): Empty delimiter in /etc/inc/interfaces.inc on line 3872
I'm not sure if this has been fixed in the latest, but it would be worth looking at.
The stristr issue is due to $ip being blank in find_carp_interface. This has been reported in Redmine http://redmine.pfsense.org/issues/2645 and a fix proposed there. Someone who knows what the various forms of possible XML data structures in the CARP VIP section of the config can be could look at this and fix it. Pi Ba has proposed a solution that someone could review, test and commit.