Unable to Sync Rules without XMLRPC Code 2 error

gjungle

G'day all,

Been running pfsense on a single PC here at work for a while now and all was running smoothly.

Recently we lashed out and bought a rack-mounted dual Wrap 1-2 kit (http://www.yawarra.com.au/product.php?productCode=HW-WR12-R) so we could add some redundancy to our firewall configuration.

I successfully managed to load our configuration from our existing PC-based pfSense onto the Wrap kit, however am experiencing difficulties in getting the CARP to work.

Our PC setup is slightly complex and is as follows:
1 x WAN + 6 x VLAN's (LAN included).

On the Wrap boards I have configured it as follows:
1 x WAN + 6 x VLAN's (LAN included) + 1 x SYNC (dedicated w/ direct cross-over cable between the two Wrap boards).

On the Wrap boards the cards are sis0 (with all the VLANs), sis1 (WAN), and sis2 (SYNC).

All in all we have approximately 230 filter rules.

The problem comes when I try to sync the rules over, everything else seems to sync fine. However when I try to sync the filter rules over I am getting the following error in the system log:

php:: New alert found: An error code was received while attempting XMLRPC sync with https://10.126.0.2:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload
php:: An error code was received while attempting XMLRPC sync with https://10.126.0.2:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload.

Each time I thought I was getting close to figuring out exactly what was causing the issue I would be proven wrong. Originally I thought it may have been because of a control character or some such in a filter description, but no such control characters where found. Then I thought it may have been because I had " => " in the rules, so I changed them to " -> " and still it didn't work.

If I choose NOT to sync the rules, it all works fine.

I also found that if I deleted the first 40 rules of the LAN interface and pretty much all the rules on all the other interfaces then I could sync successfully.

So this morning instead of doing a restore of all the filter rules I set about individually retyping each rule into the master node. I typed in a maximum of 5 rules before "Applying" so I could see if the sync worked or not. I got so far along and then the error re-occured, so I deleted that rule… it still didn't sync! So I deleted the last 5 rules I added in and STILL I got the XMLRPC error - and it was now in a state where it had worked not 10 minutes before!

Can someone help me please? I am at my wits end as to what is causing this problem and I see lots of other people on this forum using CARP fine so it must work...

Both PC version and wrap version are pfSense 1.0.1.

Glenn.

cmb

There have been a lot of changes since 1.0.1, try a recent snapshot (soon to be 1.2 beta).
http://snapshots.pfsense.com/FreeBSD6/RELENG_1/

gjungle

I'm dubious about running a snapshot as a production firewall as my understanding was they were used in development cycles…

Are you sure it should be ok to run a snapshot?

cmb

There are a lot less known bugs in the current snapshot than there are in 1.0.1. :)

But, it's possible there could be new bugs that affect your particular setup. Unlikely, but always possible. If there are issues, you'll have them before you get up and running. It's not likely to break once it's up and running.

gjungle

ok well now we're getting a different error at least.

I'm now running the snapshot from the 27th of March and once again everything sync's fine except when I select to sync the rules :'(

The error I'm getting now (and it only appears when I select "sync rules" from the Carp Settings menu") is:

php: : A communications error occured while attempting XMLRPC sync with https://10.126.0.2:443.
php: : New alert found: A communications error occured while attempting XMLRPC sync with https://10.126.0.2:443.

???

:'( :'( :'(

sullrich

1. Both machines need to be on the same version
2. Both admin passwords must match
3. Both machines need to be either https or http, not mix matched.
4. Remove any special characters in descriptions

gjungle

@sullrich:

1. Both machines need to be on the same version
2. Both admin passwords must match
3. Both machines need to be either https or http, not mix matched.
4. Remove any special characters in descriptions

1. Yup, reloaded them from the same embedded image yesterday.
2. Yup, no special characters but upper case, lower case, and numbers.
3. Yup, both machines are https.
4. Yup, just went through the exported filter set and removed everything that may have been classed as a special character - including the following -,(,),>,=,;,:,, Then re-imported the rules into the master firewall. Now the only thing in the rules is alpha numeric characters.

Still get the same error when I try to sync the rules though :-\ :'(

sullrich

Ensure that you have pass rules between the two CARP sync members on either 80 or 443 or the custom webConfigurator port.

Also, both hosts need to be running on the same webConfigurator port.

gjungle

G'day Scott,

I followed the instructions (and just double-checked them) and the SYNC port has a any => any rule on it so all traffic can pass.

Also both hosts are running on the same webConfigurator port (https).

I can also get it to sync properly as long as I tell it not to sync the rules, which indicates that the synchronization settings themselves are correct…

I have now gone out and purchased a new cat 6 cross-over cable and tried that, but to no avail. It also takes a good 4+ minutes before it gives me the error - I'm wondering if there I'm hitting a timeout or something? I didn't think my 230 odd rules was excessive, maybe it is?

sullrich

Do you have special characters (non-ascii) in the description field of the rules? If so, remove them.

Even German glyphs(sp) if they exist.

gjungle

G'day again,

Nope, no special characters. Am situated in Australia and don't even have special characters on my keyboard :)

After reading a bit more on this forum I tried enabling the device polling on both boxes. This did make a change - now the error was back to what it was originally:

php:: New alert found: An error code was received while attempting XMLRPC sync with https://10.126.0.2:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload
php:: An error code was received while attempting XMLRPC sync with https://10.126.0.2:443 - Code 2: Invalid return payload: enable debugging to examine incoming payload.

So I disabled the device polling and am now back at this error:

php: : A communications error occured while attempting XMLRPC sync with https://10.126.0.2:443.
php: : New alert found: A communications error occured while attempting XMLRPC sync with https://10.126.0.2:443.

Tomorrow I will try setting the communication to HTTP instead of HTTPS, and resetting the passwords on both boxes to something simple like 'pfsense'.

Let me know if you have any other thoughts :-\

sullrich

Please send your config.xml to sullrich@gmail.com

gjungle

I'll send it first thing tomorrow when I get to work..

sullrich

Before you send and just for grins, reboot the secondary CARP cluster member and try to sync again.

gjungle

I've rebooted both of them multiple times whilst making changes and verifying the situation to no effect :-\

However this morning I have had a breakthrough! In reconfiguring the firewall in preparation for sending you the config.xml, I found that it worked if I set the protocol on both firewalls to HTTP. Just to verify I changed the protocol on both firewally back to HTTPS and sure enough got the "communications error" again when trying to sync.

I'll reset the password to a more cryptic one and re-enable device-polling in HTTP mode to see if it still works.

At least we've managed to narrow it down a bit further… did you still want a copy of my config.xml?

sullrich

Did you change the admin username?

http://wiki.pfsense.com/wikka.php?wakka=CARPConfigurationSyncTroubleShooting

gjungle

No, I didn't even know you could change the admin username :o

smurfb

I got the same problem. Running 1.2-RC2. Did you guys ever figure out what was wrong?

superwutze

i recently updated 2 carp-firewalls from 1.0.1 to 1.2-rc2 (because of the failover-pool-feature) and am now stuck with the same problem:

php: : An error code was received while attempting XMLRPC sync with username admin https://10.10.11.252:8443 - Code 2: Invalid return payload: enable debugging to examine incoming payload

as long as there are no changes on node1 the sync works, states get synced, carp works, but when changes are made then the error shows up. but the states sync on. so live carp works, changes in rules or aliases not.

i tried all the above to no success.
any new ideas?

thanks in advance,
andy

nic

I too am having this problem. 1.2-RC2, recent upgrade. We have tried everything else listed above and are unable to get anything other than:
Oct 2 16:28:00 pri php: : Beginning XMLRPC sync to http://192.168.255.2:80.
Oct 2 16:28:00 pri php: : An error code was received while attempting XMLRPC sync with username admin http://192.168.255.2:80 - Code 2: Invalid return payload: enable debugging to examine incoming payload
Oct 2 16:28:00 pri php: : New alert found: An error code was received while attempting XMLRPC sync with username admin http://192.168.255.2:80 - Code 2: Invalid return payload: enable debugging to examine incoming payload

We have tried HTTP as well as HTTPS. We have checked the user name and password. We have no luck with sync of rules turned on or off.
Our question is how do we "enable debugging to examine incoming payload"? We see know way to do this, and cannot find mention in the documentation. We have tried changing the default setting for debug in the class constructor for XML_RPC_Client and in the "new" call for XML_RPC_Server (in xmlrpc.php). We are not even sure where we should expect to see this debugging information emerge.

Can anyone provide some guidance on this?

Thanks in advance,
-nic