A communications error occurred while attempting xmlrpc sync

[nwt_admin]

As the subject states, I'm receiving that dreadful message that so many others have posted about.  I have searched and searched these forums, and still have yet to find a definite answer to my specific problem.

We have two pfSense boxes and they were previously syncing PERFECTLY before I changed the passwords once an employee left.  As soon as I changed the passwords, that's when I noticed the communications errors.  I am still a noobie at this, and didn't realize I had to change the admin password within the CARP section, so I know I'm the cause for this break.  However, once I realized what I had done I changed all passwords to reflect the new one, and even reverted back to the old password just to test.  Still receiving the sync errors.  Both machines have the same admin password, and that password is the same one within the CARP settings of the MASTER.

The only thing I've been able to find is that I'll have to rebuild that slave box, but I prefer not to go that route if I can.  Any suggestions???

[nwt_admin]

Anyone??

[ssheikh]

Version? Logs? Setup info? Have you acknowledged the alerts?

[nwt_admin]

pfSense version 2.1-RC1 (amd64) - FreeBSD 8.3-RELEASE-p9.

Setup is exactly how it's described in the manuals. I have everything set appropriately on the Master, and ensured that I did not enter data where it was not supposed to be on the Slave.  Like I said, the setup worked perfectly before until I changed the system's admin password.  I'm just wondering if I should rebuild the sync interface and call it a day.

My Master's LAN is 192.168.1.2 and the SYNC on it is 172.x.x.2, and the Slave's LAN is 192.168.1.3 and the SYNC on it is 172.x.x.3

Here's the logs (for obvious reasons I blocked out the public IPs and our company email address):

Sep 24 13:11:51 php: rc.filter_synchronize: New alert found: A communications error occured while attempting Filter sync with username admin http://x.x.x.x:80.
Sep 24 13:11:51 php: rc.filter_synchronize: Could not send the message to xxx@xxx.com – Error: could not connect to the host "smtp.gmail.com": ??
Sep 24 12:11:52 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:12:05 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:12:19 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:12:26 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:12:35 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:13:05 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:13:50 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:13:54 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:14:36 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:15:22 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:16:05 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:16:08 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:16:46 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:16:54 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:17:26 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:17:39 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:18:06 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:18:27 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:19:06 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:19:17 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:20:05 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:20:53 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:21:41 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:22:28 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:23:16 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:23:55 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:24:35 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:25:07 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:25:13 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:25:53 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:26:10 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:26:33 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:26:51 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:27:13 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:27:27 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:27:52 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:28:08 kernel: arp: 192.168.1.3 moved from 88:f0:77:d4:b6:b1 to e0:91:f5:94:3b:52 on re2
Sep 24 12:28:31 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:29:06 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:29:12 kernel: arp: 192.168.1.3 moved from e0:91:f5:94:3b:52 to 88:f0:77:d4:b6:b1 on re2
Sep 24 12:29:41 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:30:16 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:30:51 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:31:26 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:32:01 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:32:36 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:33:11 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2
Sep 24 12:34:00 kernel: arp: 00:24:21:b9:cd:a0 attempts to modify permanent entry for 192.168.1.2 on re2

Thanks for your help.

[nwt_admin]

Anybody? Help is appreciated…this issue is urgent and I cannot figure it out...

[nwt_admin]

Resolved. Please close.

[math]

Hello nwt_admin,

I've the same problem with this version (2.1 adm64b) and a similar architecture.

And like you I searched in many forums but no solutions found.

Can you explain how you have resolved this problem ?

Thanks

[cr_hyland]

Having this issue too on a previously perfect Carp setup with 2.0x before updating to 2.1

Since the update xmlrpc sync refuses to work and also sync for pfblocker fails.

Anyone any clues?

[jimp]

An error during XMLRPC sync and an error during FILTER sync are two very different things and can happen for two very different reasons.

XMLRPC sync is the actual configuration copy from primary to secondary. The filter sync step is when the secondary reloads a few areas after the sync happened. A filter sync error on its own would indicate that the configuration synchronized but reloading those settings produced an error somewhere on the secondary.

In either case, make sure that you have checked the system logs on the secondary for potential issues.

[cr_hyland]

The exact error we are seeing is this

10-03-13 14:38:42 [ A communications error occurred while attempting XMLRPC sync with username admin http://172.17.1.2:80.]

Usernames have been checked and double checked. Firewalls rebooted and update 2.1 firmware reapplied several times but no joy.

This is all i see in the logs on master firewall
Oct 2 15:53:38 php: rc.filter_synchronize: New alert found: A communications error occurred while attempting XMLRPC sync with username admin http://172.17.1.2:80.
Oct 2 15:53:38 php: rc.filter_synchronize: A communications error occurred while attempting XMLRPC sync with username admin http://172.17.1.2:80.
Oct 2 15:53:38 php: rc.filter_synchronize: XML_RPC_Client: Connection to RPC server 172.17.1.2:80 failed. Operation timed out 103

Theres nothing in the logs on the slave.

[nwt_admin]

Just as an FYI I was receiving BOTH the XMLRPC and FILTER error messages.

For those that are still having this issue: what caused this error to occur?  In my situation, it was my own doing as I changed the admin password without changing it anywhere else within the SYNC.  Even after fixing the password throughout pfSense, the issue still persisted.  What I ended up doing was rebuilding the SYNC interface (completely remove it and recreate it) and then restarted the slave machine.  Upon reboot, boom…no more XMLRPC issues.

This may not solve everyone's problem, but it's definitely an easy first step to take.

[jimp]

Make sure the GUI is running on the same port on both systems, and that you have a firewall on the interface in question on the secondary to accept that traffic. It looks like it's either hitting the wrong port or the firewall on the secondary is dropping the packet.

[cr_hyland]

Ok so this seems to be down to the firewall rules on the slave dropping traffic (thanks jimp).

Seeing in the logs traffic being dropped from carp ip one to carp ip two even though there is an allow all rule on the carp interface which worked perfectly before 2.1.

Also the rule being triggered is @39 block drop in log quick proto tcp from webconfiguratorlockout:1to any port = http label "webConfiguratorlockout"

Tried disabling the antilockout rule for web configurator but it didn't help.</webconfiguratorlockout:1>

[jimp]

The anti-lockout rule allows access to the GUI port
The webConfiguratorlockout table is populated with IPs that fail GUI auth several times in a row.

For the firewall rules, make sure you read the 2.1 release notes, we did put a note in there about the rules needing changed if you didn't have a pfsync rule already.

[ceylani]

Another reason is that if you didn't add a gateway record for each subnet, a route record for each subnet may have been created  (all of its subnet can be routed to default gateway) in the route table on secondary server. Actually it doesn't need to routing on broadcast network.
For example:
WAN : 192.168.1.5 -> default GW -> 192.168.1.1
SYNC: 172.16.1.1 -> no GW
LAN: 10.10.10.1 -> no GW

Sometimes PfSense assign a gateway for SYNC or LAN subnets in the route table as automatically.
For example: 172.16.1.0/30 GW 192.168.1.1

So secondary server takes over the CARP IP status as MASTER to itself. Also there are some packet loss on the networks. I had added new gateways for each subnet and my all problems solved by itself :)

[gordc]

I was able to resolve this issue by assigning two new IP addresses to the carp interfaces.

Master was 172.30.4.2 - changed to 172.30.4.3
Secondary was 172.30.4.3 changed to 172.30.4.4

This seems to have fixed the problem.

[Kipp]

Also see thread http://forum.pfsense.org/index.php/topic,68439.0.html, if the suggestions in this thread don't help as they may be similar issues.