XMLRPC sync errors since upgrade to 2.4.4



  • Ever since upgrading to 2.4.4 from the previous release, I get notification e-mails periodically with the message:
    " A communications error occurred while attempting to call XMLRPC method restore_config_section".
    Config sync still appears to be working and there is no other problems, however the messages do appear to indicate an issue.
    So far I've tried rebooting both of the two firewalls, disabling and re-enabling config sync, and re-configuring the dedicated sync interface on both nodes, however the problem persists. As the sync interfaces are connected with a crossover cable, the firewall rules for that interface simple allow all traffic. Any suggestions would be most appreciated.


  • Rebel Alliance Developer Netgate

    That message usually indicates that the primary cannot reach the secondary over the sync interface. It may not even be an XMLRPC issue at all, but a physical or interface/firewall/L3 issue of some kind.

    Does it sometimes work and sometimes fail? Or does it never work?

    From the primary:

    • Can you ping the sync interface address on the secondary from Diagnostics > Ping using the default source? Using the sync interface as the source?
    • Can you reach the GUI port using Diagnostics > Test Port using the default source? Using the sync interface as the source?
    • Are you seeing any blocked entries in the firewall log for the sync interface under Status > System Logs on the Firewall tab?

    On the secondary:

    • Are there any errors in the main system log (Status > System Logs on the System/General tab) on the secondary referring to XMLRPC or the GUI server daemon (nginx)?

    On both:

    • Any interface events in the main system log on either node for the sync interface?
    • Does the sync interface look OK under Status > Interfaces? Any errors? Link type/speed/duplex look as expected?


  • Hi, thanks for your reply. As mentioned the SYNC interfaces are connected via a crossover cable with no switches in between, so aside from a possible NIC issue which has appeared since the upgrade, this should rule out L2/L3/firewall problems.

    • Ping works from the primary to the secondary from both default source and sync interface source. It also works the other way around from secondary to primary.
    • Can reach the GUI port on the sync IP from the primary using both default and sync interface source.
    • No blocked firewall logs on the secondary or primary sync interfaces.
    • No unusual log messages on the secondary or primary. Oddly as well, when the problem occurs there are no errors in the log other than the one informing me a notification message has been sent. The lines immediately before that are
      "/index.php: XMLRPC reload data success with https://172.16.0.2:443/xmlrpc.php (pfsense.exec_php)"
      which indicates the sync has been OK.
    • The sync interface looks stable, no errors reported or interface events.

    The notification messages are seemingly sparodic, and it worked perfectly before the upgrade. Any other suggestions most appreciated.



  • I've disabled config sync for now and just left pfsync enabled (which appears to work fine), but if anyone does have any further ideas they would be much appreciated. It just seems odd that this began after the upgrade - the systems were in use well over a year on the previous version and never reported a config sync problem message before. Appreciate the suggestions so far.


  • Rebel Alliance Developer Netgate

    Do your firewall rules to pass the sync data use aliases? Do those aliases contain hostnames? Maybe when the XMLRPC sync triggers a filter reload the rules don't pass the traffic immediately.

    Otherwise I'm not sure how it would end up working sometimes and not others.



  • No aliases in use for the sync interfaces - as they're connected via crossover cable, the firewall rules simple allow all ICMP and TCP/UDP traffic.



  • FWIW I don't use a crossover cable and mine works fine. The hardware should be smart enough to hand this.


  • Netgate

    Auto-MDIX generally is.

    I still use crossover cables for this. One less thing to think about.



  • Hello everyone. I'm new at the forum but not so new pfSense user. I'm having similar issue as DrNick.

    About a week ago I purchased two new Netgate XG-7100 units with preinstalled pfSense 2.4.4 to be installed to our company's main production site. The original plan was to have two similar units with only one unit up and running and network cables attached to it and the other one as warm backup with power on and cloned configurations already loaded but no network cables attached. Let's call them pfSense primary and pfSense secondary. They should be in state that both of them have same IP addresses and same configurations. So if some disaster happens to pfSense primary all that needs to be done is to move network cables from pfSense primary to ethernet ports with same numbers on pfSense secondary. A job that even our non-technical persons can also do if I'm out of town or on vacation.

    As we are having pretty frequent config changes and changing VPN users it is unpractical to always take manual backups and restore them to pfSense secondary. I thought that best solution would be XMLRPC sync from pfSense primary to pfSense secondary.

    So I configured the pfSense primary to our normal production configs and took manual backup and restored it to pfSense secondary. Then I configured dedicated interface for XMLRPC sync and firewall rule for it on both units. So configurations on both units are identical besides IP address on interface used for syncing and of course only pfSense primary's XMLRPC syncing turned on.

    At first this configuration seemed to work fine. Configuration modifications done on pfSense primary were synced to pfSense secondary and no errors could be seen on logs. Couple of days later I found option Services -> Auto Config Backup. This was new to me since I hadn't seen this option on my older versions of pfSense. I thought it would be great to have offsite backup of configurations also and I turned this ACB on. After that I started to get multiple errors about XMLRPC sync (restore_config_section error) everytime I changed configurations and pfSense secondary was about to be synced:

    Nov 1 20:46:25 check_reload_status Syncing firewall
    Nov 1 20:46:25 php-fpm /firewall_rules_edit.php: Beginning configuration backup to .https://acb.netgate.com/save
    Nov 1 20:46:26 php-fpm /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.51.0.2:443/xmlrpc.php.
    Nov 1 20:46:26 php-fpm /rc.filter_synchronize: XMLRPC reload data success with https://10.51.0.2:443/xmlrpc.php (pfsense.host_firmware_version).
    Nov 1 20:46:26 php-fpm /rc.filter_synchronize: XMLRPC versioncheck: 18.8 -- 18.8
    Nov 1 20:46:26 php-fpm /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.51.0.2:443/xmlrpc.php.
    Nov 1 20:46:27 php-fpm /firewall_rules_edit.php: End of configuration backup to https://acb.netgate.com/save (success).
    Nov 1 20:46:30 check_reload_status Reloading filter
    Nov 1 20:47:26 php-fpm /rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method restore_config_section:
    Nov 1 20:47:26 php-fpm /rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:
    Nov 1 20:47:26 php-fpm /rc.filter_synchronize: Beginning XMLRPC sync data to https://10.51.0.2:443/xmlrpc.php.
    Nov 1 20:48:26 php-fpm /rc.filter_synchronize: A communications error occurred while attempting to call XMLRPC method restore_config_section:
    Nov 1 20:48:26 php-fpm /rc.filter_synchronize: New alert found: A communications error occurred while attempting to call XMLRPC method restore_config_section:

    I have a theory (tell me if I'm wrong):

    If I plug my laptop's ethernet cable straight to the LAN port of pfSense secondary and access webGui I cannot open Auto Config Backup page at all. It just keeps loading on browser and finally request times out. If I plug also WAN cable on pfSense secondary then I can open Auto Config Backup. So it seems that accessing to Auto Config Backup needs WAN to be up. This is of course logical because ACB sends configurations to Netgate's server and needs WAN for the purpose. Problem is that normally WAN cable is not attached to pfSense secondary. Is this the problem which causes also XMLRPC to hang on restore_config_section while syncing?

    Now I cannot use XMLRPC even if I turn ACB off. I think this happens because configuration has some values to be synced on ACB configuration section even if it's turned off. XMLRPC tries to sync these values on pfSense secondary and it cannot access these values without WAN connection on pfSense secondary.

    Is there some way to turn off syncing ACB configuration section? Is my only option to go back to the configuration before I turned the ACB on at first time and just not to use ACB?

    Thank you for your help!



  • @windiz

    I am not sure you are using this as it was intended. I use this feature with High Availability. It sounds like you are a perfect candidate for it. It will keep both servers synchronized realtime and allow switching of servers in the event of overload or failure. It is well documented in the pfsense documentation. The documentation is available for you to use since the event of 2.4.4.



  • @drnick-0 Yea. I'm having issues too. Are you running pfblocker-devel? I am. I wonder if it's related.



  • @reberhar

    Thanks for you answer. I have considered High Availability also and have read the documentation about it. Since we have also multi-WAN and DMZ configured I think HA would become pretty complex. Maybe too complex for me to handle if we run into issues and I have to do some troubleshooting. Besides that our WAN2 router has only one port so we would need at least one new switch in between.

    I would like to keep the setup as simple as possible so it would be easy to troubleshoot problems and in the case of failure just throw network cables to pfSense secondary and keep on rocking. I'm not saying that HA can't be done in our environment but I'm just relying good old KISS rule :). We don't need full HA in our production environment. We can tolerate at maximum 30 minutes of downtime which is plenty of time for our 24/7 operator to move network cables from one pfSense to another. But we can't tolerate longer downtimes so if we run into trouble with HA and I'm on vacation it would be really bad situation.

    So my original question is simply that is it possible to have Auto Config Backup enabled and still sync configs with XMLRPC to other pfSense which normally has not WAN connected (actually it normally wouldn't have any other network cables connected but only SYNC)?



  • @bbrendon no we're not running pfblocker or any other packages - just using the built-in firewalling, NAT & captive portal functions. I still have config sync disabled most of the time - I just enable it when I'm making config changes I want synced then turn it off again, which I can live with.
    @windiz interesting theory about auto config backup, but we're not using that either. Not to say in your case it might not be related, but I've never turned on that option here.


  • Netgate Administrator

    If you enable autoconfig backup on the secondary and it has no WAN connection it will try to backup it's config at every change pushed from the primary and fail. It will have to timeout waiting for that and if the primary tries to push another changes during that time it may fail.
    Really you should be running those as an HA pair in that situation.

    You should be able to disable ACB on the secondary though.

    Steve



  • I'm having the exact same issue as @DrNick-0 with a similar setup. I have a pair of XG-1541 1U HAs, and have been receiving the "A communications error occurred while attempting to call XMLRPC method restore_config_section" message immediately after upgrading to 2.4.4-RELEASE. Here are the answers to @jimp's questions as well:

    • Yes, I can reach the sync address from one firewall from the other.
    • Yes, I can reach both GUI ports
    • I'm not seeing any blocked entries in the firewall log for the sync interface.
    • No XMLRPC or nginx logs on the secondary.
    • No interface events for the sync interface on either firewall.
    • Sync interface looks fine on both firewalls.

    Additionally, I'm using a direct cable for the sync interface between the two firewalls, nothing's in between. Occasionally, I'll get the message "/rc.filter_synchronize: XMLRPC reload data success with https://172.16.1.3:443/xmlrpc (pfsense.host_firmware_version)," and if I sync the configuration manually through Status>Filter Reload, it seems to sync just fine, with the following logs:

    Some time afterwards (up to 30 minutes later), it'll go back to spamming the "A communications error occurred while attempting to call XMLRPC method restore_config_section" logs again. I've tried rebooting the secondary firewall to no avail, and can't reboot the primary since it's in production. Any help would be greatly appreciated.


  • Rebel Alliance Developer Netgate

    What packages do you have installed? I've seen several HA clusters running 2.4.4 and none have sync issues like this.



  • @jimp said in XMLRPC sync errors since upgrade to 2.4.4:

    What packages do you have installed? I've seen several HA clusters running 2.4.4 and none have sync issues like this.

    I have no packages installed on either firewall.


  • Netgate

    Is the webgui healthy on the secondary at the time? Can you log in there and navigate?

    Are you trying to game things without the requisite 3 public IP addresses on WAN? Can the secondary get to the internet, resolve names, etc when it is not CARP master?



  • @derelict said in XMLRPC sync errors since upgrade to 2.4.4:

    Is the webgui healthy on the secondary at the time? Can you log in there and navigate?

    Are you trying to game things without the requisite 3 public IP addresses on WAN? Can the secondary get to the internet, resolve names, etc when it is not CARP master?

    Yup, the webgui is just fine. I'm not trying to game anything, both firewalls have their own unique upstream address, and the CARP address is a different and also unique address as well. The secondary firewall can get to the Internet and resolve DNS names when it's not CARP master, I pinged google.com to check.



  • @nima304
    is 172.16.1.3 the sync IP or the LAN IP of the second router?

    @windiz
    same question for 10.51.0.2?

    The routers I upgraded last week aren't logging comm errors...

    A long time ago I did have sync issues. I seem to recall I tracked it down to Suricata and that we had selectively disabled many of the unneeded individual rules. Turns out all that had to sync and it was timing out. Solution: don't disable individual rules and it has less to process.



  • @teamits said in XMLRPC sync errors since upgrade to 2.4.4:

    @nima304
    is 172.16.1.3 the sync IP or the LAN IP of the second router?

    @windiz
    same question for 10.51.0.2?

    The routers I upgraded last week aren't logging comm errors...

    A long time ago I did have sync issues. I seem to recall I tracked it down to Suricata and that we had selectively disabled many of the unneeded individual rules. Turns out all that had to sync and it was timing out. Solution: don't disable individual rules and it has less to process.

    That's the sync IP for the second firewall. The primary's is 172.16.1.2.


  • Netgate

    This post is deleted!


  • @nima304 Thanks for digging into your setup to get to the bottom of this. I just haven't had time on my end and since things more or less work, it hasn't been a priority.


  • Netgate Administrator

    Do you have a large number of users in the config?

    Steve



  • @bbrendon said in XMLRPC sync errors since upgrade to 2.4.4:

    @nima304 Thanks for digging into your setup to get to the bottom of this. I just haven't had time on my end and since things more or less work, it hasn't been a priority.

    No problem, hopefully there's a resolution that solves it for all of us.

    @stephenw10 said in XMLRPC sync errors since upgrade to 2.4.4:

    Do you have a large number of users in the config?

    Steve

    No, literally just the admin user, but I also have LDAP auth configured.


  • Netgate Administrator

    That should be no problem as long as the user accounts are not on pfSense. A large number can introduce delays on the secondary when the sync'c config is added preventing it responding in reasonable time.

    Hmm, I'd probably start a packet capture on the secondary sync interface. Set it for a large number and wait for it to fail. See what's actually happening there.

    Steve



  • In windiz's logs, it is exactly 60 seconds from the beginning of the sync to the error and that sounds like a timeout to me. Brainstorming, how large is your config export file? We have some decently complex ones for our data center that are about 180 KB, for reference...Suricata rules, pfBlockerNG, OpenVPN, etc.

    Router2 isn't set to sync back to router1 is it? That would be a loop.


  • Netgate Administrator

    Yes, the timeout is 60s. It used to be possible to take longer than that to load the config and respond with more than ~50 users on some hardware. There have been improvements gone in since then though.

    Steve



  • @teamits said in XMLRPC sync errors since upgrade to 2.4.4:

    In windiz's logs, it is exactly 60 seconds from the beginning of the sync to the error and that sounds like a timeout to me. Brainstorming, how large is your config export file? We have some decently complex ones for our data center that are about 180 KB, for reference...Suricata rules, pfBlockerNG, OpenVPN, etc.

    Router2 isn't set to sync back to router1 is it? That would be a loop.

    Good catch, my logs are showing the same thing. While config sync isn't set at all on the secondary, the primary is syncing states from the secondary, and the secondary from the primary, as per pfSense's documentation.

    I'm going to try to blow the firewall rules open on the sync interface for both firewalls and see if that does anything.



  • Blowing open the rules did nothing, unfortunately. I'm seeing data received on the secondary firewall, so it's not a cable issue. I'll do a packet capture and see if anything interesting turns up.



  • The transmission is encrypted using TLS, so I can't actually see what's going on.


  • Netgate Administrator

    You could set the GUI to http just while you test. However you should still be able to see the TCP sequence and lack or responses.
    Make sure both nodes are time sync'd and then compare the log entries. Does the secondary log anything during that 60s window?

    Steve