[SOLVED] Firewall rule on CARP interface keeps being deleted after sync

  • I have 2 firewalls set up with pfsense 2.0 (RC3) which are supposed to sync between each other. Unfortunately after the sync the firewall rule to ensure that communication between the 2 systems can pass through is removed on the backup server.
    Since the release is now available I upgraded via the hot update feature which was fine but the problem persists.

    I found one thread in here of someone who describes a similar behaviour in 2009 (or so) but he never got any reply.

    Hope someone can help me out on this, because it's quite unnerving to have to implement the rule everytime i make a change on the master firewall.

  • Can you post screen shots on the master node of the rules and then before and after on the backup node? Are you using a separate interface for pfsync?

  • Thanks for your quick reply.

    Since the hardware is not the same on my 2 firewalls its rather difficult to use the same interface for CARP - however i believe it should be sufficent to have them both named the same, shouldn't it?

    Here's how i set up the firewall rules on both systems:
    Master just before i sync:

    Backup before i sync:

    Backup after sync:

    Master after sync:

    As you can see the rule on the master looks fine but on the backup FW it is removed right after the sync (however the changes are synched correctly as long as the rule is available).

  • Can you post a screen shot of your CARP settings?

  • Sure:

    For obvious reasons I blanked out the username name used for the sync.

    BTW: I allready tried the option "No XMLRPC Sync" on the CARP rule (on the master FW) without success.

  • Are there any errors in either of the system logs when you sync the config?

  • Sorry for the late reply.
    Nothing eye-catching. Only what you would expect since the sync does not work.

    On the first sync i get things like this on the backup firewall:

    check_reload_status: Syncing firewall
    check_reload_status: Reloading filter
    php: : Config sync not being done because of missing sync IP (normal on secondary systems).
    check_reload_status: Syncing firewall
    check_reload_status: Reloading filter
    php: /xmlrpc.php: ROUTING: setting default route to xxx.xxx.xxx.xxx
    apinger: Exiting on signal 15.
    apinger: Starting Alarm Pinger, apinger(54067)
    php: /xmlrpc.php: Resyncing OpenVPN instances.

    and on the master firewall i get:
    check_reload_status: Syncing firewall
    check_reload_status: Reloading filter
    php: : Beginning XMLRPC sync to https://xxx.xxx.xxx.xxx:443
    php: : XMLRPC sync successfully completed with https://xxx.xxx.xxx.xxx:443
    php: : Filter sync successfully completed with https://xxx.xxx.xxx.xxx:443

    Once the first sync is completed I force another by saving the settings on carp (without changes).
    The systemlog shows this on the master firewall:
    check_reload_status: Syncing firewall
    check_reload_status: Reloading filter
    php: : A communications error occured while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443
    php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443

  • Hi out there.
    Looks like no one is able to help me further?
    I was wondering if I am the only one with this problem.

  • Hi,

    It will override the rules of the carp slave because you thick/enable the "Syncronize Rules" on your master carp settings.

    Another way to make it work perfectly is you have to fill-up the "Configuration Synchronization Settings (XMLRPC Sync)" IP, Username and Password on your master CARP Settings where the ip/username/password are the IP/Username and password of the slave.

    here's another link that can help you:



  • Thank you for your reply rootlurker.

    It did not solve my problem though.
    If I am not mistaken I have to enable "Synchronize rules" in order to have the backup firewall act exactly the way the master would do regarding my application services in the network. Otherwise changes on the firewall rules on the master wouldn't replicate to my backup (i.e. when i add another NAT/firewall rule).
    What I don't understand is that I got the "SYNC NIC" on both systems (called "CARP" on my configuration) and on BOTH servers I got the rule for it set "Any2Any".
    If the problem is that the sync overrides the setting on the backup with the same state as on the master the rule should stay the same - or at least be replaced with another that has the same values.


  • Yes, you have to enable the "Synchronize Rules" only on the master.

    Make sure also that both server has the same exact NIC's (e.g. 3 NICS on master and 3 NICS on slave for WAN, LAN and SYNC). Don't use 2 NIC's on the SLAVE while 3 NIC's on the master or vice verse because once the master goes down, the LINK between the two server will be drop specially on the LAN side.

    On the SLAVE you need to set the IP on what you have done on the MASTER at Virtual IP.

    eg. on master: (CARP) with vhid 1 = for WAN Interface
               (CARP) with vhid 2  = for LAN Interface

    here's another link for you:


  • bitsync: try to also use http instead of https and see if that makes a difference.

  • Hi.

    Thank you both for your reply.
    I got the same ammount of interfaces on both servers (3 NICs whereas 1 is used as dedicated link for CARP only). Since the machines are not of the same build (one beeing 6 years old and the other only 3 months) the interfaces are a bit different but they are "named" the same in pfSense.
    The failover is working fine though (tested and confirmed  ;)) if all rules are synched.

    And I also tried to switch from HTTPS to HTTP which had no effect. –> Edit: it did have the effect of not having https for the web-gui - which of course was expected -  but the problem still exists

    Maybe I can reformulate my problem a little bit:
    Everything in my configuration with CARP works as I would expect. If the master firewall goes down the backup system jumps in automatically and resumes all tasks until the master comes back.
    To make the whole think work though I always have to login on my backup firewall and create the firewall rule for the carp interface "any2any" to get the next sync working.
    Once I change something on the master and "apply changes" everything is synched and the backup is up2date. However the one rule that is missing is the one for the CARP interface.
    And this causes the next sync to fail.

  • Is there a rule in the master node for the CARP interface for the any-to-any?

  • Hi.

    Looking for the usual suspects are we? ;D
    Yes the rule is there and that's whats bugging me the most:
    every rule gets synched only that one does not. if it were the setting on the backup would still be there after the sync.

  • Yeah. It is always a good place to start. When you created the CARP interface and renamed them, did they both start out as opt1 before renaming them?

  • Hello.

    No. Both interfaces had different names orginially.
    On the master that is "em0" and on the slave it is "fxp0"

  • That is not a problem. I have different NIc types as well. But if you assigned them differently before renaming them, there might be a problem. Like if one was opt2 and the other was opt1, and you renamed them both to CARP, then potentially, I am just guessing, there might be a problem.

    I know clustering works, i have setup it up 5 or more times and they all are still running with no problem.

  • Those interfaces have not been used before in another manor so i could rule out that possibility.

  • Are you using the CARP network to sync settings also? Can you post a sanitized copy of /tmp/rules.debug from the master node?

  • I am not sure what you mean. The settings I use are shown in the screenshots.

  • Forgot this was the second page :). Anyway, looking back at the screen shots it does look like on the master node that CARP was originally opt2 as one of your VPN interface took opt1. I can tell by the ordering of the tabs. This should not make a difference as they are named. Try this, on the master, add a description to the allow all CARP rule (ie CARP Allow All). Sync the settings, and see if that description show up on another interfaces rules.

    Are those other interfaces (RV,OVPNS1, OVPNC1,MGMT) VLANs?

  • Yes. OVPNS1, OVPNC1, MGMT and RV are VLANs

    I tried adding a description to the CARP-interface rule on the master and started the sync. After that my rule on the backup FW is gone (as always) but the rule from the master does not show up on any other interface.

    Edit: I was wrong: the rule does show up (i had it not to replicate via "No XMLRPC Sync" option).
    It appears on the "MGMT" interface

  • Here are 2 screenshots for my interfaces:

    on master i got this:

    an on backup i got this:

  • I even went further now and found out that the rules are synced on the wrong interfaces in several occasions:

    Master -> Backup
    OVPNS1 -> CARP
    CARP -> MGMT
    OVPNC1 -> RV
    -> OVPNS1
    -> OVPNC1

    With all that i am surprised that WAN and LAN aren't synced on the wrong interface as well ;)

    Edit: Looking at the screenshots i believe that the sync does not apply to the interface names but to their creation order.

  • Rebel Alliance Developer Netgate

    That's what I was going to suggest checking.

    The number and order of interfaces in carp cluster members must be the same. What you are seeing is the result of the interfaces not being assigned in the correct order on the slave.

  • Does that mean i have to remove and recreate my interfaces on the backup server to get the correct order?
    OR can i simply update some config file through the console to get the same result?

    The rules and settings for those interfaces should be synced automatically, shouldn't they?

  • Rebel Alliance Developer Netgate

    Yes, unless you want to hand edit the config to swap things around.

  • sometimes hand editing is the easiest. especially if you have to replace a lot of IPs. But in this case, prolly easier to just redo the slave.

  • Do you know which file i need to edit?

  • Rebel Alliance Developer Netgate

    Diagnostics > Backup/Restore, make a backup file, edit the xml backup file, then restore it. If you aren't familiar with XML or can't find your way around it, you're probably better off making the changes in the GUI instead.

  • Thank you.
    I think i can handle the xml and will give it a try.
    (What's the worst that can happen  ;D)

  • It worked.

    Was pretty easy. Just export only the part for interfaces (which results in a very small XML) and change the tags for the interfaces the way the are labeled on the master. Import it again and voilá: chicken's done!

    Thank you all for your help!!!!

    Now: How do I mark this thread as resolved? No more loose ends left :)

  • edit your first post subject with [SOLVED]

Log in to reply