[SOLVED] Firewall rule on CARP interface keeps being deleted after sync

bitadmin

I have 2 firewalls set up with pfsense 2.0 (RC3) which are supposed to sync between each other. Unfortunately after the sync the firewall rule to ensure that communication between the 2 systems can pass through is removed on the backup server.
Since the release is now available I upgraded via the hot update feature which was fine but the problem persists.

I found one thread in here of someone who describes a similar behaviour in 2009 (or so) but he never got any reply.

Hope someone can help me out on this, because it's quite unnerving to have to implement the rule everytime i make a change on the master firewall.

podilarius

Can you post screen shots on the master node of the rules and then before and after on the backup node? Are you using a separate interface for pfsync?

bitadmin

Thanks for your quick reply.

Since the hardware is not the same on my 2 firewalls its rather difficult to use the same interface for CARP - however i believe it should be sufficent to have them both named the same, shouldn't it?

Here's how i set up the firewall rules on both systems:
Master just before i sync:

Backup before i sync:

Backup after sync:

Master after sync:

As you can see the rule on the master looks fine but on the backup FW it is removed right after the sync (however the changes are synched correctly as long as the rule is available).

podilarius

Can you post a screen shot of your CARP settings?

bitadmin

Sure:

For obvious reasons I blanked out the username name used for the sync.

BTW: I allready tried the option "No XMLRPC Sync" on the CARP rule (on the master FW) without success.

podilarius

Are there any errors in either of the system logs when you sync the config?

bitadmin

Sorry for the late reply.
Nothing eye-catching. Only what you would expect since the sync does not work.

On the first sync i get things like this on the backup firewall:
…
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: : Config sync not being done because of missing sync IP (normal on secondary systems).
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: /xmlrpc.php: ROUTING: setting default route to xxx.xxx.xxx.xxx
apinger: Exiting on signal 15.
apinger: Starting Alarm Pinger, apinger(54067)
php: /xmlrpc.php: Resyncing OpenVPN instances.
...

and on the master firewall i get:
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: : Beginning XMLRPC sync to https://xxx.xxx.xxx.xxx:443
php: : XMLRPC sync successfully completed with https://xxx.xxx.xxx.xxx:443
php: : Filter sync successfully completed with https://xxx.xxx.xxx.xxx:443

Once the first sync is completed I force another by saving the settings on carp (without changes).
The systemlog shows this on the master firewall:
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: : A communications error occured while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443
php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443

bitadmin

Hi out there.
Looks like no one is able to help me further?
I was wondering if I am the only one with this problem.

rootlurker

Hi,

It will override the rules of the carp slave because you thick/enable the "Syncronize Rules" on your master carp settings.

Another way to make it work perfectly is you have to fill-up the "Configuration Synchronization Settings (XMLRPC Sync)" IP, Username and Password on your master CARP Settings where the ip/username/password are the IP/Username and password of the slave.

here's another link that can help you:

http://www.howtoforge.com/how-to-configure-a-pfsense-2.0-cluster-using-carp

Thanks

bitadmin

Thank you for your reply rootlurker.

It did not solve my problem though.
If I am not mistaken I have to enable "Synchronize rules" in order to have the backup firewall act exactly the way the master would do regarding my application services in the network. Otherwise changes on the firewall rules on the master wouldn't replicate to my backup (i.e. when i add another NAT/firewall rule).
What I don't understand is that I got the "SYNC NIC" on both systems (called "CARP" on my configuration) and on BOTH servers I got the rule for it set "Any2Any".
If the problem is that the sync overrides the setting on the backup with the same state as on the master the rule should stay the same - or at least be replaced with another that has the same values.

BR

rootlurker

Yes, you have to enable the "Synchronize Rules" only on the master.

Make sure also that both server has the same exact NIC's (e.g. 3 NICS on master and 3 NICS on slave for WAN, LAN and SYNC). Don't use 2 NIC's on the SLAVE while 3 NIC's on the master or vice verse because once the master goes down, the LINK between the two server will be drop specially on the LAN side.

On the SLAVE you need to set the IP on what you have done on the MASTER at Virtual IP.

eg. on master: 192.168.100.100/24 (CARP) with vhid 1 = for WAN Interface
192.168.1.100/24 (CARP) with vhid 2 = for LAN Interface

here's another link for you:

http://www.pfsense.org/mirror.php?section=tutorials/carp/carp-cluster-new.htm

podilarius

bitsync: try to also use http instead of https and see if that makes a difference.

bitadmin

Hi.

Thank you both for your reply.
I got the same ammount of interfaces on both servers (3 NICs whereas 1 is used as dedicated link for CARP only). Since the machines are not of the same build (one beeing 6 years old and the other only 3 months) the interfaces are a bit different but they are "named" the same in pfSense.
The failover is working fine though (tested and confirmed ;)) if all rules are synched.

And I also tried to switch from HTTPS to HTTP which had no effect. –> Edit: it did have the effect of not having https for the web-gui - which of course was expected - but the problem still exists

Maybe I can reformulate my problem a little bit:
Everything in my configuration with CARP works as I would expect. If the master firewall goes down the backup system jumps in automatically and resumes all tasks until the master comes back.
To make the whole think work though I always have to login on my backup firewall and create the firewall rule for the carp interface "any2any" to get the next sync working.
Once I change something on the master and "apply changes" everything is synched and the backup is up2date. However the one rule that is missing is the one for the CARP interface.
And this causes the next sync to fail.

podilarius

Is there a rule in the master node for the CARP interface for the any-to-any?

bitadmin

Hi.

Looking for the usual suspects are we? ;D
Yes the rule is there and that's whats bugging me the most:
every rule gets synched only that one does not. if it were the setting on the backup would still be there after the sync.

podilarius

Yeah. It is always a good place to start. When you created the CARP interface and renamed them, did they both start out as opt1 before renaming them?

bitadmin

Hello.

No. Both interfaces had different names orginially.
On the master that is "em0" and on the slave it is "fxp0"

podilarius

That is not a problem. I have different NIc types as well. But if you assigned them differently before renaming them, there might be a problem. Like if one was opt2 and the other was opt1, and you renamed them both to CARP, then potentially, I am just guessing, there might be a problem.

I know clustering works, i have setup it up 5 or more times and they all are still running with no problem.

bitadmin

Those interfaces have not been used before in another manor so i could rule out that possibility.

podilarius

Are you using the CARP network to sync settings also? Can you post a sanitized copy of /tmp/rules.debug from the master node?