[SOLVED] Firewall rule on CARP interface keeps being deleted after sync

bitadmin

Sorry for the late reply.
Nothing eye-catching. Only what you would expect since the sync does not work.

On the first sync i get things like this on the backup firewall:
…
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: : Config sync not being done because of missing sync IP (normal on secondary systems).
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: /xmlrpc.php: ROUTING: setting default route to xxx.xxx.xxx.xxx
apinger: Exiting on signal 15.
apinger: Starting Alarm Pinger, apinger(54067)
php: /xmlrpc.php: Resyncing OpenVPN instances.
...

and on the master firewall i get:
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: : Beginning XMLRPC sync to https://xxx.xxx.xxx.xxx:443
php: : XMLRPC sync successfully completed with https://xxx.xxx.xxx.xxx:443
php: : Filter sync successfully completed with https://xxx.xxx.xxx.xxx:443

Once the first sync is completed I force another by saving the settings on carp (without changes).
The systemlog shows this on the master firewall:
check_reload_status: Syncing firewall
check_reload_status: Reloading filter
php: : A communications error occured while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443
php: : New alert found: A communications error occured while attempting XMLRPC sync with username admin https://xxx.xxx.xxx.xxx:443

bitadmin

Hi out there.
Looks like no one is able to help me further?
I was wondering if I am the only one with this problem.

rootlurker

Hi,

It will override the rules of the carp slave because you thick/enable the "Syncronize Rules" on your master carp settings.

Another way to make it work perfectly is you have to fill-up the "Configuration Synchronization Settings (XMLRPC Sync)" IP, Username and Password on your master CARP Settings where the ip/username/password are the IP/Username and password of the slave.

here's another link that can help you:

http://www.howtoforge.com/how-to-configure-a-pfsense-2.0-cluster-using-carp

Thanks

bitadmin

Thank you for your reply rootlurker.

It did not solve my problem though.
If I am not mistaken I have to enable "Synchronize rules" in order to have the backup firewall act exactly the way the master would do regarding my application services in the network. Otherwise changes on the firewall rules on the master wouldn't replicate to my backup (i.e. when i add another NAT/firewall rule).
What I don't understand is that I got the "SYNC NIC" on both systems (called "CARP" on my configuration) and on BOTH servers I got the rule for it set "Any2Any".
If the problem is that the sync overrides the setting on the backup with the same state as on the master the rule should stay the same - or at least be replaced with another that has the same values.

BR

rootlurker

Yes, you have to enable the "Synchronize Rules" only on the master.

Make sure also that both server has the same exact NIC's (e.g. 3 NICS on master and 3 NICS on slave for WAN, LAN and SYNC). Don't use 2 NIC's on the SLAVE while 3 NIC's on the master or vice verse because once the master goes down, the LINK between the two server will be drop specially on the LAN side.

On the SLAVE you need to set the IP on what you have done on the MASTER at Virtual IP.

eg. on master: 192.168.100.100/24 (CARP) with vhid 1 = for WAN Interface
192.168.1.100/24 (CARP) with vhid 2 = for LAN Interface

here's another link for you:

http://www.pfsense.org/mirror.php?section=tutorials/carp/carp-cluster-new.htm

podilarius

bitsync: try to also use http instead of https and see if that makes a difference.

bitadmin

Hi.

Thank you both for your reply.
I got the same ammount of interfaces on both servers (3 NICs whereas 1 is used as dedicated link for CARP only). Since the machines are not of the same build (one beeing 6 years old and the other only 3 months) the interfaces are a bit different but they are "named" the same in pfSense.
The failover is working fine though (tested and confirmed ;)) if all rules are synched.

And I also tried to switch from HTTPS to HTTP which had no effect. –> Edit: it did have the effect of not having https for the web-gui - which of course was expected - but the problem still exists

Maybe I can reformulate my problem a little bit:
Everything in my configuration with CARP works as I would expect. If the master firewall goes down the backup system jumps in automatically and resumes all tasks until the master comes back.
To make the whole think work though I always have to login on my backup firewall and create the firewall rule for the carp interface "any2any" to get the next sync working.
Once I change something on the master and "apply changes" everything is synched and the backup is up2date. However the one rule that is missing is the one for the CARP interface.
And this causes the next sync to fail.

podilarius

Is there a rule in the master node for the CARP interface for the any-to-any?

bitadmin

Hi.

Looking for the usual suspects are we? ;D
Yes the rule is there and that's whats bugging me the most:
every rule gets synched only that one does not. if it were the setting on the backup would still be there after the sync.

podilarius

Yeah. It is always a good place to start. When you created the CARP interface and renamed them, did they both start out as opt1 before renaming them?

bitadmin

Hello.

No. Both interfaces had different names orginially.
On the master that is "em0" and on the slave it is "fxp0"

podilarius

That is not a problem. I have different NIc types as well. But if you assigned them differently before renaming them, there might be a problem. Like if one was opt2 and the other was opt1, and you renamed them both to CARP, then potentially, I am just guessing, there might be a problem.

I know clustering works, i have setup it up 5 or more times and they all are still running with no problem.

bitadmin

Those interfaces have not been used before in another manor so i could rule out that possibility.

podilarius

Are you using the CARP network to sync settings also? Can you post a sanitized copy of /tmp/rules.debug from the master node?

bitadmin

I am not sure what you mean. The settings I use are shown in the screenshots.

podilarius

Forgot this was the second page :). Anyway, looking back at the screen shots it does look like on the master node that CARP was originally opt2 as one of your VPN interface took opt1. I can tell by the ordering of the tabs. This should not make a difference as they are named. Try this, on the master, add a description to the allow all CARP rule (ie CARP Allow All). Sync the settings, and see if that description show up on another interfaces rules.

Are those other interfaces (RV,OVPNS1, OVPNC1,MGMT) VLANs?

bitadmin

Yes. OVPNS1, OVPNC1, MGMT and RV are VLANs

I tried adding a description to the CARP-interface rule on the master and started the sync. After that my rule on the backup FW is gone (as always) but the rule from the master does not show up on any other interface.

Edit: I was wrong: the rule does show up (i had it not to replicate via "No XMLRPC Sync" option).
It appears on the "MGMT" interface

bitadmin

Here are 2 screenshots for my interfaces:

on master i got this:

an on backup i got this:

bitadmin

I even went further now and found out that the rules are synced on the wrong interfaces in several occasions:

Master -> Backup
OVPNS1 -> CARP
CARP -> MGMT
OVPNC1 -> RV
-> OVPNS1
-> OVPNC1

With all that i am surprised that WAN and LAN aren't synced on the wrong interface as well ;)

Edit: Looking at the screenshots i believe that the sync does not apply to the interface names but to their creation order.

jimp

That's what I was going to suggest checking.

The number and order of interfaces in carp cluster members must be the same. What you are seeing is the result of the interfaces not being assigned in the correct order on the slave.