[SOLVED] Firewall rule on CARP interface keeps being deleted after sync

bitadmin

Hi.

Thank you both for your reply.
I got the same ammount of interfaces on both servers (3 NICs whereas 1 is used as dedicated link for CARP only). Since the machines are not of the same build (one beeing 6 years old and the other only 3 months) the interfaces are a bit different but they are "named" the same in pfSense.
The failover is working fine though (tested and confirmed ;)) if all rules are synched.

And I also tried to switch from HTTPS to HTTP which had no effect. –> Edit: it did have the effect of not having https for the web-gui - which of course was expected - but the problem still exists

Maybe I can reformulate my problem a little bit:
Everything in my configuration with CARP works as I would expect. If the master firewall goes down the backup system jumps in automatically and resumes all tasks until the master comes back.
To make the whole think work though I always have to login on my backup firewall and create the firewall rule for the carp interface "any2any" to get the next sync working.
Once I change something on the master and "apply changes" everything is synched and the backup is up2date. However the one rule that is missing is the one for the CARP interface.
And this causes the next sync to fail.

podilarius

Is there a rule in the master node for the CARP interface for the any-to-any?

bitadmin

Hi.

Looking for the usual suspects are we? ;D
Yes the rule is there and that's whats bugging me the most:
every rule gets synched only that one does not. if it were the setting on the backup would still be there after the sync.

podilarius

Yeah. It is always a good place to start. When you created the CARP interface and renamed them, did they both start out as opt1 before renaming them?

bitadmin

Hello.

No. Both interfaces had different names orginially.
On the master that is "em0" and on the slave it is "fxp0"

podilarius

That is not a problem. I have different NIc types as well. But if you assigned them differently before renaming them, there might be a problem. Like if one was opt2 and the other was opt1, and you renamed them both to CARP, then potentially, I am just guessing, there might be a problem.

I know clustering works, i have setup it up 5 or more times and they all are still running with no problem.

bitadmin

Those interfaces have not been used before in another manor so i could rule out that possibility.

podilarius

Are you using the CARP network to sync settings also? Can you post a sanitized copy of /tmp/rules.debug from the master node?

bitadmin

I am not sure what you mean. The settings I use are shown in the screenshots.

podilarius

Forgot this was the second page :). Anyway, looking back at the screen shots it does look like on the master node that CARP was originally opt2 as one of your VPN interface took opt1. I can tell by the ordering of the tabs. This should not make a difference as they are named. Try this, on the master, add a description to the allow all CARP rule (ie CARP Allow All). Sync the settings, and see if that description show up on another interfaces rules.

Are those other interfaces (RV,OVPNS1, OVPNC1,MGMT) VLANs?

bitadmin

Yes. OVPNS1, OVPNC1, MGMT and RV are VLANs

I tried adding a description to the CARP-interface rule on the master and started the sync. After that my rule on the backup FW is gone (as always) but the rule from the master does not show up on any other interface.

Edit: I was wrong: the rule does show up (i had it not to replicate via "No XMLRPC Sync" option).
It appears on the "MGMT" interface

bitadmin

Here are 2 screenshots for my interfaces:

on master i got this:

an on backup i got this:

bitadmin

I even went further now and found out that the rules are synced on the wrong interfaces in several occasions:

Master -> Backup
OVPNS1 -> CARP
CARP -> MGMT
OVPNC1 -> RV
-> OVPNS1
-> OVPNC1

With all that i am surprised that WAN and LAN aren't synced on the wrong interface as well ;)

Edit: Looking at the screenshots i believe that the sync does not apply to the interface names but to their creation order.

jimp

That's what I was going to suggest checking.

The number and order of interfaces in carp cluster members must be the same. What you are seeing is the result of the interfaces not being assigned in the correct order on the slave.

bitadmin

Does that mean i have to remove and recreate my interfaces on the backup server to get the correct order?
OR can i simply update some config file through the console to get the same result?

The rules and settings for those interfaces should be synced automatically, shouldn't they?

jimp

Yes, unless you want to hand edit the config to swap things around.

podilarius

sometimes hand editing is the easiest. especially if you have to replace a lot of IPs. But in this case, prolly easier to just redo the slave.

bitadmin

Do you know which file i need to edit?

jimp

Diagnostics > Backup/Restore, make a backup file, edit the xml backup file, then restore it. If you aren't familiar with XML or can't find your way around it, you're probably better off making the changes in the GUI instead.

bitadmin

Thank you.
I think i can handle the xml and will give it a try.
(What's the worst that can happen ;D)