@mrfrenchfry
You can export the interface config from the secondary node:
Diagnostics > Backup & Restore > Backup & Restore
At Backup area select "Interfaces".

Download the file. Then load it into a text editor and order the interfaces accordingly to the primary.

Save the file and re-import it into the secondary.

More photos:

20220201_181442.jpg
20220131_180718.jpg
20220201_181457.jpg
20220119_165632.jpg
20210929_162052.jpg 20201214_141056_HDR.jpg

Sorry, it probably was only a temporary problem while the network reconfigured to the static IP.
It now seems to work properly.

@skorpio The CARP alias skew is set in each alias: https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html#configuring-the-carp-virtual-ips

"A primary node is typically set to 0 or 1, secondary nodes will be 100 or higher. This adjustment is handled automatically by XML-RPC synchronization."

@crl was this resolved? I'm having some issues myself.

Hoping you found your solution. :)

@viragomann switching to CARP VIP in the OpenVPN config solved the issue, now I'm getting to the LAN. Thank you very much for pointing me on the right direction!

Ok, found out the problem, maybe.

there was a space in the Gateway IP in VAN50 dhcp settings also checked "Time from UTC to Local"

With both changes, now DHCP works flawlessly.

Thank you.

@steveits
Hello,
When I add the rule on the master, it is duplicated on the second pfsense. The master pfsense remains master in "status/CARP".
The second pfsense is in "Backup".
Yes the problem is not temporary, I have a total loss to the internet.
The master's wan interfaces are up and communicating with their gateway. But unable to access the internet.
The interfaces are in green on the dashboard.

Regards.

Is any one knows why the URL is getting changed by adding amp; every time while saving the configuration? this is kind of miss-behavior

@klaws Mail Report just let you know in time any issues that could occur.

At least for me it helps a lot dealing with pfsense clusters.

Examples:

like when some CARP state changes states (master or backup),

17:51:09 HA cluster member "(10.0.13.1@ixl3.13): (IXL3_VLAN13_IT_ADMINS)" has resumed CARP state "BACKUP" for vhid 12

when WANs went offline or online in gateway groups:

11:07:07 MONITOR: WAN_ROUTERA_WAN2_GW is available now, adding to routing group GW_GROUP x.x.x.225|172.16.2.2|WAN_ROUTERA_WAN2_GW|34.651ms|87.308ms|18%|online|loss

when services stop working and watchdog service detect and handle the situation,

9:26:00 Service Watchdog detected service openvpn stopped. Restarting openvpn (OpenVPN server: Internal Devices)

when rules cannot load:

15:42:40 There were error(s) loading the rules: /tmp/rules.debug:51: cannot load "/var/db/aliastables/pfB_NAmerica_v6.txt": Invalid argument - The line in question reads [51]: table <pfB_NAmerica_v6> persist file "/var/db/aliastables/pfB_NAmerica_v6.txt"

when XMLRPC communication fails:

17:29:59 A communications error occurred while attempting to call XMLRPC method restore_config_section: 16:43:28 Exception calling XMLRPC method restore_config_section # Impossible to encode value '' from type 'NULL'. No analogous type in XML_RPC.

@steveits It got solved. Rebooting the firewall did not help, resetting the states neither, but disabling the WAN interface and enabling it again DID help.
And all is looking good again now

049f21fe-c7d0-4640-8834-d5f7af093f0a-image.png

@viragomann

The one thing I notice, examining config.xml: the internal ID for a gateway group is pretty unique.

No idea how that is supposed to sync or not...

I'm going to do more experiments tomorrow...

@mrpete @viragomann
I've got it working close to 100% now :)

@mrpete @netblues @Cool_Corona

I've updated the OP with results of my first set of experiments.

When I have a chance, I'll redo a full install on secondary CARP and see how that goes.

@netblues
That page does not say that... But it does link to the a page hinting at this:
https://docs.netgate.com/pfsense/en/latest/highavailability/index.html#switch-layer-2-concerns

While "CARP VIPs each have their own unique MAC address derived from their VHID" "At minimum, the switch must... Allow the CARP VIP MAC address to move between ports."

Thanks! I think I am beginning to understand this... 😏

@netblues duuuh. Thanks. That s embarrassing. I completely missed that pkg on my list to be manually installed.

Thanks! 🤠

Never mind, a reboot solved it.
-nic

@viragomann said in Multiple VLANs in HA config:

So ensure the VLAN is also properly configured on the switch.

omg , so stupid :)

Thx it all works now

@jnpetty
When you ping the CARP VIP from a connected device, it will first send an ARP request which the master should respond to.

So to investigate, sniff the traffic and check for ARP packets and if pfSense sends a respond.
If there is no ARP request, check the ARP table on the device you're pinging from for an already existing entry.

@tboston What's the mismatch?

It is also possible to save/back up the config, edit the XML file, and do a restore. But it should be possible to edit/assign interfaces in the web GUI.

sorry both hosts are 6.5u3

@mauro-tridici said in Question about multiple WAN CARP VIPs:

So, if I understand your messages correctly, I can add additional public virtual IPs as "IP alias" on top of existing CARP VIP

Yes. At interface select the CARP VIP from the drop-down.

even if is the usage of HAproxy is a better solution

What does this mean?

Could you please confirm that the assignment of multiple WAN IPs addresses (x.x.x.1,x.x.x.2,x.x.x.3,x.x.x.4,x.x.x.5,x.x.x.6) belonging to the same subnet will be not a problem?

All right.

Refer to the docs: Virtual IP Address Feature Comparison

Remember that you have to configure the outbound NAT manually to use the CARP VIP instead of the primary interface IP.

@misterto

WAN 1:

WAN Subnet: 161.12.60.232/29
ISP Gateway: 161.12.60.233
Routed Subnet: 161.12.51.32/29
Shared CARP VIP: 161.12.60.236

WAN 2:

WAN Subnet: 161.12.60.240/29
ISP Gateway: 161.12.60.241
Routed Subnet: 161.12.51.40/29
Shared CARP VIP: 161.12.60.244

Firewall 1:

WAN 1 Interface: 161.12.60.234
WAN 2 Interface: 161.12.60.242

Firewall 2:

WAN 1 Interface: 161.12.60.235
WAN 2 Interface: 161.12.60.243

I am having the same problem.

As my primary HA member is the one on 21.7 and the secondary is on 21.8, I tried to download the configuration xml and change from 21.8 to 21.7 and then restore the configuration. This didn't work.

I can try and switch the primary from 21.7 to 21.8 (after setting it to backup with carp) but will have to do it outside operating hours, as it scares me pretty good.

Does anyone else have any feedback on what might be causing this? This seems a pretty significant issue that 21.8 isn't even an acknowledged version and my primary system has no knowledge of there being an update.

@marama said in HA + VIP + MultiWAN Issue (no internet on slave):

@keyser ok, will do.
I'm a bit afraid of removing the "any", since I need to be sure to include all the relevant networks in the alias. Do I also have to include the ipsec and openvpn networks, translations/mapping networks... ?
Is there a way to leave "any", but then have explicit NAT rule handle the firewall traffic?

Yes, you need to have vpn networks and such in the alias as Well.
I normally always make an alias called private networks i use for stuff like that.
It contains:

192.168.0.0/16
172.16.0.0/12
10.0.0.0/8

That way any private (internal thing - including future uses) is covered - But not the FW and its public addresses.
Btw - that same alias is Very good in internet access allow rules instead of ANY. Use it as destination with the NOT (!) feature.

@kd Ah, I think using your own modem it is intended as a passthrough. Around here, business Comcast accounts provide the 10.1.10.x subnet and NAT. Useful for plugging in a laptop to bypass the client's router, to test.

My home modem is accessible at 192.168.100.1 but I don't think it provides NAT out. It doesn't have a "bridge mode" setting as it just passes the public IP through to my router or a laptop.

@Foxi352

Thank you for the links. I see, it is a difficult issue. I wish they had the CARP as option to take the functionality of a virtual interface (with DHCP/MAC, etc) , rather than just IP. So this can be shared between the firewalls.

It seems i have few issues:

DHCP - perhaps setting WAN as static IP would work untill the next lease, need to check MAC spoofing.. Not sure how to handle this for the two interfaces. Folks seem to use some scripts to have the interface UP and down

Perhaps I will manually plug the WAN cable when needed , if i can not find a workaround :).

thanks for your time

@wesleywillis said in VIP setup for web hosting:

I confirmed that I only get that block as described under 'Simple IP Subnet on WAN':

Yes, in this case you'd probably better go with Proxy ARP, so you can cover the whole subnet with a single VIP assignment.
It is a good way, when you want to forward the whole subnet behind pfSense.

So I'm assuming it's easiest to just setup NAT 1:1 as such:
External IP: 1.1.1.3/26
Internal IP: 10.0.10.3/26

Possibly you may have to state the network address here, when using network type.