@m4rek11 I am looking into logs I see that during applying patch there are some entries, but after patching I see only a few, and they all looks as they should (at least for me) and they have reason (eg. rebooted node). I wouldn't call them storm and definitely I don't see flipping MASTER - BACKUP entries now.

@digger30 Perfect! Glad I could be of assistance.

The maintenance mode switch is in the config and persists across reboots.

@dara said in pfSense CARP + Cisco N5k vPC:

@philippe-richard Hi Philippe, Thanks a lot. This is more complete and interesting than our setup.

I wonder how you configured the connection between the routers and switches?

In my setup, each router has a single connection to a single switch configured as an Orphan port. For now it is working perfectly.

I am not sure however how it will handle different link and device failure scenarios but I will test it sometime soon and post my findings here.

Hello, the plan below reflects our current architecture fairly well.
For the moment the leased lines and the fiber modems are in a single building but in the fairly near future, we will have the same thing in the second building.
I did not indicate it on the plan but nexus 1 and 2 are interconnected with the other switches of the other buildings.
We are working on the consolidation of the Nexus infra because we have HA problems with the Netgate (probably a layer 2 and multicast problem with the Vpc but, not sure)
inventaire_nexus_netgate_forum.png

@chrullrich I replaced the pfSense 2.6 "local router/firewall"s in my test setup with OPNsense 22.1 (this is FreeBSD 13.0 instead of pfSense 2.6's 12.3) to get a second opinion. The behavior is the same: As soon as the CARP failover happens, everything sent towards the "Internet" goes out the default route with the NATed source address appropriate for the policy route.

When I tried it the first time today I thought I saw ping (and only ping) work correctly, but now I cannot reproduce it. I probably just saw what I wanted to see.

@derelict
Yeah, same conclusion i had.

@viragomann
Yup.

@viragomann

Issue resolved with hostname override and haproxy listnening on LAN interface

Thx

@skid9000 Perhaps some screenshots of the setup? Can you get it working without the VLANs and add those in after? I've not had occasion to set HA up with VLANs but have done so with aliases for other subnets on LAN.

Just for your info. We've now seen the issue on multiple installations (even different hardware and pfsense versions) and could solve it on every single system by moving the sync-vlan to a dedicated physical interface.

@viragomann
i think it's that
https://forum.netgate.com/topic/150505/xmlrpc-restore_config_section-error

because my rule to NAT with CARP ip make the backup node not able to reach the gateway
so as it explain on that like you sent

Filter reload sees the down gateway and resets states, terminating the connection currently used for XMLRPC.

it make sense
Thanks you very much, i think you resolve my issue :)

@kayavila OK this is great info, thank you! I read your entire write up you linked to as well but I'm still trying to wrap my brain around it. Think I've got it figured out but wanted to pose an example.

This particular one will be between different VLAN/subnets rather than with WAN as I personally don't ever allow those connections via the WAN.

So in theory if you had VLAN1 and VLAN2 setup, and there was an any-any rule below a block "This Firewall" rule on VLAN1, and some device on VLAN1 tried to contact the LAN interface of VLAN2, due to state syncing this would be let through? Since the first node would see the connection to the VLAN2 IP and see that it's not in it's block list but matches the any-any rule, and then the state would sync to the secondary which wouldn't assess it's rules?

If that is the case, I would imagine not having a rule on the primary node that allows access to any would solve the issue, but since some people do use an any rule for internet access it could pose a problem (though best practice is of course to use an alias for RFC1918 and explicitly allow the inverse of that).

@viragomann
Thanks !
Went with the port forward + outbound option, NAT is working finally.

@starsandbars What questions do you have after reading this?

https://docs.netgate.com/pfsense/en/latest/highavailability/index.html

@mrfrenchfry
You can export the interface config from the secondary node:
Diagnostics > Backup & Restore > Backup & Restore
At Backup area select "Interfaces".

Download the file. Then load it into a text editor and order the interfaces accordingly to the primary.

Save the file and re-import it into the secondary.

More photos:

20220201_181442.jpg
20220131_180718.jpg
20220201_181457.jpg
20220119_165632.jpg
20210929_162052.jpg 20201214_141056_HDR.jpg

Sorry, it probably was only a temporary problem while the network reconfigured to the static IP.
It now seems to work properly.

@skorpio The CARP alias skew is set in each alias: https://docs.netgate.com/pfsense/en/latest/recipes/high-availability.html#configuring-the-carp-virtual-ips

"A primary node is typically set to 0 or 1, secondary nodes will be 100 or higher. This adjustment is handled automatically by XML-RPC synchronization."

@notorious_vr Good.

In any case, my go-to tool for speed stuff is Wireshark (and tcpdump inside of pfSense, saved to a file to examine w/ Wireshark)

I can see:

Packet timing All kinds of packet issues Smart analysis of entire streams Etc.

@crl was this resolved? I'm having some issues myself.

Hoping you found your solution. :)

@viragomann switching to CARP VIP in the OpenVPN config solved the issue, now I'm getting to the LAN. Thank you very much for pointing me on the right direction!

Ok, found out the problem, maybe.

there was a space in the Gateway IP in VAN50 dhcp settings also checked "Time from UTC to Local"

With both changes, now DHCP works flawlessly.

Thank you.

@steveits
Hello,
When I add the rule on the master, it is duplicated on the second pfsense. The master pfsense remains master in "status/CARP".
The second pfsense is in "Backup".
Yes the problem is not temporary, I have a total loss to the internet.
The master's wan interfaces are up and communicating with their gateway. But unable to access the internet.
The interfaces are in green on the dashboard.

Regards.

Is any one knows why the URL is getting changed by adding amp; every time while saving the configuration? this is kind of miss-behavior

@klaws Mail Report just let you know in time any issues that could occur.

At least for me it helps a lot dealing with pfsense clusters.

Examples:

like when some CARP state changes states (master or backup),

17:51:09 HA cluster member "(10.0.13.1@ixl3.13): (IXL3_VLAN13_IT_ADMINS)" has resumed CARP state "BACKUP" for vhid 12

when WANs went offline or online in gateway groups:

11:07:07 MONITOR: WAN_ROUTERA_WAN2_GW is available now, adding to routing group GW_GROUP x.x.x.225|172.16.2.2|WAN_ROUTERA_WAN2_GW|34.651ms|87.308ms|18%|online|loss

when services stop working and watchdog service detect and handle the situation,

9:26:00 Service Watchdog detected service openvpn stopped. Restarting openvpn (OpenVPN server: Internal Devices)

when rules cannot load:

15:42:40 There were error(s) loading the rules: /tmp/rules.debug:51: cannot load "/var/db/aliastables/pfB_NAmerica_v6.txt": Invalid argument - The line in question reads [51]: table <pfB_NAmerica_v6> persist file "/var/db/aliastables/pfB_NAmerica_v6.txt"

when XMLRPC communication fails:

17:29:59 A communications error occurred while attempting to call XMLRPC method restore_config_section: 16:43:28 Exception calling XMLRPC method restore_config_section # Impossible to encode value '' from type 'NULL'. No analogous type in XML_RPC.

@steveits It got solved. Rebooting the firewall did not help, resetting the states neither, but disabling the WAN interface and enabling it again DID help.
And all is looking good again now

049f21fe-c7d0-4640-8834-d5f7af093f0a-image.png

@viragomann

The one thing I notice, examining config.xml: the internal ID for a gateway group is pretty unique.

No idea how that is supposed to sync or not...

I'm going to do more experiments tomorrow...