Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta)

w0w

Starting from the 25.07 beta versions, I have noticed a change in the behavior of VIP address and CARP configuration during boot in my setup. Specifically, the firewall almost always boots with only one configured VIP address, while the others remain unconfigured, and of course CARP does not work until the VIPs are reinitialized manually. I don’t remember why, but I initially assumed it was a bug in my system, since it only appeared on one firewall. However, it turns out that’s not the case. Through experimentation, I found that a small fix in rc.bootup solves the issue, but I am sure the root cause lies somewhere else.
Here is the fix I've used.

--- /etc/rc.bootup
+++ /etc/rc.bootup
@@ -434,6 +434,9 @@
 /* Run a filter configure now that most all services have started */
 filter_configure_sync();
 
+/* Check VIPs configuration is complete */
+interfaces_vips_configure("");
+
 /* setup pppoe and pptp */
 vpn_setup();

This patch changes boot behaviour, it ensures CARP VIPs are brought up consistently
before pppoe setup starts.
I can describe my setup a bit:

Every interface is a LAGG, except one which is a VLAN on a physical interface. Multi-WAN setup:

WAN1 (LAGG3) – PPPoE IPv4/IPv6 (/56)
WAN2 (LAGG0) – DHCP IPv4/IPv6 (used only by pfSense itself)
LAN (LAGG2) – static IPv4/IPv6, tracking prefix ID aa on WAN1
WIFILAN (VLAN87 on igc1) – static IPv4/IPv6, tracking prefix ID cc on WAN1
SYNC (LAGG1) – direct connection between firewalls

VIPs:

2 CARP VIPs on LAN (different subnets: subn1, subn2) + 1 ALIAS on LAN (subn2)

1 CARP VIP on WIFILAN (subn3)

1 CARP VIP on WAN2 (subn4)

Does anyone have any thoughts or ideas about why this might be happening and why rc.bootup change helps?

stephenw10

Ok you only see one VIP per interface? Or only one VIP total?

And only CARP VIPs?

w0w

@stephenw10
Usually, only one interface shows the configured VIPs—WAN2. If I disable WAN2, then another interface shows VIPs configured, but only one interface at a time.
Small correction to my original post: CARP does function for the interface on which the CARP VIP is up, provided there is one.

stephenw10

Hmm, well I've definitely not seen that. HA setups here come up fine.

The only thing unusual you have there seems to be having two CARP VIPs on one interface. But testing that here it also works fine.

Is this the only setup you're seeing this on?

I'll try to test something with PPPoE. However that's not a supported config in HA so.....

w0w

@stephenw10 said in Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta):

However that's not a supported config in HA so.....

Thank you for your help!

pfSense book states

CARP cannot be used on interfaces that use PPPoE, PPTP, or other tunnel-like connections.
These interfaces do not have an address that can be shared or synchronized between nodes.
In these cases, CARP must be configured only on internal interfaces.

I don't use CARP on PPPoE interface as you can see, so is it supported config?

stephenw10

Ah, good point! It probably still changes the load sequence at boot though. Let me test.

Do you have two separate PPPoE links, one for each node then?

w0w

@stephenw10 said in Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta):

Do you have two separate PPPoE links, one for each node then?

Yes and no. Both nodes are configured for PPPoE with identical settings. I can bring up two PPPoE sessions to my ISP at the same time, but they likely don’t permit it, so I use a script that detects which HA node is Master and starts/stops the PPPoE session accordingly. The script has no impact during boot and includes a safety startup delay. I’ve tested with the script fully disabled — the behavior remains unchanged.

@stephenw10 said in Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta):

Let me test.

I tested this about two weeks ago, but I don’t remember the exact results because the run was interrupted by a continious fatal trap as you remember.
I plan to re-run some tests—I don’t clearly remember the exact steps I took. There may be a link to the new PPPoE kernel module, but that’s just a guess.

stephenw10

Ah, good point. Have you tested with and without the new if_pppoe module?