Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta)

w0w

Starting from the 25.07 beta versions, I have noticed a change in the behavior of VIP address and CARP configuration during boot in my setup. Specifically, the firewall almost always boots with only one configured VIP address, while the others remain unconfigured, and of course CARP does not work until the VIPs are reinitialized manually. I don’t remember why, but I initially assumed it was a bug in my system, since it only appeared on one firewall. However, it turns out that’s not the case. Through experimentation, I found that a small fix in rc.bootup solves the issue, but I am sure the root cause lies somewhere else.
Here is the fix I've used.

--- /etc/rc.bootup
+++ /etc/rc.bootup
@@ -434,6 +434,9 @@
 /* Run a filter configure now that most all services have started */
 filter_configure_sync();
 
+/* Check VIPs configuration is complete */
+interfaces_vips_configure("");
+
 /* setup pppoe and pptp */
 vpn_setup();

This patch changes boot behaviour, it ensures CARP VIPs are brought up consistently
before pppoe setup starts.
I can describe my setup a bit:

Every interface is a LAGG, except one which is a VLAN on a physical interface. Multi-WAN setup:

WAN1 (LAGG3) – PPPoE IPv4/IPv6 (/56)
WAN2 (LAGG0) – DHCP IPv4/IPv6 (used only by pfSense itself)
LAN (LAGG2) – static IPv4/IPv6, tracking prefix ID aa on WAN1
WIFILAN (VLAN87 on igc1) – static IPv4/IPv6, tracking prefix ID cc on WAN1
SYNC (LAGG1) – direct connection between firewalls

VIPs:

2 CARP VIPs on LAN (different subnets: subn1, subn2) + 1 ALIAS on LAN (subn2)

1 CARP VIP on WIFILAN (subn3)

1 CARP VIP on WAN2 (subn4)

Does anyone have any thoughts or ideas about why this might be happening and why rc.bootup change helps?

stephenw10

Ok you only see one VIP per interface? Or only one VIP total?

And only CARP VIPs?

w0w

@stephenw10
Usually, only one interface shows the configured VIPs—WAN2. If I disable WAN2, then another interface shows VIPs configured, but only one interface at a time.
Small correction to my original post: CARP does function for the interface on which the CARP VIP is up, provided there is one.

stephenw10

Hmm, well I've definitely not seen that. HA setups here come up fine.

The only thing unusual you have there seems to be having two CARP VIPs on one interface. But testing that here it also works fine.

Is this the only setup you're seeing this on?

I'll try to test something with PPPoE. However that's not a supported config in HA so.....

w0w

@stephenw10 said in Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta):

However that's not a supported config in HA so.....

Thank you for your help!

pfSense book states

CARP cannot be used on interfaces that use PPPoE, PPTP, or other tunnel-like connections.
These interfaces do not have an address that can be shared or synchronized between nodes.
In these cases, CARP must be configured only on internal interfaces.

I don't use CARP on PPPoE interface as you can see, so is it supported config?

stephenw10

Ah, good point! It probably still changes the load sequence at boot though. Let me test.

Do you have two separate PPPoE links, one for each node then?

w0w

@stephenw10 said in Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta):

Do you have two separate PPPoE links, one for each node then?

Yes and no. Both nodes are configured for PPPoE with identical settings. I can bring up two PPPoE sessions to my ISP at the same time, but they likely don’t permit it, so I use a script that detects which HA node is Master and starts/stops the PPPoE session accordingly. The script has no impact during boot and includes a safety startup delay. I’ve tested with the script fully disabled — the behavior remains unchanged.

@stephenw10 said in Incomplete VIP configuration on boot causing CARP failure (since 25.07 beta):

Let me test.

I tested this about two weeks ago, but I don’t remember the exact results because the run was interrupted by a continious fatal trap as you remember.
I plan to re-run some tests—I don’t clearly remember the exact steps I took. There may be a link to the new PPPoE kernel module, but that’s just a guess.

stephenw10

Ah, good point. Have you tested with and without the new if_pppoe module?

w0w

@stephenw10
Testing—getting closer. I’ve reproduced the issue in a VM running 2.8.1.
The trigger appears to be certain PPPoE settings: the new PPPoE module combined with “Request an IPv6 prefix/information through the IPv4 connectivity link.”
This appears to be a combination of other settings that triggers the problem.

I can upload either the full exported VM (Virtualbox 7.1.12) or a minimal VM with the virtual disk removed, so you can install pfSense and restore the configuration.

stephenw10

Ah, cool. OK let me try to replicate that here first then....

w0w

@stephenw10
It looks like this also comes down to IPv6 settings and requires two LAN subnets, both using the Track Interface option: one tracking the WAN (PPPoE), and the other either tracking the same WAN with a different Prefix ID or tracking WAN2’s prefix. I’m not sure if it’s necessary, but I created LAN2 as a VLAN. Overall, the configuration is similar to what I described earlier.

w0w

@stephenw10
Any luck?

w0w

@stephenw10
I can provide replicated VM config.xml if you’re still interested. This needs 5 free ports on any hardware, I think and you need to manually edit interfaces before applying it.

stephenw10

Sorry for the delay, I got stuck on some other testing. I'll try to get this setup today.

w0w

config-pfSense.home.arpa-20251018044835.xml.zip u/p=admin/pfsense

In case you are installing in the VM just import the machine into the Virtualbox, and install 2.8.1, then apply configuration.
pfsense28_small_export.7z

Should be resulted in:

vtnet0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: wan2
	options=900b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,LINKSTATE>
	ether 08:00:27:9d:bc:aa
	inet 10.0.2.15 netmask 0xffffff00 broadcast 10.0.2.255
	inet6 fe80::a00:27ff:fe9d:bcaa%vtnet0 prefixlen 64 scopeid 0x1
	inet6 fd17:625c:f037:2:a00:27ff:fe9d:bcaa prefixlen 64 autoconf pltime 14400 vltime 86400
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=23<PERFORMNUD,ACCEPT_RTADV,AUTO_LINKLOCAL>
vtnet1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,TXCSUM_IPV6>
	ether 08:00:27:f9:2b:76
	inet6 fe80::a00:27ff:fe9d:bcaa%vtnet1 prefixlen 64 scopeid 0x2
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet2: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: SYNC
	options=800b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE>
	ether 08:00:27:77:b8:2c
	inet 10.0.222.1 netmask 0xffffff00 broadcast 10.0.222.255
	inet6 fe80::a00:27ff:fe77:b82c%vtnet2 prefixlen 64 scopeid 0x3
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet3: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,TXCSUM_IPV6>
	ether 08:00:27:42:e3:96
	inet6 fe80::a00:27ff:fe9d:bcaa%vtnet3 prefixlen 64 scopeid 0x4
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet4: flags=1008802<BROADCAST,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,TXCSUM_IPV6>
	ether 08:00:27:67:ea:41
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
enc0: flags=0 metric 0 mtu 1536
	options=0
	groups: enc
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lo0: flags=1008049<UP,LOOPBACK,RUNNING,MULTICAST,LOWER_UP> metric 0 mtu 16384
	options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
	inet 127.0.0.1 netmask 0x0
	inet6 ::1 prefixlen 128
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
	groups: lo
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=100<PROMISC> metric 0 mtu 33152
	options=0
	groups: pflog
pfsync0: flags=1000041<UP,RUNNING,LOWER_UP> metric 0 mtu 1500
	options=0
	syncdev: vtnet2 syncpeer: 10.0.222.1 maxupd: 128 defer: off version: 1400
	syncok: 1
	groups: pfsync
lagg0: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: LAN
	options=4800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,TXCSUM_IPV6>
	ether 08:00:27:42:e3:96
	hwaddr 00:00:00:00:00:00
	inet 192.168.1.1 netmask 0xffffff00 broadcast 192.168.1.255
	inet6 fe80::a00:27ff:fe42:e396%lagg0 prefixlen 64 scopeid 0xa
	inet6 fe80::1:1%lagg0 prefixlen 64 scopeid 0xa
	laggproto failover lagghash l2,l3,l4
	laggport: vtnet3 flags=5<MASTER,ACTIVE>
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
lagg1: flags=1008843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	options=4800bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,LINKSTATE,TXCSUM_IPV6>
	ether 08:00:27:f9:2b:76
	hwaddr 00:00:00:00:00:00
	inet6 fe80::a00:27ff:fef9:2b76%lagg1 prefixlen 64 scopeid 0xb
	laggproto failover lagghash l2,l3,l4
	laggport: vtnet1 flags=5<MASTER,ACTIVE>
	groups: lagg
	media: Ethernet autoselect
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
vtnet0.87: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
	description: wifiap
	options=80000<LINKSTATE>
	ether 08:00:27:9d:bc:aa
	inet 10.0.87.2 netmask 0xffffff00 broadcast 10.0.87.255
	inet 10.0.87.5 netmask 0xffffff00 broadcast 10.0.87.255 vhid 3
	inet6 fe80::a00:27ff:fe9d:bcaa%vtnet0.87 prefixlen 64 scopeid 0xc
	inet6 fe80::1:1%vtnet0.87 prefixlen 64 scopeid 0xc
	groups: vlan
	carp: MASTER vhid 3 advbase 1 advskew 254
	      peer 224.0.0.18 peer6 ff02::12
	vlan: 87 vlanproto: 802.1q vlanpcp: 0 parent interface: vtnet0
	media: Ethernet autoselect (10Gbase-T <full-duplex>)
	status: active
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pppoe0: flags=1008851<UP,POINTOPOINT,RUNNING,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1492
	description: WAN
	options=0
	inet6 fe80::a00:27ff:fe9d:bcaa%pppoe0 prefixlen 64 tentative scopeid 0xd
	groups: pppoec
	nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>