25.11 : IPv6 gateway status UNKNONW after boot

patient0

G'day,

That issue exists for quite some time (at least since BETA), I was too lazy to something about it. But now is the time.

The pfSense+ runs as a VM on Proxmox, with three interfaces, WAN, LAN, TEST.

The WAN interface got static configuration for IPv4 and IPv6. The IPv6 is set to an ULA and is NAT-ed by the Proxmox firewall to GUA. PVE 8 and now 9 since Saturday, no change in behavior. The whole NAT is done since Hetzner only gives out /64 to their servers.

There is also a NAT66 for LAN and TEST.

Since I have ZFS snapshots I can switch back to 25.07.1 and there the IPv6 gateway issue does not exist. I deleted the BETA and previous RC's so I can't pin it more closely.

The IPv4 gateway comes up als ONLINE always, the IPv6 gateway never. A restart of the gateway monitoring after boot does solve the issue.

What I tried:

I did disable the IPv4 gateway completely in the hope that it's a timing issue and since then only the IPv6 would exist, that it would work -> not successful
Switching back to 25.07.1 multiple times, the IPv6 gateway is ONLINE every time. Using the same (external) monitoring IP
Tried three different monitoring IPs, Cloudflare and two Hetzner DNS server -> no change
Selected 'IPv6 will use IPv4 connectivity' in the WAN interface settings. Again thinking that if it's an timing issue that pfSense would wait until IPv4 is completely up before setting up WAN IPv6 -> no change, issue still exists

I'd like to debug that issue a bit more and if someone could give me a pointer as to where the gateway magic happens after reboot, that would be helpful.

stephenw10

Are you able to connect to the gateway manually in that situation? Ping6 it for example?

If you check the states or a pcap do you see dpinger sending the monitoring traffic?

patient0

@stephenw10 indeed, I did fail to mention two important points:

ping to the gateway works and so does IPv6 (flawless on the client and on the pfSense)
there is no dpinger process for the IPv6 WAN interface. No entry in the gateway log about dpinger being killed or crashed

stephenw10

Ah so dpinger fails to start entirely. Hmm, that does seem like an ordering issue.

patient0

@stephenw10 said in 25.11 : IPv6 gateway status UNKNONW after boot:

Hmm, that does seem like an ordering issue.

I do suspect so, is there a simple way to add a delay or follow the process of the WAN interface status while booting? Or add a simple sleep(5) in the right file?

stephenw10

If you run: touch /verbose_rc

You will see a lot more info at boot as the rc scripts start the processes.

patient0

@stephenw10 excellent, I'll give that a go.

patient0

@stephenw10 unfortunately I wasn't able to draw any more insights out of the boot up with /verbose_rc and it's content.

I'll keep digging.

patient0

@stephenw10 Adding a crontab for the root user that restarts dpinger on @reboot helped.

@reboot /usr/local/sbin/pfSsh.php playback svc restart dpinger

Is there an undocumented option to enable more output from dpinger so I cloud see if it silently crashes or if not being called.

And are you guys using dpinger straight from @dennypage repo? Meaning could I compile it with debug options and temporarily replace the build-in one?

Gertjan

@patient0 said in 25.11 : IPv6 gateway status UNKNONW after boot:

straight from @dennypage repo?

That's his main forum profile page.
But you talked abut its repo page - github probably ? can you link that ?

patient0

@Gertjan said in 25.11 : IPv6 gateway status UNKNONW after boot:

But you talked abut its repo page - github probably ? can you link that ?

I of course talked about the GH repo, yes:

https://github.com/dennypage/dpinger

stephenw10

Yeah we don't carry any patches to dpinger, you should be able to do that.

dennypage

@patient0 said in 25.11 : IPv6 gateway status UNKNONW after boot:

Is there an undocumented option to enable more output from dpinger so I cloud see if it silently crashes or if not being called.

Dpinger logs on successful startup, and logs at exit. If it is failing to start it is generally from requesting to bind to an address that doesn't exist.

FWIW, it's on my list to change fatal logging to support syslog as I have recently done for my other daemons. In the interim, you can just modify the startup scripts to redirect stdout/stderr to a file and see what's what. Here is an example:

/usr/local/bin/dpinger -S -r 0 -i WAN2_DHCP -B 78.78.200.128 -p /var/run/dpinger_WAN2_DHCP~78.78.200.128~1.0.0.1.pid -u /var/run/dpinger_WAN2_DHCP~78.78.200.128~1.0.0.1.sock -d 1 -s 1000 -l 2000 -t 60000 -A 1000 -D 750 -L 20 1.0.0.1 >> /tmp/dpinger.out 2>&1

dennypage

said in 25.11 : IPv6 gateway status UNKNONW after boot:

FWIW, it's on my list to change fatal logging to support syslog as I have recently done for my other daemons.

I pushed a change to GitHub for this.

patient0

@dennypage thanks a lot @dennypage.

I added a log entry in /etc/int/gwlb.inc before dpinger is called. And in the rc.bootup script I get only one log entry which would indicate that dpinger is only called once. And there is only one dpinger PID file in the /var/run folder, dpinger did not crash I'd say otherwise the PID file would be a leftover.

The cron job I added results in two log entries, two time mwexec is called with dpinger.

I'll add more log output to gwlb.inc to narrow it down.

patient0

Ok, I tracked it down to where the timing issue happens.

In /etc/rc.bootup, setup_gateway_monitor() (/etc/inc/gwlb.inc) is called.
And in setup_gateway_monitor in

foreach ($gateways_arr as $gwname => $gateway) {

(line 308) on line 376 gwifip is set with

$gwifip = find_interface_ipv6($gateway['interface'], true);.

where $gateway['interface'] is vtnet0 in my case. But the find_interface_ip6() (in /etc/inc/interfaces.inc) return null setting gwifip to null. It seems that the pfSense wasn't yep able to get the IPv6 address of it. Adding a sleep(1) before find_interface_ipv6 gives it enough time (0.75 sec was not enough).

I don't have clever solution to it, don't now how to fool-proof find_interface_ipv6.

Btw: I reinstalled pfSense 25.11-RC to make sure that there were not leftovers from earlier version.

dennypage

@patient0 Just out of curiosity, what setting do you have for

Do not wait for a RA

on the interface?

patient0

@dennypage said in 25.11 : IPv6 gateway status UNKNONW after boot:

Do not wait for a RA

That option/checkbox is not available, it's a static interface configuration.

dennypage

@patient0 said in 25.11 : IPv6 gateway status UNKNONW after boot:

@dennypage said in 25.11 : IPv6 gateway status UNKNONW after boot:

Do not wait for a RA

That option/checkbox is not available, it's a static interface configuration.

You did mention that earlier. My bad.

patient0

I do have to re-evaluate my problem solving skills: The issue stems from the fact that the WAN address is not a GUA. Should have tried that sooner.

On the Proxmox host I setup an additional NAT66 rule to NAT some made up GUA to my actual GUA. And now it all works.

@stephenw10 do you have the possibility to test with WAN set to an ULA?

I had a look into get_interface_addresses() (/etc/inc/interfaces.inc) but it's quite a beast and (if confirmed to be the issue) someone at Netgate will be way more efficient to fix it.