25.07.r.20250709.2036 First Boot WireGuard Service not running

Bob.Dig

@stephenw10 said in 25.07.r.20250709.2036 First Boot WireGuard Service not running:

Do you have kern.cam.boot_delay=10000 in /boot/loader.conf?

Don't think so, changed nothing myself I would say.

opensolaris_load="YES"
zfs_load="YES"
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gptid.enable="0"
kern.ipc.nmbclusters="1000000"
kern.ipc.nmbjumbo9="524288"
kern.ipc.nmbjumbop="524288"
loader_conf_files="/boot/loader.conf.lua"
boot_serial="NO"
autoboot_delay="3"
debug.ddb.capture.bufsize="524288"
hw.e6000sw.default_disabled=1
net.link.ifqmaxlen="128"
machdep.hwpstate_pkg_ctrl="1"
net.pf.states_hashsize="262144"

if _ENV["smbios.system.maker"] == "Netgate" then
	if _ENV["smbios.system.product"] == "4100" or _ENV["smbios.system.product"] == "6100" or _ENV["smbios.system.product"] == "6200" or _ENV["smbios.system.product"] == "8200" then
		print("Netgate Cordoba System detected.")
		_ENV["console"]="efi"
		_ENV["hint.cordbuc.0.at"]="isa"
		_ENV["hint.cordbuc.0.port"]="0x800"

		_ENV["hint.gpioled.0.at"]="gpiobus0"
		_ENV["hint.gpioled.0.pins"]="0x001"
		_ENV["hint.gpioled.0.name"]="red1"
		_ENV["hint.gpioled.0.invert"]="1"

		_ENV["hint.gpioled.1.at"]="gpiobus0"
		_ENV["hint.gpioled.1.pins"]="0x002"
		_ENV["hint.gpioled.1.name"]="green1"
		_ENV["hint.gpioled.1.invert"]="1"

		_ENV["hint.gpioled.2.at"]="gpiobus0"
		_ENV["hint.gpioled.2.pins"]="0x004"
		_ENV["hint.gpioled.2.name"]="blue1"
		_ENV["hint.gpioled.2.invert"]="1"

		_ENV["hint.gpioled.3.at"]="gpiobus0"
		_ENV["hint.gpioled.3.pins"]="0x008"
		_ENV["hint.gpioled.3.name"]="amber1"
		_ENV["hint.gpioled.3.invert"]="1"

		_ENV["hint.gpioled.4.at"]="gpiobus0"
		_ENV["hint.gpioled.4.pins"]="0x010"
		_ENV["hint.gpioled.4.name"]="red2"
		_ENV["hint.gpioled.4.invert"]="1"

		_ENV["hint.gpioled.5.at"]="gpiobus0"
		_ENV["hint.gpioled.5.pins"]="0x020"
		_ENV["hint.gpioled.5.name"]="green2"
		_ENV["hint.gpioled.5.invert"]="1"

		_ENV["hint.gpioled.6.at"]="gpiobus0"
		_ENV["hint.gpioled.6.pins"]="0x040"
		_ENV["hint.gpioled.6.name"]="blue2"
		_ENV["hint.gpioled.6.invert"]="1"

		_ENV["hint.gpioled.7.at"]="gpiobus0"
		_ENV["hint.gpioled.7.pins"]="0x080"
		_ENV["hint.gpioled.7.name"]="amber2"
		_ENV["hint.gpioled.7.invert"]="1"

		_ENV["hint.gpioled.8.at"]="gpiobus0"
		_ENV["hint.gpioled.8.pins"]="0x100"
		_ENV["hint.gpioled.8.name"]="red3"
		_ENV["hint.gpioled.8.invert"]="1"

		_ENV["hint.gpioled.9.at"]="gpiobus0"
		_ENV["hint.gpioled.9.pins"]="0x200"
		_ENV["hint.gpioled.9.name"]="green3"
		_ENV["hint.gpioled.9.invert"]="1"

		_ENV["hint.gpioled.10.at"]="gpiobus0"
		_ENV["hint.gpioled.10.pins"]="0x400"
		_ENV["hint.gpioled.10.name"]="blue3"
		_ENV["hint.gpioled.10.invert"]="1"

		_ENV["hint.gpioled.11.at"]="gpiobus0"
		_ENV["hint.gpioled.11.pins"]="0x800"
		_ENV["hint.gpioled.11.name"]="amber3"
		_ENV["hint.gpioled.11.invert"]="1"
	end
	if _ENV["smbios.system.product"] == "4200" then
		print("Netgate 4200 detected.")
		_ENV["console"]="efi"
		_ENV["hw.uart.console"]="mm:0xfe03e000"
		_ENV["dev.igc.0.iflib.override_nrxqs"]="1";
		_ENV["dev.igc.1.iflib.override_nrxqs"]="1";
		_ENV["dev.igc.2.iflib.override_nrxqs"]="1";
		_ENV["dev.igc.3.iflib.override_nrxqs"]="1";
	end
	if _ENV["smbios.system.product"] == "8300" then
		print("Netgate 8300 detected.")
		_ENV["ice_ddp_load"]="yes"
		_ENV["led_8300_load"]="yes"
		_ENV["igpio_load"]="yes"
		_ENV["kern.crypto.iimb.max_threads"]="12"
	end
end

if string.sub(_ENV["smbios.planar.product"], 1, 11) == "80300-0134-" then
	print("Netgate 7100 detected.")
	_ENV["boot_serial"]="YES"
	_ENV["console"]="comconsole"
	_ENV["hint.mdio.0.at"]="ix2"
	_ENV["hint.e6000sw.0.addr"]="0"
	_ENV["hint.e6000sw.0.is8190"]="1"
	_ENV["hint.e6000sw.0.port0disabled"]="1"
	_ENV["hint.e6000sw.0.port9cpu"]="1"
	_ENV["hint.e6000sw.0.port10cpu"]="1"
	_ENV["hint.e6000sw.0.port9speed"]="2500"
	_ENV["hint.e6000sw.0.port10speed"]="2500"
end

if _ENV["smbios.system.product"] == "DFFv2" or _ENV["smbios.system.product"] == "RCC" or _ENV["smbios.system.product"] == "RCC-VE" then
	print("Netgate RCC detected.")
	_ENV["boot_serial"]="YES"
	_ENV["console"]="comconsole"
	_ENV["comconsole_port"]="0x2F8"
	_ENV["hint.uart.0.at"]="isa"
	_ENV["hint.uart.0.flags"]="0x00"
	_ENV["hint.uart.1.at"]="isa"
	_ENV["hint.uart.1.flags"]="0x10"
end

comconsole_speed="115200"

stephenw10

Hmm, try adding it there or to loader.conf.local and see if that changes anything at boot.

Bob.Dig

@stephenw10 said in 25.07.r.20250709.2036 First Boot WireGuard Service not running:

or to loader.conf.local and see if that changes anything at boot.

WG-problem still there. Or did you mean something different?

stephenw10

Nope that's what I meant, the delay at boot could have impacted a race condition at boot.

Looking at that boot log there is a lot of stuff happening during boot. Some of that might be better suppressed but that can cause other issues.

What was the last version that booted correctly?

Bob.Dig

@stephenw10 said in 25.07.r.20250709.2036 First Boot WireGuard Service not running:

What was the last version that booted correctly?

It looks like 25.07.r.20250715.1733 is most problematic with every reboot.
25.07.r.20250709.2036 only had this problem on the first boot after the upgrade.

I just booted through all the snapshots to verify this. Although my network is evolving all the time, but it looks like 25.07.r.20250715.1733 is the reason.

stephenw10

Hmm, OK those dpinger error logs above are not in the given systen log txt. I assume those do appear in the system log? But not at boot?

Bob.Dig

@stephenw10 I can wait 20 minutes and post again if this helps. Can I upload to another place than the public forum? And I guess I should post the logfile and not copying stuff from the browser.

stephenw10

Yup you can upload here: https://nc.netgate.com/nextcloud/s/rLWRrGHTF6MoNaW

Bob.Dig

@stephenw10 This dpinger stuff is only in the log after I "enabled" the gateways by hand.
This time, I had halted pfSense before, kea-dhcp4 wasn't running according to service status widget.

stephenw10

Hmm, so in that log after 'Bootup complete' none of the WG tunnels or gateways were up?

Bob.Dig

@stephenw10 I dropped you some pictures what I actually see and this is interesting, at least to me. I just rebooted to have the same problem.

Dashboard shows none of the WireGuard Gateways and Service Status shows WireGuard not running. If I push the button to start WireGuard, nothing happens.
WireGuard Status shows that only the connection to Debian is not running, others do. (Others in red are expected to not run and are not of interest.)
In Gateways I enable all of the WireGuard Gateways by hand, they are for whatever reason not enabled.
Going back to WireGuard Status I now see a warning that WireGuard is not running (why not before?). Then I press the button to start the WireGuard Service and this time it does.
Everything is running and working like it should.

Bob.Dig

@stephenw10 I made some further changes. I removed the gateway for that problematic tunnel and also removed keep alive etc so that it is not expected to be running at start.
That didn't changed anything for me. At next reboot, gateways are down as is WireGuard. So it seems more of a general problem, although no one else is reporting it...

Bob.Dig

@stephenw10 Today I rebooted the host (Hyper-V) and had no problem at all. Don't know if this points towards being a weird virtualization issue... But then, why would WireGuard be effected...

Bob.Dig

Running /etc/pppoe_restart_pppoe0 via Command Prompt fixes the issue after reboot for me, maybe other things do too...

Edit: Interestingly, doing it as a Services: Shellcmd Settings like shown below doesn't do it, so I removed that after testing (it one time).
Screenshot 2025-07-27 104416.png

Edit2: So to me it is looking like a timing problem between 25.07.r.20250709 and 25.07.r.20250715.

stephenw10

Yup it does look like a timing issue or race condition. You have quite a lot of stuff running there. I'm not able to replicate it here. Yet.

Are you using failover or loadbalance with those WG gateways? If not you might try disabling the monitoring action on them. That will significantly reduce the boot processes.

Bob.Dig

@stephenw10 said in 25.07.r.20250709.2036 First Boot WireGuard Service not running:

Are you using failover or loadbalance with those WG gateways? If not you might try disabling the monitoring action on them.

Not with them but with others. Disabling the monitoring action only on them didn't make a difference.

Edit: Disabling it on all but WAN also made no difference.