SOLVED: after restarting Netgate 6100, miniupnpd and WireGuard service fail to start

ChrisJenk

UPDATE - mystery solved...

Yesterday I installed the 'shellcmd' package so that I could run a script on boot up to start the iperf3 server (since I always want it to be running). In that script I forgot to include the -D parameter to run the iperf3 server as a daemon. As a result that script never exited. It seems that this makes pfSense think that startup has not finished (I did notice that all three lights continued to flash blue, which is not usual). This seems to prevent some services from starting or at least from remaining running.

Anyway, I modified that script to include the -D option and now everything works again.

Phew!

I rebooted my Netgate 6100 (running 23.09.1) this morning as part if an investigation into some Internet performance issues (doesn't;t seem to be related to the negate as it happens). After the restart neither the miniupnpd or the wireguard service had started (they should and normally do).

I rebooted again with the same result. I was able, after a delay, to manually start miniupnpd from the dashboard and it remained running. However, Whenever I try to start WireGuard it seems to start but them after a short time shows as not running again. I tried reinstalling the WireGuard package but it didn't change the behaviour.

Bizarrely, my WireGuard tunnel is up and clients can connect and work as expected, but I am not able to make any config changes etc.

I grepped the logs for wireguard related messages but to my untrained eye nothing stood out. Each attempt to restart the service generates these log messages:

system.log:Jan 10 11:53:12 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s).
system.log:Jan 10 11:53:13 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s).
system.log:Jan 10 11:53:13 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard).
system.log:Jan 10 11:53:13 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard).
system.log:Jan 10 11:53:13 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard).
system.log:Jan 10 11:53:13 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard).
system.log:Jan 10 11:53:13 router php_wg[20890]: /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary.

I'm kind of stuck at this point. Can anyone suggest what to look for? Has anyone experienced this before? Prior to this reboot everything was working just fine.