Wireguard fails after reboot (2.8.0)
-
@stephenw10 This is immediately after I rebooted without reinstalling WG. Let me know if you need another one after WG has been reinstalled.
-
So what new logs appear there if you try to start WG when it fails?
-
@stephenw10 This is immediately after I tried restarting WG (and fails) from the UI Service Status page, without restarting dpinger.
-
Hmm, that was some time it booted?
It's triggering a bunch of rc.bootup events which seems odd. Perhaps you have something preventing it ever reaching 'bootup complete'.
Do you see 'bootup complete' in the system logs after booting?
Do you see /var/run/booting still present?
-
@stephenw10 Can you tell me exactly where I can see this information?
-
Bootup Complete should appear in the main system log like:
You can check the boot file from the CLI like:
[2.8.0-RELEASE][admin@t70.stevew.lan]/root: ls /var/run charon.ctl ld-elf.so.hints charon.pid ld-elf32.so.hints charon.vici log charon.wlst logpriv check_reload_status miniupnpd.pid cron.pid nginx-webConfigurator.pid daemon-charon.pid ntpd.pid daemon_sshguard.pid openvpn_server1.pid devd.pid pfSense_version devd.pipe pfSense_version.rc devd.seqpacket.pipe php-fpm.pid dhclient.igb0.pid php-fpm.socket dmesg.boot ping_hosts.pid dnsbl.pid powerd.pid dnsmasq.pid sshd.pid dpinger_VTI0_VTIV4~10.66.11.1~10.66.11.2.pid sshguard.pid dpinger_VTI0_VTIV4~10.66.11.1~10.66.11.2.sock suricata-ctrl-socket-60565 dpinger_WAN_DHCP~172.21.16.239~172.21.16.1.pid suricata_igb160565.pid dpinger_WAN_DHCP~172.21.16.239~172.21.16.1.sock syslog.pid expire_accounts.pid unbound.pid filter_reload_status update_alias_url_data.pid filterdns.pid updaterrd.sh.pid filterlog.pid utmp ipsec_keepalive.pid utx.active -
@stephenw10 Sorry for the delayed reply - I have just got back from a business trip. Anyway, this is the output from the CLI
[2.8.1-RC][root@pfSense.mymain.local]/root: ls /var/run
check_reload_status
cron.pid
daemon_sshguard.pid
devd.pid
devd.pipe
devd.seqpacket.pipe
dhclient.igb0.pid
dmesg.boot
dnsbl.pid
dpinger_VPNUNLIMITED_L2TP~10.240.0.2~10.240.0.1.pid
dpinger_VPNUNLIMITED_L2TP~10.240.0.2~10.240.0.1.sock
dpinger_WANV6_TUNNELV6~2001:470:1f08:84a::2~2001:470:1f08:84a::1.pid
dpinger_WANV6_TUNNELV6~2001:470:1f08:84a::2~2001:470:1f08:84a::1.sock
dpinger_WAN_DHCP~82.13.203.142~82.13.202.1.pid
dpinger_WAN_DHCP~82.13.203.142~82.13.202.1.sock
dpinger_wg1GW~10.102.1.114~10.102.1.114.pid
dpinger_wg1GW~10.102.1.114~10.102.1.114.sock
dpinger_wg2GW~10.102.100.206~10.102.100.206.pid
dpinger_wg2GW~10.102.100.206~10.102.100.206.sock
expire_accounts.pid
filter_reload_status
filterlog.pid
ipsec_keepalive.pid
kea
kea2fib6.cache
kea4-ctrl-socket
kea4-ctrl-socket.lock
kea6-ctrl-socket
kea6-ctrl-socket.lock
l2tp_opt9.pid
ld-elf.so.hints
ld-elf32.so.hints
log
logpriv
mdns-bridge.pid
miniupnpd.pid
nginx-webConfigurator.pid
ntpd.pid
pfSense_version
pfSense_version.rc
php-fpm.pid
php-fpm.socket
ping_hosts.pid
radvd.pid
sshd.pid
sshguard.pid
syslog.pid
unbound.pid
update_alias_url_data.pid
updaterrd.sh.pid
utmp
utx.active
wireguardd.pid
[2.8.1-RC][root@pfSense.mymain.local]/root: [2.8.1-RC][root@pfSense.mymain.local]/root: ls /var/run
kea4-ctrl-socket.lock
kea6-ctrl-socket
kea6-ctrl-socket.lock
l2tp_opt9.pid
[2.8.1-RC][root@pfSense.mymain.local]/root:: Too many arguments.
[2.8.1-RC][root@pfSense.mymain.local]/root: check_reload_status
ld-elf.so.hints
ld-elf32.so.hints
log
logpriv
mdns-bridge.pid
miniupnpd.pid
nginx-webConfigurator.pid
ntpd.pid
pfSense_version
pfSense_version.rc
php-fpm.pid
php-fpm.socket
ping_hosts.pid
radvd.pid
sshd.pid
sshguard.pid
syslog.pid
unbound.pid
update_alias_url_data.pid
updaterrd.sh.pid
utmp
utx.active
wireguardd.pid -
OK so the file is not present which indicates it has completed boot.
Do you see the 'bootup complete' line in the system logs?
-
-
This post is deleted! -
@stephenw10 I decided to do a fresh install of pfsense 2.8.1, and restored my original config.xml. The PHP_errors have now completely disappeared. However, the wireguard package still does not start at bootup.
I installed the shellcmd package and noticed that wireguard has an earlyshellcmd at bootup. I think it starts, but then immediately stops as I noticed that both my wireguard gateways that I have setup on tun_wg1 and tun_wg2 in System>Routing are 'stopped' (greyed out) after a reboot - I have to manually start them both and then start Wireguard on the opening GUI page.
I have uninstalled wireguard and reinstalled it, but the issue still persists.
Could there be an issue in my config.xml file? I am happy to send you the relevant parts of the xml file if you tell me what to send.
Otherwise, in the meantime, would you know the line of code that i can use in shellcmd to autommatically restart the two wireguard gateway interfaces so I don't have to manually start them on every reboot?
-
It feels more like some sort of race condition at boot. Like Wireguard tries to start on an interface that hasn't been setup yet, maybe a VPN or other tunnel type. That GIF tunnel perhaps? Can you try disabling that as a test?
-
This post is deleted! -
@stephenw10 I think we are getting somewhere and maybe to the route of the issue.
I disabled the GIF tunnel (Hurricane Electric) as you suggested, rebooted, and the wireguard gateways and wireguard service status were stopped. However, when i started the GIF tunnel again, the wireguard gateways and wireguard service status automatically started up.
There is a secondary issue however - once the wireguard service starts, the GIF Tunnel changes to 'Offline, Packetloss'. If I then restart the GIF tunnel, it remains 'Offline, Packetloss'.
There definitely seems to be a link between the wireguard service and the GIF tunnel.
-
Hmm, is there some IP/subnet conflict between the GIF and Wireguard tunnels?
-
@stephenw10 Not that I can see. The GIF tunnel (Hurricane Electric) is using 216.66.80.26 and the two wireguard tunnels use 10.102.1.114 and 10.102.100.206.
-
This post is deleted! -
@stephenw10 @stephenw10 I decided to consult chatgpt and have it write a bash script that runs immediately after boot in /usr/local/etc/rc.d/restart_gateways.sh. The wireguard tunnels now start automatically without any manual intervention. It seems that it was just a timing issue all along. The bash script works perfectly by first stopping and then restarting dpinger, and then waiting 20 secs to give the wireguard tunnels time to fully enable, before starting the wireguard service.
I did try to use the shellcmd package, but it didn't work as a delay is needed between making sure the wireguard gateways have fully started, before enabling the wireguard service.
#!/bin/sh # # Restart dpinger 5 sec after boot, then delay starting WireGuard by 20 secs # case "$1" in start) # Wait 5 secs before stopping dpinger /bin/sleep 5 /usr/local/sbin/pfSsh.php playback svc stop dpinger /usr/local/sbin/pfSsh.php playback svc start dpinger # Wait 20 secs before starting WireGuard /bin/sleep 20 /usr/local/sbin/pfSsh.php playback svc start wireguard ;; stop) # Nothing special on stop ;; restart) $0 stop $0 start ;; esacI haven't had any issues with the GIF tunnel (Hurricane Electric) since - i think it was just an anomally, as I have tested this script many times and it has started online every time since.
-
You could try an afterfilterchange shellcmd to trigger a script. That would be triggered when any tunnel comes up.
-
This post is deleted!
