pfS won't reboot via ssh session
-
I have seen this issue for a while but today wanted to test a little more.
Here is what I do and see.
- ssh'ed to pfS, option 5, Y
- See in the ssh session term:
Netgate pfSense Plus is rebooting now.
Stopping package arpwatch...done.
Stopping package snort...done.
Stopping package Avahi...done.
Stopping package Tailscale...- See email Notification "Netgate pfSense Plus is rebooting now."
Then another email in 1 min:
"12:39:00 Service Watchdog detected service arpwatch stopped. Restarting arpwatch (Arpwatch Daemon)
12:39:00 Service Watchdog detected service snort stopped. Restarting snort (Snort IDS/IPS Daemon)" (suspect Watchdog and tested with all services removed and it did not help)- UI and ssh (new session) stopped working
- Routing seems to be working, e..g I see YouTubeTV streaming
- Router is stuck, no UI/ssh access
- The console is responding fine and finally I rebooted successfully via it.
PS: Reboot via the UI and console work fine
Any thoughts?
-
What do you see in the system log for that after you reboot?
What happens if you disable the service watchdog?
-
@stephenw10 said in pfS won't reboot via ssh session:
What do you see in the system log for that after you reboot?
What happens if you disable the service watchdog?
I saw nothing unusual in the log
I am aware that watchdog can be disabled, can it?
I simply removed all services and the problem persisted -
But what is logged? Does it show the reboot script running?
For example:
Mar 26 22:14:21 sshd-session 37383 Accepted keyboard-interactive/pam for admin from 172.21.16.8 port 55400 ssh2 Mar 26 22:14:34 reboot 89782 rebooted by admin Mar 26 22:14:34 syslogd exiting on signal 15 Mar 26 22:15:51 syslogd kernel boot file is /boot/kernel/kernel Mar 26 22:15:51 kernel pflog0: promiscuous mode disabled Mar 26 22:15:51 kernel controltun0: link state changed to DOWN Mar 26 22:15:51 kernel Waiting (max 60 seconds) for system process `vnlru' to stop... done Mar 26 22:15:51 kernel Waiting (max 60 seconds) for system process `syncer' to stop... Mar 26 22:15:51 kernel Syncing disks, vnodes remaining... 0 0 done Mar 26 22:15:51 kernel All buffers synced. Mar 26 22:15:51 kernel Uptime: 2h42m3s Mar 26 22:15:51 kernel ---<<BOOT>>--- -
@stephenw10 said in pfS won't reboot via ssh session:
rebooted by admin
See two reboots below:
Mar 26 15:47:12 kernel FreeBSD 16.0-CURRENT #7 plus-RELENG_25_11_1-n256519-3d5e07ee0abe: Mon Jan 19 17:34:47 UTC 2026 Mar 26 15:47:12 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Mar 26 15:47:12 kernel The Regents of the University of California. All rights reserved. Mar 26 15:47:12 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Mar 26 15:47:12 kernel Copyright (c) 1992-2025 The FreeBSD Project. Mar 26 15:47:12 kernel ---<<BOOT>>--- Mar 26 15:47:12 syslogd kernel boot file is /boot/kernel/kernel Mar 26 15:46:28 syslogd exiting on signal 15 Mar 26 15:46:28 reboot 13532 rebooted by root Mar 26 15:46:26 php-fpm 99267 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_DHCP. Mar 26 15:46:26 php-fpm 94338 /rc.dyndns.update: Dynamic DNS (chudak.no-ip.org) Could not determine the request IP address (using "wan", "igb0"): gateway not online Mar 26 15:46:26 php-fpm 99267 /rc.openvpn: Gateway, NONE AVAILABLE Mar 26 15:46:26 php_pfb 96316 [pfBlockerNG] filterlog daemon stopped Mar 26 15:46:26 tail_pfb 96011 [pfBlockerNG] Firewall Filter Service stopped Mar 26 15:46:26 lighttpd_pfb 93609 [pfBlockerNG] DNSBL Webserver stopped Mar 26 15:46:26 php_wg 66367 /usr/local/pkg/wireguard/includes/wg_service.inc: The command '/usr/sbin/arp -d -i 'tun_wg0' -a > /dev/null 2>&1 ' returned exit code '1', the output was '' Mar 26 15:46:26 php_wg 66367 /usr/local/pkg/wireguard/includes/wg_service.inc: The command '/sbin/ifconfig 'tun_wg0' -staticarp ' returned exit code '1', the output was 'ifconfig: interface tun_wg0 does not exist' Mar 26 15:46:25 check_reload_status 682 Reloading filter Mar 26 15:46:25 check_reload_status 682 Restarting OpenVPN tunnels/interfaces Mar 26 15:46:25 check_reload_status 682 Restarting IPsec tunnels Mar 26 15:46:25 check_reload_status 682 updating dyndns WAN_DHCP Mar 26 15:46:25 rc.gateway_alarm 86029 >>> Gateway alarm: WAN_DHCP (Addr:135.180.64.1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%) Mar 26 15:46:23 php_wg 66367 /usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 135.180.64.1 and adding a new route through 157.131.248.1 Mar 26 15:46:23 kernel tun_wg0: link state changed to DOWN Mar 26 15:46:23 avahi-daemon 94612 Withdrawing workstation service for tun_wg0. Mar 26 15:46:23 devd 1390 notify_clients: send() failed; dropping unresponsive client Mar 26 15:46:23 php-cgi 44511 rc.initial.reboot: The command '/usr/local/etc/rc.d/pfsense_tailscaled stop' returned exit code '1', the output was 'Stopping tailscaled. Waiting for PIDS: 19787.' Mar 26 15:46:22 kernel tailscale0: link state changed to DOWN Mar 26 15:46:22 avahi-daemon 94612 Withdrawing workstation service for tailscale0. Mar 26 15:46:15 php-cgi 47647 notify_monitor.php: Message sent to XYZ OK Mar 26 15:46:14 snort 22174 Shutting down Legacy Blocking Mode custom output plugin, 'alert_pf'. Mar 26 15:46:14 kernel igb0: promiscuous mode disabled Mar 26 15:46:14 snort 22174 *** Caught Term-Signal Mar 26 15:46:13 kernel igb2: promiscuous mode disabled Mar 26 15:46:13 kernel igb1: promiscuous mode disabled Mar 26 15:46:13 SnortStartup 51248 Snort STOP for WAN snort protect(igb0)... Mar 26 15:46:13 php-cgi 44511 rc.initial.reboot: Stopping all packages. Mar 26 15:46:08 login 4733 login on ttyv0 as root Mar 26 15:43:10 snort 22174 Firewall interface IP address change notification monitoring thread started for Legacy Blocking Mode. ABOVE REBOOTING VIA CONSOLE ^^^ ROUTER STUCK HERE Mar 26 15:42:19 php-cgi 47392 notify_monitor.php: Message sent to XYZ OK Mar 26 15:42:03 kernel igb0: promiscuous mode enabled Mar 26 15:42:03 kernel igb2: promiscuous mode enabled Mar 26 15:42:03 kernel igb1: promiscuous mode enabled Mar 26 15:42:03 arpwatch 35020 listening on igb0 Mar 26 15:42:03 arpwatch 34665 listening on igb2 Mar 26 15:42:03 arpwatch 34426 listening on igb1 Mar 26 15:42:03 kernel igb0: promiscuous mode enabled Mar 26 15:42:03 kernel igb2: promiscuous mode enabled Mar 26 15:42:03 kernel igb1: promiscuous mode enabled Mar 26 15:42:03 arpwatch 35020 listening on igb0 Mar 26 15:42:03 arpwatch 34665 listening on igb2 Mar 26 15:42:03 arpwatch 34426 listening on igb1 Mar 26 15:42:01 tailscale 24415 Bringing up tailscale0 with --auth-key=file:/dev/null --login-server=https://controlplane.tailscale.com --advertise-exit-node --accept-routes --accept-dns --advertise-routes=192.168.90.0/24,192.168.70.0/24,192.168.20.0/24 Mar 26 15:42:01 tailscale 23927 Added tailscale0 to interface group Tailscale Mar 26 15:42:01 tailscale 22973 Found device tailscale0 Mar 26 15:42:01 snort 9711 Added firewall interface tun_wg0 IPv4 address 10.0.20.1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface ovpns2 IPv4 address 192.168.20.1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface ovpns2 IPv6 address fe80::20e:c4ff:fed1:6f27 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface lo0 IPv4 address 127.0.0.1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface lo0 IPv6 address fe80::1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface lo0 IPv6 address ::1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb2 IPv4 address 192.168.70.1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb2 IPv6 address fe80::20e:c4ff:fed1:6f29 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb1 IPv4 address 10.10.10.1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb1 IPv4 address 192.168.90.1 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb1 IPv6 address fe80::20e:c4ff:fed1:6f28 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb0 IPv4 address 157.131.250.34 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Added firewall interface igb0 IPv6 address fe80::20e:c4ff:fed1:6f27 to automatic interface IP Pass List. Mar 26 15:42:01 snort 9711 Populating the internal list of firewall interface IP addresses for auto-whitelisting. Mar 26 15:42:01 snort 9711 Initializing 'alert_pf' custom output module for Legacy Mode Blocking. Mar 26 15:42:00 kernel tun0: changing name to 'tailscale0' Mar 26 15:42:00 kernel tun0: link state changed to UP Mar 26 15:42:00 avahi-daemon 94612 Withdrawing workstation service for tun0. Mar 26 15:42:00 tailscale 20642 Waiting for device tailscale0 Mar 26 15:42:00 SnortStartup 9668 Snort START for WAN snort protect(igb0)... Mar 26 15:42:00 php-cgi 95071 servicewatchdog_cron.php: Service Watchdog detected service snort stopped. Restarting snort (Snort IDS/IPS Daemon) Mar 26 15:42:00 php-cgi 95071 servicewatchdog_cron.php: Service Watchdog detected service tailscale stopped. Restarting tailscale (Tailscale Daemon) Mar 26 15:42:00 php-cgi 95071 servicewatchdog_cron.php: Service Watchdog detected service arpwatch stopped. Restarting arpwatch (Arpwatch Daemon) Mar 26 15:41:48 kernel tailscale0: link state changed to DOWN Mar 26 15:41:47 sshd-session 75269 Connection closed by 192.168.90.1 port 63717 Mar 26 15:41:47 avahi-daemon 94612 Withdrawing workstation service for tailscale0. Mar 26 15:41:40 snort 42404 Shutting down Legacy Blocking Mode custom output plugin, 'alert_pf'. Mar 26 15:41:40 php-cgi 47392 notify_monitor.php: Message sent to XYZ OK Mar 26 15:41:39 kernel igb0: promiscuous mode disabled Mar 26 15:41:39 snort 42404 *** Caught Term-Signal Mar 26 15:41:38 SnortStartup 51247 Snort STOP for WAN snort protect(igb0)... Mar 26 15:41:38 kernel igb2: promiscuous mode disabled Mar 26 15:41:38 kernel igb1: promiscuous mode disabled Mar 26 15:41:38 php-cgi 34072 rc.initial.reboot: Stopping all packages. Mar 26 15:40:55 sshd-session 73598 Accepted keyboard-interactive/pam for admin from 192.168.90.1 port 63717 ssh2 REBOOT VIA SSH ^^^ -
Mmm, so it looks like the service watchdog is restarting a number of services after the reboot script stops them. So the reboot script never finishes.
Try that again with everything removed from the watchdog.
-
Was doing just that.
No loveMar 26 16:11:02 pkg-static 11278 pfSense-pkg-Service_Watchdog-1.8.7_4 deinstalled
Mar 26 16:13:58 kernel Origin="GenuineIntel" Id=0x306d4 Family=0x6 Model=0x3d Stepping=4 Mar 26 16:13:58 kernel CPU: Intel(R) Core(TM) i5-5250U CPU @ 1.60GHz (1596.35-MHz K8-class CPU) Mar 26 16:13:58 kernel VT(efifb): resolution 800x600 Mar 26 16:13:58 kernel FreeBSD clang version 19.1.7 (https://github.com/llvm/llvm-project.git llvmorg-19.1.7-0-gcd708029e0b2) Mar 26 16:13:58 kernel root@pfsense-build-release-amd64-1.eng.atx.netgate.com:/var/jenkins/workspace/pfSense-Plus-snapshots-25_11_1-main/obj/amd64/8uazGBdh/var/jenkins/workspace/pfSense-Plus-snapshots-25_11_1-main/sources/FreeBSD-src-plus-RELENG_25_11_1/amd64.amd64/sys/pfSense amd64 Mar 26 16:13:58 kernel FreeBSD 16.0-CURRENT #7 plus-RELENG_25_11_1-n256519-3d5e07ee0abe: Mon Jan 19 17:34:47 UTC 2026 Mar 26 16:13:58 kernel FreeBSD is a registered trademark of The FreeBSD Foundation. Mar 26 16:13:58 kernel The Regents of the University of California. All rights reserved. Mar 26 16:13:58 kernel Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 Mar 26 16:13:58 kernel Copyright (c) 1992-2025 The FreeBSD Project. Mar 26 16:13:58 kernel ---<<BOOT>>--- Mar 26 16:13:58 syslogd kernel boot file is /boot/kernel/kernel Mar 26 16:13:14 syslogd exiting on signal 15 Mar 26 16:13:14 reboot 63025 rebooted by root Mar 26 16:13:12 php-fpm 25680 /rc.openvpn: OpenVPN: One or more OpenVPN tunnel endpoints may have changed IP addresses. Reloading endpoints that may use WAN_DHCP. Mar 26 16:13:12 php-fpm 9779 /rc.dyndns.update: Dynamic DNS (chudak.no-ip.org) Could not determine the request IP address (using "wan", "igb0"): gateway not online Mar 26 16:13:12 php-fpm 25680 /rc.openvpn: Gateway, NONE AVAILABLE Mar 26 16:13:11 php_pfb 46628 [pfBlockerNG] filterlog daemon stopped Mar 26 16:13:11 tail_pfb 45766 [pfBlockerNG] Firewall Filter Service stopped Mar 26 16:13:11 lighttpd_pfb 43001 [pfBlockerNG] DNSBL Webserver stopped Mar 26 16:13:11 php_wg 17461 /usr/local/pkg/wireguard/includes/wg_service.inc: The command '/usr/sbin/arp -d -i 'tun_wg0' -a > /dev/null 2>&1 ' returned exit code '1', the output was '' Mar 26 16:13:11 php_wg 17461 /usr/local/pkg/wireguard/includes/wg_service.inc: The command '/sbin/ifconfig 'tun_wg0' -staticarp ' returned exit code '1', the output was 'ifconfig: interface tun_wg0 does not exist' Mar 26 16:13:11 check_reload_status 683 Reloading filter Mar 26 16:13:11 check_reload_status 683 Restarting OpenVPN tunnels/interfaces Mar 26 16:13:11 check_reload_status 683 Restarting IPsec tunnels Mar 26 16:13:11 check_reload_status 683 updating dyndns WAN_DHCP Mar 26 16:13:11 rc.gateway_alarm 35429 >>> Gateway alarm: WAN_DHCP (Addr:135.180.64.1 Alarm:1 RTT:0ms RTTsd:0ms Loss:100%) Mar 26 16:13:09 php_wg 17461 /usr/local/pkg/wireguard/includes/wg_service.inc: Removing static route for monitor 135.180.64.1 and adding a new route through 157.131.248.1 Mar 26 16:13:09 avahi-daemon 95711 Withdrawing workstation service for tun_wg0. Mar 26 16:13:08 kernel tun_wg0: link state changed to DOWN Mar 26 16:13:08 php-cgi 10494 rc.initial.reboot: Stopping all packages. Mar 26 16:13:08 php-cgi 10494 rc.initial.reboot: Suppressing repeat e-mail notification message. Mar 26 16:13:02 login 98737 login on ttyv0 as root Mar 26 16:13:01 login 89132 login on ttyv0 as root Mar 26 16:12:08 php 28934 /usr/local/sbin/acbupload.php: Completed AutoConfigBackup encrypted configuration backup upload to https://acb.netgate.com (success) Mar 26 16:12:05 php 28934 /usr/local/sbin/acbupload.php: Completed AutoConfigBackup encrypted configuration backup upload to https://acb.netgate.com (success) Mar 26 16:12:05 kernel tailscale0: link state changed to DOWN Mar 26 16:12:05 sshd-session 6257 Connection closed by 192.168.90.1 port 33760 Mar 26 16:12:05 avahi-daemon 95711 Withdrawing workstation service for tailscale0. Mar 26 16:12:00 php 28934 /usr/local/sbin/acbupload.php: Starting upload of staged AutoConfigBackup encrypted configuration backups to https://acb.netgate.com REBOOT VIA CONSOLE ^^^ STUCK HERE!!!!!! Mar 26 16:11:58 snort 54382 Shutting down Legacy Blocking Mode custom output plugin, 'alert_pf'. Mar 26 16:11:57 php-cgi 16883 notify_monitor.php: Message sent to XYZ OK Mar 26 16:11:57 kernel igb0: promiscuous mode disabled Mar 26 16:11:57 snort 54382 *** Caught Term-Signal Mar 26 16:11:56 SnortStartup 20909 Snort STOP for WAN snort protect(igb0)... Mar 26 16:11:56 kernel igb2: promiscuous mode disabled Mar 26 16:11:56 kernel igb1: promiscuous mode disabled Mar 26 16:11:56 php-cgi 13723 rc.initial.reboot: Stopping all packages. Mar 26 16:11:51 sshd-session 4955 Accepted keyboard-interactive/pam for admin from 192.168.90.1 port 33760 ssh2 Mar 26 16:11:02 pkg-static 11278 pfSense-pkg-Service_Watchdog-1.8.7_4 deinstalled -
I guess it's not that I can't live without it. I still can use the UI or console. But this does feel like a bug or miss- configuration...
-
How are you running the reboot exactly?
Mar 26 16:12:05 kernel tailscale0: link state changed to DOWN Mar 26 16:12:05 sshd-session 6257 Connection closed by 192.168.90.1 port 33760 Mar 26 16:12:05 avahi-daemon 95711 Withdrawing workstation service for tailscale0. Mar 26 16:12:00 php 28934 /usr/local/sbin/acbupload.php: Starting upload of staged AutoConfigBackup encrypted configuration backups to https://acb.netgate.com REBOOT VIA CONSOLE ^^^ STUCK HERE!!!!!! Mar 26 16:11:58 snort 54382 Shutting down Legacy Blocking Mode custom output plugin, 'alert_pf'. Mar 26 16:11:57 php-cgi 16883 notify_monitor.php: Message sent to XYZ OK Mar 26 16:11:57 kernel igb0: promiscuous mode disabled Mar 26 16:11:57 snort 54382 *** Caught Term-SignalIf it gets stuck there what are you doing to allow it to continue? There's only a 2s gap in the logs....
-
As I said initially
ssh, opt 5, y
Wait a minute
Go check on console, see that it’s stuck and reboot via consoleI guess 2 seconds logs is as pfS does it
-
Ah OK I see. So how are you rebooting it at the console?
-
@stephenw10 said in pfS won't reboot via ssh session:
Ah OK I see. So how are you rebooting it at the console?
Opt 5, yes
-
Hmm, OK.
So does it reboot correctly from the console if you just run there without first trying via SSH?
-
@stephenw10 said in pfS won't reboot via ssh session:
Hmm, OK.
So does it reboot correctly from the console if you just run there without first trying via SSH?
From the console or via the UI/browser it works fine
-
@stephenw10 said in pfS won't reboot via ssh session:
Ah OK I see. So how are you rebooting it at the console?
Any clues?
-
Does it work if you ssh in as root?
-
@stephenw10 said in pfS won't reboot via ssh session:
Does it work if you ssh in as root?
I am root
ssh as adminwhoami root -
shutdown -r nowworks like a charm tho!
It does sound like the pfS thing bug then?
-
Just running reboot doesn't shutdown services, backup temp files etc. Something you have running there is failing to stop correctly.
-
C chudak referenced this topic on
-
@stephenw10 said in pfS won't reboot via ssh session:
Just running reboot doesn't shutdown services, backup temp files etc. Something you have running there is failing to stop correctly.
I removed TS from Watchdog, stopped it and it worked!
Maybe relate to the way I deal with running TS as here https://forum.netgate.com/topic/199968/oauth-credentials-trust-credentials-for-tailscale-connection/35