some services show can't start
-
I am having similar issues on the 2.7.1 concerning wireguard. I am using as a lab environment at home. It is a fresh install in which I restored from a previous backup. The wireguard service does not start and I am unable to manually start it.
I am a bit concerned about updating our company firewalls which are currently running on 23.05.1 since we heavily rely on wireguard tunnels. Quite a few sites are quite far away to drive if we face a major failure.
The log includes the following errors:
Nov 19 17:30:28 php-cgi 467 rc.bootup: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Nov 19 17:30:36 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: The command '/sbin/route -n6 get 'default' 2>/dev/null | /usr/bin/egrep 'flags: <.*PROTO.*>'' returned exit code '1', the output was '' Nov 19 17:30:36 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary. Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard). Nov 19 17:30:35 check_reload_status 432 Syncing firewall Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard). Nov 19 17:30:35 check_reload_status 432 Syncing firewall Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s). Nov 19 17:30:35 kernel tun_wg0: link state changed to UP Nov 19 17:30:35 kernel wg1: changing name to 'tun_wg0' Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary. Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s). Nov 19 17:30:35 kernel wg0: changing name to 'tun_wg1' Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary. Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s). Nov 19 17:30:35 php_wg 82338 /usr/local/pkg/wireguard/includes/wg_service.inc: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s). Nov 19 17:30:35 kernel tun_wg0: link state changed to DOWN Nov 19 17:30:34 kernel tun_wg1: link state changed to DOWN Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary. Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard). Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard). Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard). Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard). Nov 19 17:30:34 check_reload_status 432 Syncing firewall Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s). Nov 19 17:30:34 check_reload_status 432 Syncing firewall Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s). Nov 19 17:30:34 php-fpm 18459 /rc.start_packages: Restarting/Starting all packages. Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Applied package default settings as necessary. Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed Unbound ACL group (WireGuard). Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed Unbound ACL group (WireGuard). Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed interface group (WireGuard). Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed interface group (WireGuard). Nov 19 17:30:43 check_reload_status 432 Syncing firewall Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] Installed earlyshellcmd(s). Nov 19 17:30:43 check_reload_status 432 Syncing firewall Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Configuration Change: (system): [pfSense-pkg-WireGuard] De-installed earlyshellcmd(s). Nov 19 17:30:43 php-fpm 86806 /rc.start_packages: Restarting/Starting all packages. Nov 19 17:30:42 syslogd kernel boot file is /boot/kernel/kernel Nov 19 17:30:42 syslogd exiting on signal 15 Nov 19 17:30:42 kernel done.
Interesting is that everything runs fine as soon as I dis- and re-enable one of the wireguard interfaces. Maybe that is some kind of help?
-
Hmm, the only thing that actually looks like an error there is that route command and that could be because it's run at boot.
What new errors do you see if you try to start the service after boot?
What if you manually run:
/sbin/route -n6 get 'default'
Steve
-
@stephenw10
First of all thanks for your almost instant reply :-)@stephenw10 said in some services show can't start:
What if you manually run: /sbin/route -n6 get 'default'
route: route has not been found: No error: 0
We do not have a default gateway IPv6 set though - we solely use IPv4 for our uplinks.
@stephenw10 said in some services show can't start:
What new errors do you see if you try to start the service after boot?
Actually none in the logs at least and none in the gui as well. Do you want me to check out a certain log?Interesting enough is that wireguard is actually working and I can ping the wg host on the other side. Pfsense just doesn't bring up the gateway for the static routes since the wireguard service is down.
[2.7.1-RELEASE][admin@labgate]/root: ps aux | grep php_wg root 63787 0.0 0.1 12752 2388 0 S+ 21:07 0:00.00 grep php_wg [2.7.1-RELEASE][admin@labgate]/root:
-
Update on the wirguard issue:
I am pretty sure I hit bug #14613 (https://redmine.pfsense.org/issues/14613).
The wireguard gui part is not responsive and cannot be started.
[2.7.1-RELEASE][admin@labgate]/root: fstat /var/run/wireguardd.pid USER CMD PID FD MOUNT INUM MODE SZ|DV R/W NAME root dpinger 59313 10 /var/run 57 -rw-r--r-- 5 rw /var/run/wireguardd.pid root dpinger 58892 10 /var/run 57 -rw-r--r-- 5 rw /var/run/wireguardd.pid root dpinger 58557 10 /var/run 57 -rw-r--r-- 5 rw /var/run/wireguardd.pid
As soon as I do a
[2.7.1-RELEASE][admin@labgate]/root: /usr/local/bin/php_wg -f /usr/local/pkg/wireguard/includes/wg_service.inc stop
I can start the wireguard gui through the webinterface and php_wg is running:
[2.7.1-RELEASE][admin@labgate]/root: ps aux | grep php_wg root 61681 0.0 2.3 68272 46144 - Ss 21:47 0:00.06 php_wg: WireGuard service (php_wg) root 23988 0.0 0.1 12752 2392 0 S+ 21:51 0:00.00 grep php_wg
-
Ah, interesting. But you only saw this after upgrading to 2.7.1?
This should probably be in a new thread.
-
@stephenw10
Actually it is a fresh install of 2.7.0 and I restored from a backup. Since I immediately updated to 2.7.1 I cannot really tell. Everything worked until the first reboot… -
Ah, well it would be interesting to test that if you can. If this is a new issue in 2.7.1 it's probably not the same as the one you linked to.
-
@stephenw10
I will give it a try and reinstall the machine. Maybe its the same bug in 2.7.0 and 2.7.1… -
It probably is. In which case we have somewhere to start digging and more data points to pin it down. Otherwise if it's a regression in 2.7.1 we need to know about it.
-
@stephenw10
I just checked the 2.7.0 as well. The same error occurs after it's first restart. php-wg is offline but wireguard itself seems to be working. -
Ok so it likely is that same issue. You should add any notes you have there.
-
@stephenw10
Unfortunately I don't have much more to offer. If you want to have a look at the machine just pm me ;-) -
pfSense-repoc
pfSense-repoc: failed to fetch the repo data
failed to read the repo data.
pkg-static update
Updating pfSense-core repository catalogue...
pkg-static: No SRV record found for the repo 'pfSense-core'
pkg-static: An error occured while fetching package
pkg-static: packagesite URL error for pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_09_amd64-core/meta.txz -- pkg+:// implies SRV mirror type
repository pfSense-core has no meta file, using default settings
pkg-static: packagesite URL error for pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_09_amd64-core/packagesite.pkg -- pkg+:// implies SRV mirror type
pkg-static: packagesite URL error for pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_09_amd64-core/packagesite.txz -- pkg+:// implies SRV mirror type
Unable to update repository pfSense-core
Updating pfSense repository catalogue...
pkg-static: No SRV record found for the repo 'pfSense'
pkg-static: An error occured while fetching package
pkg-static: packagesite URL error for pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_09_amd64-pfSense_plus_v23_09/meta.txz -- pkg+:// implies SRV mirror type
repository pfSense has no meta file, using default settings
pkg-static: packagesite URL error for pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_09_amd64-pfSense_plus_v23_09/packagesite.pkg -- pkg+:// implies SRV mirror type
pkg-static: packagesite URL error for pkg+https://pfsense-plus-pkg.netgate.com/pfSense_plus-v23_09_amd64-pfSense_plus_v23_09/packagesite.txz -- pkg+:// implies SRV mirror type
Unable to update repository pfSense
Error updating repositories! -
Crash report begins. Anonymous machine information:
amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #0 plus-RELENG_23_09-n256163-2763857e770: Wed Nov 1 21:18:24 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/obj/amd64/WrS3lKLo/var/jenkins/workspace/pfSense-Plus-snapshots-23_09-main/Crash report details:
PHP Errors:
[12-Dec-2023 20:23:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:23:01 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:23:08 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:23:34 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:24:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:24:36 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:25:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:25:05 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:25:38 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:26:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:26:40 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:27:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:27:42 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:28:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:28:44 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:29:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:29:46 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:30:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:30:49 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:31:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:31:51 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:32:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:32:53 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:33:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:33:55 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:34:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:34:58 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:35:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:36:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0
[12-Dec-2023 20:36:00 Asia/Shanghai] PHP Fatal error: Unable to start pfSense module in Unknown on line 0No FreeBSD crash data found.
-
@yon-0 said in some services show can't start:
Updating pfSense repository catalogue...
pkg-static: No SRV record found for the repo 'pfSense'
pkg-static: An error occured while fetching packageThis is a DNS error. Check pfSense can resolve.