RC 2.8.1 error in routing table of source site after system update on another site

reberhar

So I updated a second site from 2.8.0 to RC 2.8.1 via openvpn and after that I was unable to access the target, second site, over OpenVPN. However the target machine did update successfully.

All machines were running the RC except the target which was running 2.8.0.

After some seaching around I discovered that the routing table on the source machine through which I was working, and not the one being updated, was missing some routing table entries for other sites normally connected through openvpn, including the target site. I have HA so I switched to the backup of the source pair and that worked, neither was was the backup machine missing the said entries in the routing table.

Numbers include
10.0.0.1 target table that was updated and later no longer accessible.
10.1.0.1 missing from the routing table as was 10.0.0.1
10.5.0.1 the local source machine was present.
10.56.0.1 is the OpenVPN number.
10.3.0.1 is the head pfSense machine that all machines connect to. It has four other pairs connecting to it including 10.0.0.1, 10.1.0.1, and 10.5.0.1, the machine with the fault.
From machines connected to the 10.3.0.1, I could access all the other machines.

I tried to just clear the state table on the primary but that did not work.

Rebooting did not fix the routing table. Neither did restarting the OpenVPN instances. Restoring from auto config backup has not helped, although that did seem to repair the routing table, but access via OpenVPN has not returned. The problem is just with the OpenVPN instances. Everything else seems to be fine. Secondary HA machine continues to work properly. I am wondering about the install being corrupted.

I will power the machine off and try a cold restart maybe tomorrow.

Running on secondary server ...

See traceroutes at the end.

The same error message mentioned below is generated each time I reboot. Rebooting seems to be failing as can be seen in the copy of the console reboot farther below.

On reboot of source the primary source machine the following error was generated.

[27-Aug-2025 21:06:48 America/Phoenix] PHP Fatal error: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(67): system_reboot_sync(true)
#5 {main}
thrown in /etc/inc/util.inc on line 142
[27-Aug-2025 21:07:00 America/Phoenix] PHP Fatal error: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(67): system_reboot_sync(true)
#5 {main}
thrown in /etc/inc/util.inc on line 142

After a reboot after restoring from autobackup

Crash report begins. Anonymous machine information:

amd64
15.0-CURRENT
FreeBSD 15.0-CURRENT #20 RELENG_2_8_1-n256095-47c932dcc0e9: Fri Aug 8 19:27:12 UTC 2025 root@pfsense-build-release-amd64-1.eng.atx.netgate.com:/var/jenkins/workspace/pfSense-CE-snapshots-2_8_1-main/obj/amd64/AupY3aTL/var/jenkins/workspace/pfSense-CE-

Crash report details:

PHP Errors:
[27-Aug-2025 21:06:48 America/Phoenix] PHP Fatal error: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(67): system_reboot_sync(true)
#5 {main}
thrown in /etc/inc/util.inc on line 142
[27-Aug-2025 21:07:00 America/Phoenix] PHP Fatal error: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(67): system_reboot_sync(true)
#5 {main}
thrown in /etc/inc/util.inc on line 142
[27-Aug-2025 21:43:17 America/Phoenix] PHP Fatal error: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3023): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(71): system_reboot()
#5 {main}
thrown in /etc/inc/util.inc on line 142
[27-Aug-2025 21:43:26 America/Phoenix] PHP Fatal error: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3023): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(71): system_reboot()
#5 {main}
thrown in /etc/inc/util.inc on line 142

No FreeBSD crash data found.

From errors posted on dashboard

Notices
PHP {$errortype}s
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 142, Message: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(67): system_reboot_sync(true)
#5 {main}
thrown @ 2025-08-27 21:06:48
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 142, Message: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(67): system_reboot_sync(true)
#5 {main}
thrown @ 2025-08-27 21:07:00
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 142, Message: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3023): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(71): system_reboot()
#5 {main}
thrown @ 2025-08-27 21:43:17
PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 142, Message: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3023): system_reboot_cleanup()
#4 /usr/local/www/diag_reboot.php(71): system_reboot()
#5 {main}
thrown @ 2025-08-27 21:43:26

trace route to target machine from primary source that does not work.

~$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
1 _gateway.lxd (100.115.92.193) 0.326 ms 0.050 ms 0.045 ms
2 100.115.92.25 (100.115.92.25) 1.188 ms 1.026 ms 0.697 ms
3 * * *
4 * * *
5 * * *

trace route to target machine from secondary source that works.

~$ traceroute 10.0.0.1
traceroute to 10.0.0.1 (10.0.0.1), 30 hops max, 60 byte packets
1 _gateway.lxd (100.115.92.193) 0.074 ms 0.015 ms 0.017 ms
2 100.115.92.25 (100.115.92.25) 6.600 ms 6.543 ms 6.515 ms
3 MountainVistaSecondary.Lan (10.5.0.2) 9.582 ms 9.558 ms 9.532 ms
4 10.56.0.1 (10.56.0.1) 133.615 ms 133.587 ms 140.940 ms
5 * * *
6 * * *
7 * * *
8 * * *
9 * * *

A reboot from the command line

Reboot system 14) Disable Secure Shell (sshd)
Halt system 15) Restore recent configuration
Ping host 16) Restart PHP-FPM
Shell

Enter an option: 5

pfSense will reboot. This may take a few minutes, depending on your hardware.
Do you want to proceed?

Y/y: Reboot normally
R/r: Reroot (Stop processes, remount disks, re-run startup sequence)
S: Reboot into Single User Mode (requires console access!)

Enter an option: y

pfSense is rebooting now.
Stopping package suricata...done.
Stopping package ntopng...done.
Stopping package apcupsd...done.
<pre style="white-space: pre-wrap;">PHP ERROR: Type: 1, File: /etc/inc/util.inc, Line: 142, Message: Uncaught TypeError: is_process_running(): Argument #1 ($name) must be of type string, null given, called in /etc/inc/service-utils.inc on line 290 and defined in /etc/inc/util.inc:142
Stack trace:
#0 /etc/inc/service-utils.inc(290): is_process_running(NULL)
#1 /etc/inc/pkg-utils.inc(1258): is_service_running(NULL)
#2 /etc/inc/system.inc(3067): stop_packages()
#3 /etc/inc/system.inc(3035): system_reboot_cleanup()
#4 /etc/rc.initial.reboot(78): system_reboot_sync()
#5 {main}
thrown</pre>pfSense - Netgate Device ID: a870069a615145dbfe12

*** Welcome to pfSense 2.8.1-RC (amd64) on MountainVistaPrimary ***

This morning I can again access all machines and the routing tables look the same. There is obviously something I am not understanding here. I did a power off and on to get the primary machine to reboot.

Reboot through the interface still invokes the same error and the machine does not reboot. I am contemplating going back to 2.8.0, but, except for these oddities everything seems to be working very well, quite fast really.