PHP error after reboot

[magoo_it]

Hello all,
I'm getting this error after a reboot and somehow I cant figure out why.

Hope someone can shine some light on this or point a way of analysis.

Regards.
JG

Log:

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #1 RELENG_2_7_0-n255866-686c8d3c1f0: Wed Jun 28 04:21:19 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/obj/amd64/LwYAddCr/var/jenkins/workspace/pfSense-CE-snapshots-2_7_0-main/sources/FreeBSD-src-REL

Crash report details:

PHP Errors:
[31-Aug-2023 20:45:10 Europe/Lisbon] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.gui.inc(53): parse_config()
#3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...')
#4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...')
#5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...')
#6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...')
#7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....')
#8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#10 /etc/inc/config.inc(51): require_once('/etc/inc/notice...')
#11 /usr/local/bin/telegraf_pfifgw.php(3): require_once('/etc/inc/config...')
#12 {main}
thrown in /etc/inc/config.lib.inc on line 172
[31-Aug-2023 20:45:10 Europe/Lisbon] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101
[31-Aug-2023 20:59:30 Europe/Lisbon] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.gui.inc(53): parse_config()
#3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...')
#4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...')
#5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...')
#6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...')
#7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....')
#8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#10 /etc/inc/config.inc(51): require_once('/etc/inc/notice...')
#11 /usr/local/bin/telegraf_pfifgw.php(3): require_once('/etc/inc/config...')
#12 {main}
thrown in /etc/inc/config.lib.inc on line 172
[31-Aug-2023 20:59:30 Europe/Lisbon] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1148): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101
[31-Aug-2023 21:03:00 Europe/Lisbon] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:42:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /usr/local/bin/telegraf_pfifgw.php(3): require_once('/etc/inc/config...')
#4 {main}
thrown in /etc/inc/config.lib.inc on line 172

No FreeBSD crash data found.

[Gertjan]

@magoo_it

You are using UFS or ZFS as a file system ?

pfSense (PHP) has a hard time writing a file ... Try this :
How to Run a pfSense Software File System Check (5/2020)

Also check available disk space, as 'disk full' would explain this situation also.

[magoo_it]

@Gertjan Thank you for your quick reply.

I'm running ZFS as it defaults for 2.7 installation.

The system has 200 GB of disk space unless it uses so sort of partitioning that I'm not yet aware.

This is running on an SSD 256 disk 8 GB RAM ..

I'll do the FS check ASAP

I'll let you know later.

Regards.
JG

-- df-h reports all ok ..
Filesystem Size Used Avail Capacity Mounted on
pfSense/ROOT/default 206G 1.0G 205G 0% /
devfs 1.0K 1.0K 0B 100% /dev
pfSense/var 205G 3.5M 205G 0% /var
pfSense/tmp 205G 660K 205G 0% /tmp
pfSense/home 205G 236K 205G 0% /home
pfSense 205G 96K 205G 0% /pfSense
pfSense/var/db 205G 44M 205G 0% /var/db
pfSense/var/log 205G 3.7M 205G 0% /var/log
pfSense/var/empty 205G 96K 205G 0% /var/empty
pfSense/var/cache 205G 96K 205G 0% /var/cache
pfSense/var/tmp 205G 136K 205G 0% /var/tmp
pfSense/reservation 227G 96K 227G 0% /pfSense/reservation
pfSense/ROOT/default/cf 205G 5.4M 205G 0% /cf
pfSense/ROOT/default/var_cache_pkg 205G 192M 205G 0% /var/cache/pkg
pfSense/ROOT/default/var_db_pkg 205G 3.6M 205G 0% /var/db/pkg
tmpfs 4.0M 152K 3.9M 4% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev

[Gertjan]

@magoo_it

Not sure then ...

	$configcache = fopen(g_get('tmp_path') . '/config.cache', "w");
	fwrite($configcache, serialize($config));
	fclose($configcache);

It tries to open a file for writing in the /tmp/ folder.
It want to create this file /tmp/config.cache

Can you SSH - or console - into your device, see if that file is there.
What size ?
What owner rights ?

[23.05.1-RELEASE][root@pfSense.bhf.net]/tmp: ls -al config.cache
-rw-r--r--  1 root  wheel  557852 Sep  4 09:50 config.cache

Can you create file in that folder ?
Just take a big one, and copy it like

cp config.cache config.old

[magoo_it]

@Gertjan said in PHP error after reboot:

cp config.cache config.old

All seams ok, and I can copy the file in place, should I delete it and reboot ???
-rw-r--r-- 1 root wheel 329076 Sep 4 09:04 config.cache
-rw-rw-rw- 1 root wheel 0 Sep 4 09:08 config.lock
-rw-r--r-- 1 root wheel 329076 Sep 4 09:07 config.old

[magoo_it]

Ok, I deleted the file and now I see something different:

-rw-r--r-- 1 telegraf wheel 329076 Sep 4 09:11 config.cache

The owner has changed from root to the telegraf user. I'm using telegraf for metrics with a v2 influx DB so I have to tweak the config file to be able to connect to the v2 DB..

Every time the system reboot I have to copy it over the default one.
I wish we could have some form of using influx v2 on the GUI..

[Gertjan]

@magoo_it said in PHP error after reboot:

The owner has changed from root to the telegraf user.

Yeah, that's not good at all.
Although, thinking out load : a root process should be able to over write a file owned by 'telegraf'.

When you ditch / disable the telegraf package, things behave normally ?

[stephenw10]

Looks like this: https://redmine.pfsense.org/issues/14432

But if you're doing something custom there it could be that.

Steve

[magoo_it]

@Gertjan
Actually, I think when the owner is telegraf it seems ok.

I disabled the Telegraf service and removed the file. The file was not created until the watchdog service detected the service down and restarted it.
The file was then created with the root owner, but so far no PHP error thrown.

This is weird but it seems in fact an issue with the owner of the file, as you said, root should overcome the telegraf user.

I've stopped the service, removed the file, watchdog starts the service, and now it's always created with the root user ...

Well, no further investigation as we can conclude this is related to the telegaf service and the root user, and the creation of that file, and I'll keep an eye on It.

Thank you for your insights and support.

Best regards.
JG

[magoo_it]

@stephenw10 said in PHP error after reboot:

Looks like this: https://redmine.pfsense.org/issues/14432

But if you're doing something custom there it could be that.

Steve

Actually, the only custom is the addition of the influx db v2 part inside the telegraf.conf file.

I guess I can change the config of the influxDB to accept it but on the other hand why? why can't we use the either version v1 or v2 influx DB connections?

But that's not the point of this thread.

And as said above I thank you all for your insights and help, I now understand a bit more thanks to you.

Best regards.
JG

[stephenw10]

Well there's a patch on that bug you can apply to correct it if that's the issue.

[magoo_it]

@stephenw10

Of course, no point in asking for help/support/advice and acting if all is normal.
I do want to see if the patch solves the issue, even if I kinda of concluded that I'm able to manage the issue.
So if you please can you direct/provide the link/access to the patch?

Much appreciated.
Regards,
Jg

[stephenw10]

You can just add the commit hash to the system patches package directly:

Then fetch and apply it.

Steve

[magoo_it]

@stephenw10 well, I think this is for the non-ce version of PFsense ..
Since I cannot find, or see, that SYSTEM->Patches option.

[rcoleman-netgate]

@magoo_it It's an add-on package for CE and Plus: https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

[magoo_it]

@rcoleman-netgate said in PHP error after reboot:

@magoo_it It's an add-on package for CE and Plus: https://docs.netgate.com/pfsense/en/latest/development/system-patches.html

And you beat it to me ... I just opened the thread tom come here to inform you the a simple duck search just told me so ...
Nevertheless, I thank you so much.

It seems that today is not my best day on IT ... well better days will come.

Regards.
JG

[rcoleman-netgate]

@magoo_it said in PHP error after reboot:

It seems that today is not my best day on IT ... well better days will come.

aww, don't be so hard on yourself. I have to keep a sticky on the Reddit group about this tool because it seems very few people there know about it, either.