Upgrade to 23.01 - crash report

pfguy2018

Just upgraded to 23.01. Everything seems to work except there is a message at the main screen indicating there was a crash, with a link that takes me to the following report:

Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[15-Feb-2023 19:20:24 America/Toronto] PHP Fatal error:  Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:46:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/bin/notify_monitor.php(24): include_once('/etc/inc/notice...')
#7 {main}
  thrown in /etc/inc/config.lib.inc on line 172
[15-Feb-2023 19:20:24 America/Toronto] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1162): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
  thrown in /etc/inc/notices.inc on line 101



No FreeBSD crash data found.

How do I fix this? (or do I need to?)

stephenw10

That looks like the errors you see if the drive is full. Though normally it would not boot normally in that case.

Check the disk usage on the dashboard.

If it's high and you are running ZFS check the BE snapshots and remove some.

Steve

pfguy2018

@stephenw10

Disk usage is showing as 1%
I am not sure how to check or remove BE snapshots - where do I do that?

stephenw10

If it's at 1% that's not the problem. Hard to say what caused that then.

You can check the upgrade log. There will be a very large number of errors there but that's expected due to the php change. If you see pkg install scripts failing that could be an issue though.

Steve

pfguy2018

@stephenw10

All I see in /cf/conf/upgrade_log.txt
is

>>> Setting vital flag on php81... done.
Your system is up to date

Should I be checking somewhere else instead?

Edit - I found the log at /conf/upgrade_log.latest.txt - much more detailed - checking that now

stephenw10

You should also see /cf/conf/upgrade_log.latest.txt
That should have the full upgrade log if you just upgraded

pfguy2018

@stephenw10

The file is almost 4000 lines long. What string should I be searching for?

stephenw10

Something like the errors you initially showed. So anything including Stack trace for example.

pfguy2018

@stephenw10
Nothing seems to be coming up using terms like phperror, stack, stack trace, /etc/inc/config.lib.inc, /etc/inc/config.lib, config.lib.

When I search for error, I get the following list

~/Documents/backup/pfSense backups/upgrade_log.latest.txt:86:   - php81-intl-8.1.11_1 [pfSense] conflicts with php74-intl-7.4.28 [installed] on /usr/local/include/php/ext/intl/intl_error.h
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:257: 	libgpg-error: 1.44 -> 1.45 [pfSense]
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:387: [6/261] Upgrading libgpg-error from 1.44 to 1.45...
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:388: [6/261] Extracting libgpg-error-1.45: .......... done
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:589: openldap24-client-2.4.59_4: missing file /usr/local/man/man3/ldap_error.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:699: openldap24-client-2.4.59_4: missing file /usr/local/man/man3/ldap_perror.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:707: openldap24-client-2.4.59_4: missing file /usr/local/man/man3/ldap_result2error.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:3139: nut-2.7.4_20: missing file /usr/local/man/man3/upscli_strerror.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:3140: nut-2.7.4_20: missing file /usr/local/man/man3/upscli_upserror.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:3234: libmaxminddb-1.6.0: missing file /usr/local/man/man3/MMDB_strerror.3.gz

Any clues there?

stephenw10

Nope that's all expected output at upgrade.

If you're not seeing that error return after removing the report it was likely a conflict that was only present during the upgrade because of the php change and not an issue.

Steve

pfguy2018

@stephenw10
That makes sense - thanks for your help!

madas

@stephenw10

Same error for me. I've cleared it 3 times since the upgrade and it comes back

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[28-Feb-2023 02:02:51 EST5EDT] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:41:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[28-Feb-2023 02:02:51 EST5EDT] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1162): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101

No FreeBSD crash data found.

stephenw10

Have you rebooted since the upgrade?

Do those errors only appear after rebooting?

If not what triggers the errors?

Steve

madas

@stephenw10

No - not on a reboot. I have rebooted it 2-3 times since the upgrade and it doesn't occur on reboot.

This happened at 2am so I have no clue what triggered it

Brad Bishop

I'm getting similar errors:

PHP ERROR: Type: 1, File: /usr/local/www/xmlrpc.php(147) : eval()'d code, Line: 14, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php(147) : eval()'d code:14
Stack trace:
#0 /usr/local/www/xmlrpc.php(147): eval()
#1 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
#3 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#4 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#5 {main}
thrown @ 2023-02-28 15:23:50

I have rebooted. This seems to be happening throughout the day.

madas

@stephenw10

Not sure if its related but i have up/down notifications on my gateways and they are now getting stuck in a loop and i get the notifications over and over again.

Notifications in this message: 3

10:28:59 MONITOR: VPN_TOR_VPNV4 is available now, adding to routing group VPN_NORMAL
9.9.9.9|1x|VPN_TOR_VPNV4|98.764ms|3.098ms|10%|online|none
10:29:02 DynDNS updated IP Address on VPN_TOR (ovpnc9) to xx
10:34:40 MONITOR: VPN_TOR_VPNV4 has packet loss, omitting from routing group VPN_NORMAL
9.9.9.9|1x|VPN_TOR_VPNV4|99.52ms|3.805ms|57%|down|highloss

Its not really down. I received 1000 of these in the last 5 minutes.

stephenw10

@brad-bishop said in Upgrade to 23.01 - crash report:

I'm getting similar errors

That's a different error, you should open a new thread.

Though IIRC there is already a patch for it. See: https://redmine.pfsense.org/issues/14000

Try the patch shown here: https://redmine.pfsense.org/issues/13944

stephenw10

@madas said in Upgrade to 23.01 - crash report:

This happened at 2am so I have no clue what triggered it

Presumably something that triggered a notification. What's in the logs at that time?

Steve

madas

@stephenw10

Sorry - i think i'm confusing issues.

I see your last post about the home assistant integration - i think that was for brad only (i'm not using HA).

I don't see any fix for my PHP issue?

Separately - i'm getting this notification loop and I actually managed to catch pfsense stuck in a loop bringing the gateway on the VPN up and down over and over again. the VPN is an openvpn client and I actually shutdown openvpn completely and PFSense was still stuck in a loop bringing it up/down. Maybe I should open a new thread for this one?

stephenw10

Yes, that link to the fix was for Brad. He's seeing a different bug entirely.

Yes you should open a new thread for a looping VPN connection. That appears to be triggering the notifications but it's actually notifying correctly for the incorrect connection attempts.

Do you still see that error if you don't have email notifications enabled in nut?
I note OP here was seeing it without nut.