Upgrade to 23.01 - crash report

stephenw10

You should also see /cf/conf/upgrade_log.latest.txt
That should have the full upgrade log if you just upgraded

pfguy2018

The file is almost 4000 lines long. What string should I be searching for?

stephenw10

Something like the errors you initially showed. So anything including Stack trace for example.

pfguy2018

@stephenw10
Nothing seems to be coming up using terms like phperror, stack, stack trace, /etc/inc/config.lib.inc, /etc/inc/config.lib, config.lib.

When I search for error, I get the following list

~/Documents/backup/pfSense backups/upgrade_log.latest.txt:86:   - php81-intl-8.1.11_1 [pfSense] conflicts with php74-intl-7.4.28 [installed] on /usr/local/include/php/ext/intl/intl_error.h
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:257: 	libgpg-error: 1.44 -> 1.45 [pfSense]
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:387: [6/261] Upgrading libgpg-error from 1.44 to 1.45...
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:388: [6/261] Extracting libgpg-error-1.45: .......... done
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:589: openldap24-client-2.4.59_4: missing file /usr/local/man/man3/ldap_error.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:699: openldap24-client-2.4.59_4: missing file /usr/local/man/man3/ldap_perror.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:707: openldap24-client-2.4.59_4: missing file /usr/local/man/man3/ldap_result2error.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:3139: nut-2.7.4_20: missing file /usr/local/man/man3/upscli_strerror.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:3140: nut-2.7.4_20: missing file /usr/local/man/man3/upscli_upserror.3.gz
~/Documents/backup/pfSense backups/upgrade_log.latest.txt:3234: libmaxminddb-1.6.0: missing file /usr/local/man/man3/MMDB_strerror.3.gz

Any clues there?

stephenw10

Nope that's all expected output at upgrade.

If you're not seeing that error return after removing the report it was likely a conflict that was only present during the upgrade because of the php change and not an issue.

Steve

pfguy2018

@stephenw10
That makes sense - thanks for your help!

madas

@stephenw10

Same error for me. I've cleared it 3 times since the upgrade and it comes back

Crash report begins. Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT #0 plus-RELENG_23_01-n256037-6e914874a5e: Fri Feb 10 20:30:29 UTC 2023 root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/obj/amd64/VDZvZksF/var/jenkins/workspace/pfSense-Plus-snapshots-23_01-main/sources/FreeBS

Crash report details:

PHP Errors:
[28-Feb-2023 02:02:51 EST5EDT] PHP Fatal error: Uncaught TypeError: fwrite(): Argument #1 ($stream) must be of type resource, bool given in /etc/inc/config.lib.inc:172
Stack trace:
#0 /etc/inc/config.lib.inc(172): fwrite(false, 'a:41:{s:7:"vers...')
#1 /etc/inc/config.lib.inc(147): generate_config_cache(Array)
#2 /etc/inc/config.inc(141): parse_config()
#3 /etc/inc/gwlb.inc(25): require_once('/etc/inc/config...')
#4 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#5 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#6 /usr/local/pkg/nut/nut_email.php(24): require_once('/etc/inc/notice...')
#7 {main}
thrown in /etc/inc/config.lib.inc on line 172
[28-Feb-2023 02:02:51 EST5EDT] PHP Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:101
Stack trace:
#0 /etc/inc/notices.inc(101): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1162): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
thrown in /etc/inc/notices.inc on line 101

No FreeBSD crash data found.

stephenw10

Have you rebooted since the upgrade?

Do those errors only appear after rebooting?

If not what triggers the errors?

Steve

madas

@stephenw10

No - not on a reboot. I have rebooted it 2-3 times since the upgrade and it doesn't occur on reboot.

This happened at 2am so I have no clue what triggered it

Brad Bishop

I'm getting similar errors:

PHP ERROR: Type: 1, File: /usr/local/www/xmlrpc.php(147) : eval()'d code, Line: 14, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/www/xmlrpc.php(147) : eval()'d code:14
Stack trace:
#0 /usr/local/www/xmlrpc.php(147): eval()
#1 /usr/local/share/pear/XML/RPC2/Server/CallHandler/Instance.php(141): pfsense_xmlrpc_server->exec_php('\nini_set('displ...')
#2 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(135): XML_RPC2_Server_Callhandler_Instance->__call('pfsense.exec_ph...', Array)
#3 /usr/local/share/pear/XML/RPC2/Backend/Php/Server.php(99): XML_RPC2_Backend_Php_Server->getResponse()
#4 /usr/local/www/xmlrpc.php(988): XML_RPC2_Backend_Php_Server->handleCall()
#5 {main}
thrown @ 2023-02-28 15:23:50

I have rebooted. This seems to be happening throughout the day.

madas

@stephenw10

Not sure if its related but i have up/down notifications on my gateways and they are now getting stuck in a loop and i get the notifications over and over again.

Notifications in this message: 3

10:28:59 MONITOR: VPN_TOR_VPNV4 is available now, adding to routing group VPN_NORMAL
9.9.9.9|1x|VPN_TOR_VPNV4|98.764ms|3.098ms|10%|online|none
10:29:02 DynDNS updated IP Address on VPN_TOR (ovpnc9) to xx
10:34:40 MONITOR: VPN_TOR_VPNV4 has packet loss, omitting from routing group VPN_NORMAL
9.9.9.9|1x|VPN_TOR_VPNV4|99.52ms|3.805ms|57%|down|highloss

Its not really down. I received 1000 of these in the last 5 minutes.

stephenw10

@brad-bishop said in Upgrade to 23.01 - crash report:

I'm getting similar errors

That's a different error, you should open a new thread.

Though IIRC there is already a patch for it. See: https://redmine.pfsense.org/issues/14000

Try the patch shown here: https://redmine.pfsense.org/issues/13944

stephenw10

@madas said in Upgrade to 23.01 - crash report:

This happened at 2am so I have no clue what triggered it

Presumably something that triggered a notification. What's in the logs at that time?

Steve

madas

@stephenw10

Sorry - i think i'm confusing issues.

I see your last post about the home assistant integration - i think that was for brad only (i'm not using HA).

I don't see any fix for my PHP issue?

Separately - i'm getting this notification loop and I actually managed to catch pfsense stuck in a loop bringing the gateway on the VPN up and down over and over again. the VPN is an openvpn client and I actually shutdown openvpn completely and PFSense was still stuck in a loop bringing it up/down. Maybe I should open a new thread for this one?

stephenw10

Yes, that link to the fix was for Brad. He's seeing a different bug entirely.

Yes you should open a new thread for a looping VPN connection. That appears to be triggering the notifications but it's actually notifying correctly for the incorrect connection attempts.

Do you still see that error if you don't have email notifications enabled in nut?
I note OP here was seeing it without nut.

madas

@stephenw10

I just disabled NUT as I did see that losing connection to the UPS over time. So maybe thats it?

I can monitor to see if it still occurs now NUT is disabled

NUT worked fine with the UPS prior to 23.01

stephenw10

Yes, it should still work. There may be a bug you've hit. Let's see what that changes.

madas

@stephenw10 do you want me to shutdown NUT completely or just disable the email notification?

stephenw10

Just disable the notification.

Brad Bishop

@stephenw10

Sorry I didn't start a new thread. This worked. My fault / my side.

I appreciate the help.

Have a great day!