• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

Random crash report in notices.inc and wg.inc

Scheduled Pinned Locked Moved General pfSense Questions
30 Posts 3 Posters 899 Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S
    stephenw10 Netgate Administrator
    last edited by stephenw10 May 2, 2025, 2:48 PM May 2, 2025, 2:47 PM

    It's a setting in System > Advanced > Misc. You would see /tmp and/or /var mounted as tmpfs:

    [25.03-BETA][admin@6100.stevew.lan]/root: mount -p
pfSense2/ROOT/default-2_20250221111341 /			zfs	rw,noatime,nfsv4acls 	0 0
devfs			/dev			devfs	rw		0 0
/dev/gpt/efiboot0	/boot/efi		msdosfs	rw		2 2
pfSense2/home		/home			zfs	rw,noatime,nfsv4acls 	0 0
pfSense2		/pfSense2		zfs	rw,noatime,nfsv4acls 	0 0
tmpfs			/tmp			tmpfs	rw		0 0
tmpfs			/var			tmpfs	rw		0 0
pfSense2/ROOT/default-2_20250221111341/cf /cf			zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense2/ROOT/default-2_20250221111341/var_cache_pkg /var/cache/pkg		zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
pfSense2/ROOT/default-2_20250221111341/var_db_pkg /var/db/pkg		zfs	rw,noexec,nosuid,noatime,nfsv4acls 	0 0
devfs			/var/dhcpd/dev		devfs	rw		0 0

    Your filesystem looks fine there.

    W 1 Reply Last reply May 2, 2025, 2:53 PM Reply Quote 0
    • W
      whinis @stephenw10
      last edited by May 2, 2025, 2:53 PM

      Ya, In System-> Advanced -> Misc Ram discs is unchecked.

      1 Reply Last reply Reply Quote 1
      • W
        whinis @stephenw10
        last edited by May 2, 2025, 9:46 PM

        @stephenw10
        So I had a chance to do this but was unsuccessful for many reasons
        The GUI reboot method only has Normal and Reroot but not file system check as mentioned in the article

        Similarly attempting to reboot into file system check from console only gives me reroot and reboot, if I enter F anyways I go back to the main menu

        I finally attempted single user mode but whenever I ran fsck -fy / I got an error about pfSense2/ROOT/default-2_20250221111341 not existing.

        Any other test?

        1 Reply Last reply Reply Quote 0
        • S
          stephenw10 Netgate Administrator
          last edited by May 3, 2025, 12:02 AM

          Because those are for UFS intals and you are running ZFS. Sorry I should have been clearer there.

          You just run zpool scrub pfSense to check that in ZFS. Then run zpool status to see the result.

          W 1 Reply Last reply May 3, 2025, 12:37 AM Reply Quote 0
          • W
            whinis @stephenw10
            last edited by May 3, 2025, 12:37 AM 

              pool: pfSense
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
        The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub in progress since Fri May  2 20:36:28 2025
        1.32G / 1.32G scanned, 1.24G / 1.32G issued at 212M/s
        0B repaired, 94.06% done, 00:00:00 to go
config:

        NAME        STATE     READ WRITE CKSUM
        pfSense     ONLINE       0     0     0
          da0p4     ONLINE       0     0     0

            Everything looks good to me and I have had 3 more of those crashes today appearntly

            1 Reply Last reply Reply Quote 0
            • S
              stephenw10 Netgate Administrator
              last edited by May 3, 2025, 3:31 AM

              Hmm, unlikely that then.

              Does SMART show the drive as OK?

              And to confirm you've tried a clean re-install?

              W 1 Reply Last reply May 3, 2025, 10:40 AM Reply Quote 0
              • W
                whinis @stephenw10
                last edited by May 3, 2025, 10:40 AM

                So this particular instance is on a VM so no real smart information. The VM host has no reported smart errors or other VMs with issues. Once this started about a year ago, and I think I posted here then and didn't get any real replies, I also tried installing on similar raw hardware and had the same issue. I have also clean installed twice which seems to suggest to me some config issue that transfers.

                1 Reply Last reply Reply Quote 0
                • S
                  stephenw10 Netgate Administrator
                  last edited by May 3, 2025, 3:33 PM

                  Is there anything logged immediately before that?

                  You actually saw the same identical errors when running bare metal?

                  W 1 Reply Last reply May 3, 2025, 5:19 PM Reply Quote 0
                  • W
                    whinis @stephenw10
                    last edited by May 3, 2025, 5:19 PM

                    I did see the exact same error with this config on bare metal.
                    Here is the full php error log for the latest crashes it seems

                    Crash report begins.  Anonymous machine information:

amd64
14.0-CURRENT
FreeBSD 14.0-CURRENT amd64 1400094 #1 RELENG_2_7_2-n255948-8d2b56da39c: Wed Dec  6 20:45:47 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/obj/amd64/StdASW5b/var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/F

Crash report details:

PHP Errors:
[02-May-2025 19:45:00 US/Eastern] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
#0 /etc/inc/notices.inc(135): fopen('', 'w')
#1 /etc/inc/config.lib.inc(95): file_notice('config.xml', 'No config.xml f...', 'pfSenseConfigur...', '')
#2 /etc/inc/config.gui.inc(53): parse_config()
#3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...')
#4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...')
#5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...')
#6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...')
#7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....')
#8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#10 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#11 /usr/local/www/xmlrpc.php(32): require_once('/etc/inc/config...')
#12 {main}
  thrown in /etc/inc/notices.inc on line 135
[02-May-2025 19:45:00 US/Eastern] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
#0 /etc/inc/notices.inc(135): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1154): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
  thrown in /etc/inc/notices.inc on line 135
[02-May-2025 21:10:00 US/Eastern] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
#0 /etc/inc/notices.inc(135): fopen('', 'w')
#1 /etc/inc/config.lib.inc(95): file_notice('config.xml', 'No config.xml f...', 'pfSenseConfigur...', '')
#2 /etc/inc/config.gui.inc(53): parse_config()
#3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...')
#4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...')
#5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...')
#6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...')
#7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....')
#8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#10 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#11 /usr/local/www/xmlrpc.php(32): require_once('/etc/inc/config...')
#12 {main}
  thrown in /etc/inc/notices.inc on line 135
[02-May-2025 21:10:00 US/Eastern] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
#0 /etc/inc/notices.inc(135): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1154): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
  thrown in /etc/inc/notices.inc on line 135
[03-May-2025 04:03:00 US/Eastern] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
#0 /etc/inc/notices.inc(135): fopen('', 'w')
#1 /etc/inc/config.lib.inc(95): file_notice('config.xml', 'No config.xml f...', 'pfSenseConfigur...', '')
#2 /etc/inc/config.gui.inc(53): parse_config()
#3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...')
#4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...')
#5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...')
#6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...')
#7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....')
#8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
#9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
#10 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
#11 /usr/local/www/xmlrpc.php(32): require_once('/etc/inc/config...')
#12 {main}
  thrown in /etc/inc/notices.inc on line 135
[03-May-2025 04:03:00 US/Eastern] PHP Fatal error:  Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
#0 /etc/inc/notices.inc(135): fopen('', 'w')
#1 /etc/inc/config.lib.inc(1154): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors')
#2 [internal function]: pfSense_clear_globals()
#3 {main}
  thrown in /etc/inc/notices.inc on line 135



No FreeBSD crash data found.
			```
                    1 Reply Last reply Reply Quote 0
                    • S
                      stephenw10 Netgate Administrator
                      last edited by May 3, 2025, 5:38 PM

                      Is there anything in the system log just before that though? Maybe something common to all the crashes? For some reason there it's unable to open the filesystem which is triggering those php errors but they are a symptom of some other issue.

                      W 1 Reply Last reply May 3, 2025, 6:05 PM Reply Quote 0
                      • W
                        whinis @stephenw10
                        last edited by May 3, 2025, 6:05 PM

                        Nothing that jumps out. From the system log there is only one pattern I see

                         /getstats.php: No config.xml found, attempting last known config restore.
May  2 13:00:00 pfsense php-fpm[68498]: /getstats.php: PHP ERROR: Type: 1, File: /etc/inc/notices.inc, Line: 135, Message: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
May  2 13:00:00 pfsense php-fpm[68498]: Stack trace:
May  2 13:00:00 pfsense php-fpm[68498]: #0 /etc/inc/notices.inc(135): fopen('', 'w')
May  2 13:00:00 pfsense php-fpm[68498]: #1 /etc/inc/config.lib.inc(95): file_notice('config.xml', 'No config.xml f...', 'pfSenseConfigur...', '')
May  2 13:00:00 pfsense php-fpm[68498]: #2 /etc/inc/config.gui.inc(53): parse_config()
May  2 13:00:00 pfsense php-fpm[68498]: #3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...')
May  2 13:00:00 pfsense php-fpm[68498]: #4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...')
May  2 13:00:00 pfsense php-fpm[68498]: #5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...')
May  2 13:00:00 pfsense php-fpm[68498]: #6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...')
May  2 13:00:00 pfsense php-fpm[68498]: #7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....')
May  2 13:00:00 pfsense php-fpm[68498]: #8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...')
May  2 13:00:00 pfsense php-fpm[68498]: #9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...')
May  2 13:00:00 pfsense php-fpm[68498]: #10 /etc/inc/config.inc(37): require_once('/etc/inc/notice...')
May  2 13:00:00 pfsense php-fpm[68498]: #11 /usr/local/www/includes/functions.inc.php(27): require_once('/etc/inc/config...')
May  2 13:00:00 pfsense php-fpm[68498]: #12 /usr/local/www/getstats.php(38): include_once('/usr/local/www/...')
May  2 13:00:00 pfsense php-fpm[68498]: #13 {main}
May  2 13:00:00 pfsense php-fpm[68498]:   thrown

                        Some thing attempts to load config.xml, its not found and then the stack trace.

                        May  2 16:18:00 pfsense php-fpm[29876]: /xmlrpc.php: No config.xml found, attempting last known config restore.
May  2 16:18:00 pfsense php-fpm[29876]: /xmlrpc.php: PHP ERROR: Type: 1, File: /etc/inc/notices.inc, Line: 135, Message: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
May  2 16:18:00 pfsense php-fpm[29876]: Stack trace:
May  2 16:18:00 pfsense php-fpm[29876]: #0 /etc/inc/notices.inc(135): fopen('', 'w')

                        May  2 19:45:00 pfsense php-fpm[53488]: /xmlrpc.php: No config.xml found, attempting last known config restore.
May  2 19:45:00 pfsense php-fpm[53488]: /xmlrpc.php: PHP ERROR: Type: 1, File: /etc/inc/notices.inc, Line: 135, Message: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
May  2 19:45:00 pfsense php-fpm[53488]: Stack trace:

                        Here is one with several hours and no obvious error

                        May  3 00:47:22 pfsense kernel: arp: 192.168.0.64 moved from xx:xx:xx:xx:xx:xx to xx:xx:xx:xx:xx:xx on bridge0
May  3 01:47:52 pfsense kernel: arp: 192.168.0.64 moved from xx:xx:xx:xx:xx:xx to xx:xx:xx:xx:xx:xx on bridge0
May  3 01:54:00 pfsense sshguard[91998]: Exiting on signal.
May  3 01:54:00 pfsense sshguard[50666]: Now monitoring attacks.
May  3 02:15:00 pfsense sshguard[50666]: Exiting on signal.
May  3 02:15:00 pfsense sshguard[79301]: Now monitoring attacks.
May  3 02:38:00 pfsense sshguard[79301]: Exiting on signal.
May  3 02:38:00 pfsense sshguard[4925]: Now monitoring attacks.
May  3 02:48:22 pfsense kernel: arp: 192.168.0.64 moved from xx:xx:xx:xx:xx:xx to xx:xx:xx:xx:xx:xx on bridge0
May  3 02:59:00 pfsense sshguard[4925]: Exiting on signal.
May  3 02:59:00 pfsense sshguard[45638]: Now monitoring attacks.
May  3 03:01:00 pfsense php-cgi[52259]: rc.periodic: New alert found: The following CA/Certificate entries are expiring:
May  3 03:01:00 pfsense php-cgi[52259]: Certificate: webConfigurator default (62294113a9989) (62294113a9989): Expiring soon, in 21 days
May  3 03:03:00 pfsense sshguard[45638]: Exiting on signal.
May  3 03:03:00 pfsense sshguard[54381]: Now monitoring attacks.
May  3 03:48:53 pfsense kernel: arp: 192.168.0.64 moved from xx:xx:xx:xx:xx:xx to xx:xx:xx:xx:xx:xx on bridge0
May  3 04:03:00 pfsense php-fpm[53488]: /xmlrpc.php: No config.xml found, attempting last known config restore.
May  3 04:03:00 pfsense php-fpm[53488]: /xmlrpc.php: PHP ERROR: Type: 1, File: /etc/inc/notices.inc, Line: 135, Message: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
May  3 04:03:00 pfsense php-fpm[53488]: Stack trace:
May  3 04:03:00 pfsense php-fpm[53488]: #0 /etc/inc/notices.inc(135): fopen('', 'w')
                        1 Reply Last reply Reply Quote 0
                        • S
                          stephenw10 Netgate Administrator
                          last edited by May 3, 2025, 9:32 PM

                          Hmm, that's very odd. What hypervisor are you using?

                          W G 2 Replies Last reply May 3, 2025, 9:54 PM Reply Quote 0
                          • W
                            whinis @stephenw10
                            last edited by May 3, 2025, 9:54 PM

                            Proxmox

                            1 Reply Last reply Reply Quote 0
                            • G
                              GPz1100 @stephenw10
                              last edited by May 3, 2025, 9:56 PM

                              @stephenw10 If some part of the config (config.xml) was causing this issue. How would one even begin to troubleshoot that?

                              Restore portions of it to a new install until crash is triggered or is there a better way?

                              1 Reply Last reply Reply Quote 0
                              • S
                                stephenw10 Netgate Administrator
                                last edited by May 3, 2025, 11:10 PM

                                Nothing should ever result in the config file being inaccessible. It looks like some filesystem or drive issue though because in addition to not being able to open the config file the PHP error is being unable to open the notices file to post an alert for it. 🤔

                                W 1 Reply Last reply May 5, 2025, 7:17 PM Reply Quote 0
                                • W
                                  whinis @stephenw10
                                  last edited by May 5, 2025, 7:17 PM

                                  Just to be proactive, the SSD its on is at 30% wearout, I moved it to another ssd in the VM and still same issue. I checked the logs on the VM host (proxmox) and the sys logs of pfsense and no obvious errors in either.

                                  W 1 Reply Last reply May 6, 2025, 2:12 AM Reply Quote 0
                                  • W
                                    whinis @whinis
                                    last edited by May 6, 2025, 2:12 AM

                                    And here is the error that occasionally occurs that causes the config to be reloaded

                                    May  5 21:12:00 pfsense php_wg[93764]: /usr/local/pkg/wireguard/includes/wg_service.inc: XML error: no pfsense or pfsense object found!
May  5 21:12:00 pfsense php_wg[93764]: 
May  5 21:12:00 pfsense php_wg[93764]: /usr/local/pkg/wireguard/includes/wg_service.inc: pfSense is restoring the configuration /cf/conf/backup/config-1746449192.xml
May  5 21:12:00 pfsense php_wg[93764]: /usr/local/pkg/wireguard/includes/wg_service.inc: New alert found: pfSense is restoring the configuration /cf/conf/backup/config-1746449192.xml
May  5 21:12:00 pfsense php_wg[93764]: /usr/local/pkg/wireguard/includes/wg_service.inc: PHP ERROR: Type: 1, File: /usr/local/pkg/wireguard/includes/wg_globals.inc, Line: 197, Message: Uncaught Error: Cannot use a scalar value as an array in /usr/local/pkg/wireguard/includes/wg_globals.inc:197
May  5 21:12:00 pfsense php_wg[93764]: Stack trace:
May  5 21:12:00 pfsense php_wg[93764]: #0 /usr/local/pkg/wireguard/includes/wg_globals.inc(238): wg_init_config_arr(-1, Array)
May  5 21:12:00 pfsense php_wg[93764]: #1 /usr/local/pkg/wireguard/includes/wg.inc(345): wg_globals()
May  5 21:12:00 pfsense php_wg[93764]: #2 /usr/local/pkg/wireguard/includes/wg_service.inc(360): wg_resolve_endpoints()
May  5 21:12:00 pfsense php_wg[93764]: #3 /usr/local/pkg/wireguard/includes/wg_service.inc(294): wg_service_daemon()
May  5 21:12:00 pfsense php_wg[93764]: #4 /usr/local/pkg/wireguard/includes/wg_service.inc(60): wg_service_cli_start(false)
May  5 21:12:00 pfsense php_wg[93764]: #5 {main}
May  5 21:12:00 pfsense php_wg[93764]:   thrown
May  5 21:12:00 pfsense php_wg[93764]: /usr/local/pkg/wireguard/includes/wg_service.inc: New alert found: PHP ERROR: Type: 1, File: /usr/local/pkg/wireguard/includes/wg_globals.inc, Line: 197, Message: Uncaught Error: Cannot use a scalar value as an array in /usr/local/pkg/wireguard/includes/wg_globals.inc:197
May  5 21:12:00 pfsense php_wg[93764]: Stack trace:
May  5 21:12:00 pfsense php_wg[93764]: #0 /usr/local/pkg/wireguard/includes/wg_globals.inc(238): wg_init_config_arr(-1, Array)
May  5 21:12:00 pfsense php_wg[93764]: #1 /usr/local/pkg/wireguard/includes/wg.inc(345): wg_globals()
May  5 21:12:00 pfsense php_wg[93764]: #2 /usr/local/pkg/wireguard/includes/wg_service.inc(360): wg_resolve_endpoints()
May  5 21:12:00 pfsense php_wg[93764]: #3 /usr/local/pkg/wireguard/includes/wg_service.inc(294): wg_service_daemon()
May  5 21:12:00 pfsense php_wg[93764]: #4 /usr/local/pkg/wireguard/includes/wg_service.inc(60): wg_service_cli_start(false)
May  5 21:12:00 pfsense php_wg[93764]: #5 {main}
May  5 21:12:00 pfsense php_wg[93764]:   thrown
                                    1 Reply Last reply Reply Quote 0
                                    • S
                                      stephenw10 Netgate Administrator
                                      last edited by May 6, 2025, 2:58 AM

                                      Hmm, you've only seen this in 2.7.2?

                                      W 1 Reply Last reply May 6, 2025, 3:38 AM Reply Quote 0
                                      • W
                                        whinis @stephenw10
                                        last edited by May 6, 2025, 3:38 AM

                                        As far as I know. This config I think used to be 23. Something and then I rebuilt it once I stopped using commercial for my home.

                                        1 Reply Last reply Reply Quote 0
                                        • S
                                          stephenw10 Netgate Administrator
                                          last edited by May 6, 2025, 1:34 PM

                                          Are you able to test 2.8-Beta?

                                          W 1 Reply Last reply May 6, 2025, 1:52 PM Reply Quote 0
                                          25 out of 30
                                          • First post
                                            25/30
                                            Last post
                                          Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                                            This community forum collects and processes your personal information.
                                            consent.not_received