Stack trace when starting pfSense - System wont start
-
So I've been running pfSense virtualized on proxmox for awhile now. I rebooted the system this morning and now the pfSense VM will start, but the pfSense application will not start appropriately. From within proxmox I can get to a console and a screenshot is below. None of the options in the command line prompt will work -- producing a stack trace -- other than reboot the system. I'm not sure what to do at this point since I haven't upgraded anything in awhile. I'm running pfsense 2.72.
-
That is actually caused by being unable to open the file to write the notice to.
If you're running UFS the first thing to try there is a manual FSCK:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filesystem-check.html#manual-filesystem-checkSteve
-
Ok I'm running ufs. I did a fsck -fy / command 6 times and rebooted and am still getting a stack trace error.
I'm still however getting a /dev/ufsid/663aaa2ab6190ed9 (No WRITE)
Perhaps this is source of the error??Any other suggestions??
-
You ran that from single user mode? Is the drive somehow read-only? I assume it's a virtual disk?
That should not appear like that:
Enter full pathname of shell or RETURN for /bin/sh: # fsck -fy / ** /dev/diskid/DISK-987032200144s2a ** Last Mounted on / ** Root file system ** Phase 1 - Check Blocks and Sizes ** Phase 2 - Check Pathnames ** Phase 3 - Check Connectivity ** Phase 4 - Check Reference Counts ** Phase 5 - Check Cyl groups 58473 files, 542452 used, 6259451 free (5811 frags, 781705 blocks, 0.1% fragmentation) ***** FILE SYSTEM IS CLEAN *****Even if it's mounted read-only:
# mount -p /dev/diskid/DISK-987032200144s2a / ufs ro,noatime 1 1 devfs /dev devfs rw 0 0 -
I ran from single mode (option #2 or option #3 with boot -s at CLI).
After some reading fsck can not be performed on mounted disks, so I guess that's why this was mounted RO -- which why is it mounted???
The disk is a 200GB virtualized disk. Proxmox for the VMs is a pair of mirrored 2Tb ZFS drives. I'm assuming when configuring the 200GB VirtualDrive, the OS behind the scene is creating a 200GB Zvol which is a block device. Within the block device its being formated for UFS.
Anyway a lot of theoretical discussion. No idea what to do at this point, since I don't know how to exactly "rescue" the device at this point. I'm open to any suggestions,
-
This - same procedure as the link shown above - is the one you need :
How to Run a pfSense Software File System Check (5/2020) -
Is it otherwise still routing? Passing traffic?
Can you access the webgui?
If you allow it to boot to the menu and choose option 8 there that should still work. Run
mount -pat that point. Is it still mounted read only? -
I need to sit down and try things again, however the router webgui is totally inaccessible, and even post of the options on the menu (like reconfigure interfaces, reboot, etc) do not work since the throw a stack trace when selecting these options . The shell option works and that's how I reboot. I need tr try the other options others have proposed here - thanks for reaching out.
-
Sorry about the delay. Was gone for a week.
mount -p shows rw mount/dev/ufsid/6663aaa2ab6190ed89 / usf rw,noatime 1 1
devfs /dev devfs 0 0
/dev/vtbd0p1 /boot/efi msdosfs rw 2 2
tmpfs /var/run tmpfs rw 0 0Even with this I'm getting a lot of errors, specifically
PHP ERROR: Type 1, File /etc/inc/notices.inc Line 135, Message: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135
Stack trace:
...
...I went an opened /etc/inc/notices.inc and looked at line 135:
$queueout = fopen($notice_path, "w");however it looks like in the code if the file handle is empty it would throw an error stating Could not open %s for writing. I'm getting a stack trace here which would suggest possible the $notice_path variable is empty??? Looking further up the code $notice_path = $notice_root . '/notices.serial';
Honestly I'm not sure where to go from here. I've done the fsck method from option 2 and posted above what it gave me.
Where is the config.xml supposed to be located at? Line #1 of the stack trace:
#1 /etc/inc/config.lig.inc(95) : file_notice('config.xml', "no config.xml f...', pfSenseConfigur...,'')Perhaps I'm just digging here. However just going down this path:
# find / -name config.xml /conf.default/config.xml /cf/conf/config.xml/conf.default/config.xml represents default config.xml file
/cf/conf/config.xml -- contents of this file are nothing -- it's an empty file
-rw-r--r-- 1 root wheel 0 June 23 13:00 config.xmlHowever I have a bunch of files within /cf/conf/backup and each backup config.xml within this directory is 311536 bytes in length for example:
-rw-r--r-- 1 root wheel 311536 Jun 23 12:00 config-1719158400.xmlNot sure if this means anything.
-
At this point I would just re-install. You can never be 100% sure that anything you did to recover the filesystem from there isn't going to leave some broken file to catch you out later.
Since you're running as a VM you might have a snapshot you can just revert to anyway.
-
@kevdog said in Stack trace when starting pfSense - System wont start:
/cf/conf/config.xml -- contents of this file are nothing -- it's an empty file
That's scary.
[24.03-RELEASE][root@pfSense.bhf.tld]/root: ll /cf/conf/config.xml -rw-r--r-- 1 root wheel 1455699 Jul 3 00:47 /cf/conf/config.xmlIt is the currently used main pfSense config file.
It contains all my settings, without it, pfSense "won't work".
