config.xml empty, subsequent inability to restart properly, and what i did.
-
Hi there -
I don't know really what happened here but, in the event that this is useful to someone, I'd thought I'd share some experiences that ended up with my router not functioning and how I brought it back to life.
Up until about 2pm today everything was fine for months (NB: pfSense 2.7.2, recent upgrade on a 3 year old Protectli)
Around then a service worker needed to move some stuff around and so he pulled out the Cable modem Ether (into WAN)
Subsequently my pfSense dropped off the (local 192.168.1.x) network.
I "consoled" in and did some snooping.
The error I saw was a PHP stack trace that ultimately pointed at config.xml being empty (pasted after my closing)
I looked and saw that...it was an empty file (zero length)
I found a recent backup that was non-empty (the latest was empty) and copied that into place.
I then rebooted and things returned to normal.There are details I didn't notice/check (e.g. whether there was a reboot of the router before the reboot I mentioned) so perhaps those might be helpful.
Thanks,
Scott
FreeBSD/amd64 (Amnesiac) (ttyu0) 2024-01-03T19:24:30.959094-08:00 - login 881 - - login on ttyu0 as root Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135 Stack trace: #0 /etc/inc/notices.inc(135): fopen('', 'w') #1 /etc/inc/config.lib.inc(95): file_notice('config.xml', 'No config.xml f...', 'pfSenseConfigur...', '') #2 /etc/inc/config.gui.inc(53): parse_config() #3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...') #4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...') #5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...') #6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...') #7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....') #8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...') #9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...') #10 /etc/inc/config.inc(37): require_once('/etc/inc/notice...') #11 /etc/rc.banner(27): require_once('/etc/inc/config...') #12 {main} thrown in /etc/inc/notices.inc on line 135 PHP ERROR: Type: 1, File: /etc/inc/notices.inc, Line: 135, Message: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135 Stack trace: #0 /etc/inc/notices.inc(135): fopen('', 'w') #1 /etc/inc/config.lib.inc(95): file_notice('config.xml', 'No config.xml f...', 'pfSenseConfigur...', '') #2 /etc/inc/config.gui.inc(53): parse_config() #3 /etc/inc/auth.inc(34): require_once('/etc/inc/config...') #4 /etc/inc/openvpn.inc(36): require_once('/etc/inc/auth.i...') #5 /etc/inc/filter.inc(30): require_once('/etc/inc/openvp...') #6 /etc/inc/ipsec.inc(25): require_once('/etc/inc/filter...') #7 /etc/inc/gwlb.inc(27): require_once('/etc/inc/ipsec....') #8 /etc/inc/functions.inc(35): require_once('/etc/inc/gwlb.i...') #9 /etc/inc/notices.inc(26): require_once('/etc/inc/functi...') #10 /etc/inc/config.inc(37): require_once('/etc/inc/notice...') #11 /etc/rc.banner(27): require_once('/etc/inc/config...') #12 {main} thrown Fatal error: Uncaught ValueError: Path cannot be empty in /etc/inc/notices.inc:135 Stack trace: #0 /etc/inc/notices.inc(135): fopen('', 'w') #1 /etc/inc/config.lib.inc(1154): file_notice('phperror', 'PHP ERROR: Type...', 'PHP errors') #2 [internal function]: pfSense_clear_globals() #3 {main} thrown in /etc/inc/notices.inc on line 135 0) Logout (SSH only) 9) pfTop 1) Assign Interfaces 10) Filter Logs 2) Set interface(s) IP address 11) Restart webConfigurator 3) Reset webConfigurator password 12) PHP shell + pfSense tools 4) Reset to factory defaults 13) Update from console 5) Reboot system 14) Enable Secure Shell (sshd) 6) Halt system 15) Restore recent configuration 7) Ping host 16) Restart PHP-FPM 8) Shell Enter an option: -
@scottmsilver said in config.xml empty, subsequent inability to restart properly, and what i did.:
that ultimately pointed at config.xml being empty
Definition pfSense = "a firewall router with a GUI" where the entire config is stored in one ( 1 ) file.
If the creation of this one and only file fails, things go down hill fast.
The question is : why wasn't the file not created ? The most probable answer is because no more extra blocks (a file system is a huge table of portions of disk space, see them as the sectors on the disk).
So, no more sectors free : the OS goes down, or at least, won't be able to boot anymore.
Not really special, as this will happen with any OS on planet earth : try to allocate every available block (sector) on your any "Windows PC" and you will see Windows start to behave "strangely", and then fail. Probably unable to boot.More info : when any system, like a windows PC, your phone, and also pfSense, boots, a minimal file system test is executed. If this fails, the file system is probably activated, but in read only mode.
Figure this : try writing a file on a read only disk ;)There is good news : you have al ready the console access, right ? For ones, no need to sort the manual (documentation), go straight to here Netgate Official video's.
use this one : How to Run a pfSense Software File System Check (5/2020) -
Thanks!
Yeah, I suspected that the filesystem might be full or unwriteable. The filesystem on which /cf sits was neither full nor unwriteable when I consoled in.
I'm curious when this zero length config.xml got created. I think it somehow seemed related to the action of the service worked, I wonder if a gateway coming in and out can cause config.xml to get generated again (seems insane).
Thanks,
Scott
-
@scottmsilver said in config.xml empty, subsequent inability to restart properly, and what i did.:
I wonder if a gateway coming in and out can cause config.xml to get generated again (seems insane).
Gateway can be dynamic. Most SOHO ISPs use DHCP (so the IP isn't the same 'for ever') or something else : a lousy modem DSL/cable connection that interrupts all the time. And more. All reasons why the upstream connection gets regenerated.
And changes on pfSense are save in the config.
Your upstream connection is of the 'insane' type (new one very frequently) then yes, the config file gets updated.
And it even doesn't get really updated. The one in place gets renamed, a new file is created - and at that moment it exists, but with 0 bytes - and then the content in memory gets written to the file in one go. Then the file gets 'closed' and at that moment the real file size is calculated by the OS.So true : get yourself a static IPv4 access, and the gateway, all uplink related stuff gets never updated or refreshed as it is ... static.
That is : rip out the pfSense WAN cable and you still get 'gateway' changes.
( so you add an UPS to all your boxes, and the uplink stays up for years ;) ) -
My question is whether gateway status (up or down, for example) is recorded in config.xml. That would be an unusual design. Most folks build software to keep the static or semi-static changes in something they are super, super careful to manage. In any case, my experience will live out here for someone to notice should it happen more often.
scott
-
This happens when the system shutdown cold or hangs during any file writing updates not completed yet. I have plenty of HDD space.
Unfortunately, I have enough experience with this issue several times now :) and I know exactly why. My APC unit does a os system shutdown first at exactly 10pm each night. Then its a hard off. I need to set a longer time for off. My equipment is hosted on prem in my home for now. Sometimes I forget it's time to quit. Yes, I do this on purpose otherwise I'll never sleep.
In short during the file updates, or adding any kind of rule updates when the system shuts off cold you get this message. I have not had this happen on Linux but it does happen in BDS.
Hope this helps others experience the same thing. Easy fix is to have backups.
-
I will say that I've only ever seen issues like that when using UFS without RAM disks enabled. Using ZFS or enabling RAM disks should prevent it for the vast majority of cases.
