config.xml empty all of a sudden

jan_p

Just had a crash of pfsense it seams.

I network went down, and I couldn't even reach the web admin page.

Got terminal access to it, and found error messages, that pointed my to the issue.
My config.xml had been cleared, and was now empty.

I've seen other report the issue as well in the forum, and some pointing to bad that change WAN IP connections frequently, running out of disk space, or loosing power.

None of these appear to be case in for me.
I additionally have an UPS on the firewall.
Anyone have any suggestions to why this appear to be happening?

stephenw10

Anything logged at that time?

Did you have any packages installed that make automated changes/updates like pfBlocker?

Even if the config becomes damaged so it can no longer be loader pfSense should normally revert to the last good config instead of ending up without anything.

jan_p

@stephenw10 I added a static ip mapping on the DHCP service, about 30 - 1 hour prior to the crash.
I still had the the web admin open, as I was referring to static bindings, but made no further changes.

I also check the harddrive. It's only 18% full, and I don't see any hardware errors reported on the drive.
I've had this fiirewall for quite a long time, and have never seen this before.

stephenw10

Do you still see the old configs in /conf/backup ? Do the timestamps look correct?

jan_p

@stephenw10 Yes, I copied one of the backup configs over to restore. After I made the edit, I stepped away for about 20 minutes, and network was down when I came back.
I saw that several backup files was 0 bytes in size.
Several files, because I restarted the firewall a few times, and re-plugged the WAN link before I released, I had to connect a monitor and keyboard to the firewall to resolve the issue.

Hmm, just had another look. Something clearly was happening, that I don't think was my single edit.
-rw-r--r-- 1 root wheel 218625 Jan 22 04:36 config-1737542176.xml
-rw-r--r-- 1 root wheel 218882 Jan 22 04:36 config-1737542180.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:36 config-1737542181.xml
-rw-r--r-- 1 root wheel 218882 Jan 22 04:36 config-1737542186.xml
-rw-r--r-- 1 root wheel 218884 Jan 22 04:36 config-1737542187.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:37 config-1737542188.xml
-rw-r--r-- 1 root wheel 218871 Jan 22 04:37 config-1737542232.xml
-rw-r--r-- 1 root wheel 218884 Jan 22 04:37 config-1737542233.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:37 config-1737542234.xml
-rw-r--r-- 1 root wheel 218625 Jan 22 04:37 config-1737542237.xml
-rw-r--r-- 1 root wheel 218882 Jan 22 04:37 config-1737542238.xml
-rw-r--r-- 1 root wheel 218890 Jan 22 04:37 config-1737542239.xml
-rw-r--r-- 1 root wheel 218871 Jan 22 04:37 config-1737542241.xml
-rw-r--r-- 1 root wheel 218887 Jan 22 04:37 config-1737542242.xml

stephenw10

Hmm, yeah. Try diff'ing those and see what was changing.

jan_p

@stephenw10 Looks like it's WireGuard.

I tried posting example of the changes, but I was not allowed to.
It contains applying default settings, installing earlysheelcmd(s) and de-installing and then installing ACL group WireGuard

Wireguard was locked down to only accept connections from 2 specific IP's. I don't know if my phone ( one of the ip's ) could have triggered these. A software update maybe?

stephenw10

Ah, that can happen at boot with Wireguard so I imagine it was repeatedly rebooting? Though it's a lot inside 1 min....

jan_p

@stephenw10
hmmm.
In system.log, I see 2 boots at 04:36:33. As soon as the first boot completed, it booted again.
I don't know why it booted though. I have a UPS providing power.
I do see the same pattern on the Jan 13th.
Maybe the UPS, was testing the battery, and briefly interrupted power?
Don't know that I have a method of verifying that though.

I can understand the config file is lost, if WireGuard updates the config file, and then a second boot occur, right in between the old config is moved aside, but before the new file is written.

Maybe something to discuss with dev's. If the process can be optimized somehow, so the config file is not actually deleted, but the firmware keep a pointer to the most recent file instead.
Or maybe split the config file, separating out 3rd party add-on's, as well as more dynamic content, like DHCP address updates into separate files. Ensuring that the core configuration is never lost?

stephenw10

Nothing shown in the log before it starts booting again?

jan_p

@stephenw10 Not sure I follow, but in system.log there's nothing that stands out to me, other than boot messages I would expect. After first boot completes, straight in to booting again.

Jan 22 04:36:33 pfSense kernel: All buffers synced.
Jan 22 04:36:33 pfSense kernel: Uptime: 16m7s
Jan 22 04:36:33 pfSense kernel: ---<<BOOT>>---
Jan 22 04:36:33 pfSense kernel: Copyright (c) 1992-2023 The FreeBSD Project.

Uptime, confuses me though. that doesn't map to what I see in system.log.
There are only two instances of the copyright message for example. When it boots the first time at 04:36:33 and the second time, within the same minute.
Previous boot prior to this on the January 13th as far as I can tell.

Gertjan

@jan_p said in config.xml empty all of a sudden:

Jan 22 04:36:33 pfSense kernel: All buffers synced.
Jan 22 04:36:33 pfSense kernel: Uptime: 16m7s
Jan 22 04:36:33 pfSense kernel: ---<<BOOT>>---
Jan 22 04:36:33 pfSense kernel: Copyright (c) 1992-2023 The FreeBSD Project.

That's you (an admin) rebooting the system.

Visit Youytube, the Negate channel and look at the file system checker video.
If the file system is dirty, it can become read only what would explain "no more writes" == no more config.xml.

stephenw10

Well not necessarily admin rebooting but something rebooted deliberately. Those last logs are pfSense correctly shutting down. I.e. it didn't spontaneously reboot.

So there should be something logged when it starts rebooting that might show why.

Gertjan

@stephenw10 said in config.xml empty all of a sudden:

but something rebooted deliberately

Don't wake up the "pfSense auto re boot" as, afaik, it can't / it doesn't.
( right ? you tell me ! )

A reboot is a mouse click in the GUI or someone using menu option 5 or menu option 8 and typing the reboot command.

So something is someone - a person.

stephenw10

It reboots at upgrade for example. Or you could add a cron job to reboot. A UPS battery low signal could shut it down.

Either way something triggered it. It didn't see a hard reset.

jan_p

@stephenw10 I did shut it down twice while troubleshooting, to make it restart. This was while the network was down, and I was trying to figure out why, before I connected a monitor directly to it.
I'll go through the logs again, and see if I can find any clues to why it went down in the first place ( and what caused the config file to be left empty ).