Pfsense 2.5 stacks at boot with dots

SjM

@chamilton_ccn as previously mentioned I had to wipe the installation and perform a clean install ASAP to restore my service. The only data I have is contained in these photos.

Given the helpful comments I'll know what action to take if there is a next time.

Boot into single user more and run fsck multiple times to clear any fs corruption errors.
Delete any zeroed config.xml files including backups.
Attempt a reboot.
In addition try to determine the sequence of events. Crash reboot dots(fs corruption) loop or dots(fs corruption) loop etc.
Save any and all config.xml files and log files for further checking.

chamilton_ccn

@sjm Gotcha.

If anyone in this thread is willing to attempt to recreate the problem, you might go about it by first creating a backup of your config and downloading/saving it somewhere safe, then interrupting the boot process and dropping to a shell, then:

echo > /cf/conf/config.xml
echo > /cf/conf/backup/config-`date +%s`.xml
reboot

All my machines are in production and I don't have a VM at the ready.

Gertjan

@chamilton_ccn said in Pfsense 2.5 stacks at boot with dots:

echo > /cf/conf/config.xml
echo > /cf/conf/backup/config-date +%s.xml
reboot

That's like deleting the Windows Registry files and rebooting Windows to see if it boot up again.
It won't.
Don't believe my word on it. Try it out yourself. Or better : some have made video's about it.

The real issue is : why is the file system (is it ?) messed up ? Why are (config only ?) files created, but not 'filled' up ?

chamilton_ccn

@gertjan said in Pfsense 2.5 stacks at boot with dots:

@chamilton_ccn said in Pfsense 2.5 stacks at boot with dots:

echo > /cf/conf/config.xml
echo > /cf/conf/backup/config-date +%s.xml
reboot

That's like deleting the Windows Registry files and rebooting Windows to see if it boot up again.
It won't.
Don't believe my word on it. Try it out yourself. Or better : some have made video's about it.

It looks nothing like that. The backup directory contains multiple backup files, so question should be: Why, upon encountering an empty or bad file, doesn't it roll over to the next one? Or at the least, why isn't an error logged ... or something other than printing an infinite string of dots to the console?

That said, one of the most helpful steps in any troubleshooting scenario is the ability to reliably recreate the problem. That's all I was suggesting here, recreating the conditions that are suspected to lead to the observed symptoms, to see if they do in fact lead to the same end.

Gertjan

@chamilton_ccn said in Pfsense 2.5 stacks at boot with dots:

Why, upon encountering an empty or bad file

.... this situation was probably not tested.

Why would the most important system file disappear ??
It's the one and only file you should backup regular - as with this file you can rebuild an entire pfSense system from scratch in a couple of minutes.
The absence of this will take down the same system - but faster ^^

@chamilton_ccn said in Pfsense 2.5 stacks at boot with dots:

or something other than printing an infinite string of dots to the console?

These dots are just what should be "low-bud" progress bar, as it could take some time to look over 'all' the saved config files. It could be the last one that is good.
Or the first.

edit : see below : I found out what it does when running ecl.php .....it's a lengthy, non-standard process. I never saw this file running.

@chamilton_ccn said in Pfsense 2.5 stacks at boot with dots:

That said, one of the most helpful steps in any troubleshooting scenario is the ability to reliably recreate the problem.

I totally agree.
But by just wiping (zeroing out) the config files you test the behaviour of what the system would do without a valid config file. That is : with the multiple config files - with non content.
It should load in default config (this one has no interface assigned), and wait for you at the console level so you can re init the entire system again.

I want to know a test that creates failing disk system .... or whatever is needed so that only empty files are created.

Btw : I just found ecl.php - it's in /etc/ and not in /etc/inc/
(wtf ...)

It stand for External ( external from to what ??? ) Config Loader ....
It scan other disks / partitions / slices / .... !!!!

It look like that it found something somewhere -dono what 'disks and partions' you have , and then the system tries to 'upgrade' it (every pfSense version has its own config 'level' version) and that process fails.
True, that's an issue.

What I would like to know is : why is the (your) main /cf/conf/config.xml fckd up ?
I never saw that before.

And I'm pretty sure that when you install a clean pfSense :

On another system
Not changing any settings
that you can not create the problem.

I'll leave it up to you to draw the (a ?) conclusion ......

So, tell us :
Your - all - the device details.
Your - all - settings.

I know, this is silly - and hard to answer.
But where would I start to debug test this ? By nuking my own systems ..... no thanks.

netblues

@gertjan Its time to fire up your beloved hyper-v system, and take some snapshots, before nuking the whole thing.

Maybe this will give as some clues.
I will try to do the same.
Having said that, the same kvm is running pf 24x7 ever since the event happened and I posted the issue

AKEGEC

@sjm what a bummer glad that you resolved it. Btw did you noticed the date 5.5. that was also the date when cyberattacks happened to my Belgian clients.

netblues

@akegec Its just a coincidence that happen at about the same time as the belgian attacks.
It has happened again weeks before too.

vajonam

@netblues said in Pfsense 2.5 stacks at boot with dots:

belgian attacks

This happens when you badly edited config.xml or somehow its been corrupted, (borked save etc.) quite rare but this happend after I was futzing around with the config file.

SjM

@vajonam I never directly edit configure.xml. So the only direct potential for corruption was through the web interface??

Dilligaf

@sjm Similarly. This issue is happening for me also fairly frequently through the web interface and can confirm this issue has arisen in 2.5.x. Its not rare and happened 2 days previously.

I'm running 2.5.1 CE on a NoName box.

I had to tweak the firewall rules temporarily yesterday so I could install a camera on a normally wan-disconnected vlan. After reverting the change using enable/disable rule button I noted the firewall playing up (some clients on a different lan disconnected) and I rebooted to the dots.

Fortunately I've learned to keep a second installation on an alternative disk up to date and backup frequently via a cron job on my NAS, but its still a PITA to clean install and bring in a recent backup.

Gertjan

Foundthis about the 'ecel' messages :
https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.html

From what I make of it : ecl.php is used if the main pfSense FreeBSD boot drive (partition) is hosed (contains non valid info), so pfSne, at boot, starts to look at all other partitions in the system, and tries to locate a "config.cml". This files is then 'updated' if needed, and if successful, it is used.

To isolate the zero byte issues, install pfSense on another system - or, at least, change the drive in the system used.

Dilligaf

@gertjan Handy tip there to place the backup in the fat partition on the USB key - thanks for that. No reason why it can't live attached to another machine and be continuously updated with a backup for when needed.

Not sure what else I can do to isolate the zero bytes issue. Pretty sure I can fairly reliably bork my 2.5.1 install by changing firewall rules already.

Gertjan

@dilligaf said in Pfsense 2.5 stacks at boot with dots:

Pretty sure I can fairly reliably bork my 2.5.1 install by changing firewall rules already.

Thousands of pfSense user update their firewall rules on a daily bases.
On every change change, the current config.xml is updated.
Before the update, a dated copy is made. You can see them here : Diagnostics > Backup & Restore > Config History (with dates and file sizes).

I guess it's easy to create 0 bytes files : one way of doing so is : fill up your drive ,and leave some 30 Kbytes of space.
Now, files can get crated, but when written,, it fails.

Keep in mind : you and I use the same code.
What differs is :
Your hardware.
Your setup (I'm using the default setup).

Dilligaf

@gertjan OK thanks for that. All of which I'm aware of already.

My disk usage is 3%. Degrading disks is unlikely as I've had the fault on 2 already. And it didn't happen at 2.4.5 as far as I can recall.

And like you already mentioned - its never really a good time to attempt to deliberately induce such a fault on a system you're dependent on.

(If it helps I'm happy to sanitise my config.xml file and attach as soon as I've time to edit it for any personal information)

darcy

@chamilton_ccn Saved me. Thanks a lot!

chamilton_ccn

@darcy

Dilligaf

@gertjan Had another zero bytes config file event. I'm relieved 2.5.2 Snapshots have gone a long way to solving this.The new code does find an uncorrupted config file as intended.

I did a clean install in migrating to 2.5.2 snapshots.

However, this issue with the initial corruption remains a mystery. I noted in the change report mention of corruption possibly being caused by the shaper and so disabled that in my setup but I've had the issue reoccur regardless.

The sequence of events I have is this:

Reboot
All interfaces come up for a very short while, then crash to power off.
Boot (needing attendance to power up)
Reboot - system finds the good config and all happy (but no crash report)

pfSenseConfigurator

    
* list itemNo config.xml found, attempting last known config restore. @ 2021-06-03 09:36:06
    
* list itempfSense is restoring the configuration /conf/backup/config-1622706943.xml @ 2021-06-03 09:36:07

I can see the scenario coming eventually when I'm not near the office to physically power up the router. I'm wondering power supply? Though that wouldn't be consistent with other users here reporting similar issues.

Is there anything at all I can do to help diagnose the cause of the crash and corruption?

Gertjan

I tend to think the issue is related to 'power'.
That's a hard one to debug, as, for myself, I never reboot my pfSense. It neither powers down without me knowing about it.
And the day I happened because I've been ripping out the wrong cables, I will reboot with the console activated, me watching the boot process, looking for 'unusual' messages.

Also, I'm still not using SSD for pfSense, but old fashioned hard disks. These have stickers with the install date, and after 3 à 5 years will get changed. I'm now on my fourth disk change iteration.

@dilligaf said in Pfsense 2.5 stacks at boot with dots:

I can see the scenario coming eventually when I'm not near the office

That one is valid for everybody I guess. The presence of an UPS will lower the risks, though.
I'm lucky, as the company where I works is a 24/24 - 365/365 and there there is always someone that could "push a button". Because of this, the inverted Murphy's law implies : it won't happen.

netblues

@gertjan How on earth this can be related to power when it is happenig on a vm, and the host is there, with no downtime/disk events.