Pfsense 2.5 stacks at boot with dots
-
@sjm what a bummer glad that you resolved it. Btw did you noticed the date 5.5. that was also the date when cyberattacks happened to my Belgian clients.
-
@akegec Its just a coincidence that happen at about the same time as the belgian attacks.
It has happened again weeks before too. -
@netblues said in Pfsense 2.5 stacks at boot with dots:
belgian attacks
This happens when you badly edited config.xml or somehow its been corrupted, (borked save etc.) quite rare but this happend after I was futzing around with the config file.
-
@vajonam I never directly edit configure.xml. So the only direct potential for corruption was through the web interface??
-
@sjm Similarly. This issue is happening for me also fairly frequently through the web interface and can confirm this issue has arisen in 2.5.x. Its not rare and happened 2 days previously.
I'm running 2.5.1 CE on a NoName box.
I had to tweak the firewall rules temporarily yesterday so I could install a camera on a normally wan-disconnected vlan. After reverting the change using enable/disable rule button I noted the firewall playing up (some clients on a different lan disconnected) and I rebooted to the dots.
Fortunately I've learned to keep a second installation on an alternative disk up to date and backup frequently via a cron job on my NAS, but its still a PITA to clean install and bring in a recent backup.
-
Foundthis about the 'ecel' messages :
https://docs.netgate.com/pfsense/en/latest/backup/restore-during-install.htmlFrom what I make of it : ecl.php is used if the main pfSense FreeBSD boot drive (partition) is hosed (contains non valid info), so pfSne, at boot, starts to look at all other partitions in the system, and tries to locate a "config.cml". This files is then 'updated' if needed, and if successful, it is used.
To isolate the zero byte issues, install pfSense on another system - or, at least, change the drive in the system used.
-
@gertjan Handy tip there to place the backup in the fat partition on the USB key - thanks for that. No reason why it can't live attached to another machine and be continuously updated with a backup for when needed.
Not sure what else I can do to isolate the zero bytes issue. Pretty sure I can fairly reliably bork my 2.5.1 install by changing firewall rules already.
-
@dilligaf said in Pfsense 2.5 stacks at boot with dots:
Pretty sure I can fairly reliably bork my 2.5.1 install by changing firewall rules already.
Thousands of pfSense user update their firewall rules on a daily bases.
On every change change, the current config.xml is updated.
Before the update, a dated copy is made. You can see them here : Diagnostics > Backup & Restore > Config History (with dates and file sizes).
I guess it's easy to create 0 bytes files : one way of doing so is : fill up your drive ,and leave some 30 Kbytes of space.
Now, files can get crated, but when written,, it fails.Keep in mind : you and I use the same code.
What differs is :
Your hardware.
Your setup (I'm using the default setup). -
@gertjan OK thanks for that. All of which I'm aware of already.
My disk usage is 3%. Degrading disks is unlikely as I've had the fault on 2 already. And it didn't happen at 2.4.5 as far as I can recall.
And like you already mentioned - its never really a good time to attempt to deliberately induce such a fault on a system you're dependent on.
(If it helps I'm happy to sanitise my config.xml file and attach as soon as I've time to edit it for any personal information)
-
@chamilton_ccn Saved me. Thanks a lot!
-
-
@gertjan Had another zero bytes config file event. I'm relieved 2.5.2 Snapshots have gone a long way to solving this.The new code does find an uncorrupted config file as intended.
I did a clean install in migrating to 2.5.2 snapshots.
However, this issue with the initial corruption remains a mystery. I noted in the change report mention of corruption possibly being caused by the shaper and so disabled that in my setup but I've had the issue reoccur regardless.
The sequence of events I have is this:
Reboot
All interfaces come up for a very short while, then crash to power off.
Boot (needing attendance to power up)
Reboot - system finds the good config and all happy (but no crash report)pfSenseConfigurator * list itemNo config.xml found, attempting last known config restore. @ 2021-06-03 09:36:06 * list itempfSense is restoring the configuration /conf/backup/config-1622706943.xml @ 2021-06-03 09:36:07I can see the scenario coming eventually when I'm not near the office to physically power up the router. I'm wondering power supply? Though that wouldn't be consistent with other users here reporting similar issues.
Is there anything at all I can do to help diagnose the cause of the crash and corruption?
-
I tend to think the issue is related to 'power'.
That's a hard one to debug, as, for myself, I never reboot my pfSense. It neither powers down without me knowing about it.
And the day I happened because I've been ripping out the wrong cables, I will reboot with the console activated, me watching the boot process, looking for 'unusual' messages.Also, I'm still not using SSD for pfSense, but old fashioned hard disks. These have stickers with the install date, and after 3 à 5 years will get changed. I'm now on my fourth disk change iteration.
@dilligaf said in Pfsense 2.5 stacks at boot with dots:
I can see the scenario coming eventually when I'm not near the office
That one is valid for everybody I guess. The presence of an UPS will lower the risks, though.
I'm lucky, as the company where I works is a 24/24 - 365/365 and there there is always someone that could "push a button". Because of this, the inverted Murphy's law implies : it won't happen. -
@gertjan How on earth this can be related to power when it is happenig on a vm, and the host is there, with no downtime/disk events.
-
@netblues I've not generated a zero bytes config file since I disabled ClamAV over 3 weeks ago. Pfsense rock solid on a 2.5.2 snapshot every since.
Now I've said that it'll probably happen this morning. -
@dilligaf So you had clamav running on freepbsd pfsense?
WHY?
-
@netblues Running antivirus never seems like a bad idea to me.
I noticed that it triggers a config file write immediately after boot at about the same time I was getting crashes and zero bytes config. May be unrelated, who knows.
I decided to disable as much as possible as the frequency of these zero bytes config was becoming a nuisance. Hey, the router's stable at the mo so I'm happy.
-
@dilligaf said in Pfsense 2.5 stacks at boot with dots:
Running antivirus never seems like a bad idea to me.
Ask yourself : what does the "antivirus " see ?
Packets that come into an interface, like, for example WAN, are about 1k5 bytes in size, they are ALL heavily TLS 'encoded' (so not visible at all for whatever process running on 'pfSense'), and transmitted on a LAN interface.
Or do you think that tools like (example) pfBlockerNGdevel starts to download executables instead of DNSBLfeeds ? And executes them ?
If you even think this is possible, do the right thing : remove pfBlocker from your router.Or some one takes over your pfSense at root level (and kills the anti vir first as he could) and starts installing nasty spy wares ?
Maybe, if the antvirus is proxying all mail traffic
AND
this traffic is send received in 'clear', then an antivir could actually be useful.But let me check my mail server, if there are still mails today that are send 'clear' :
TLS error : scanner-21.ch1.censys-scanner.com[162.142.125.128] 92.118.160.41.netsystemsresearch.com[92.118.160.41] --------- 456 TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits) 245 TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits) 17 TLSv1.2 with cipher ECDHE-RSA-AES128-SHA256 (128/128 bits) 9 TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits) 9 TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits) 6 TLSv1 with cipher DHE-RSA-AES128-SHA (128/128 bits) 5 TLSv1.2 with cipher AES256-SHA (256/256 bits) 4 TLSv1.2 with cipher DHE-RSA-AES256-SHA (256/256 bits) 3 TLSv1.2 with cipher ECDHE-RSA-AES256-SHA384 (256/256 bits) 2 TLSv1.2 with cipher RC4-SHA (128/128 bits) 1 TLSv1.1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)Humm : mails are all hidden behind at least a "TLSv1" layer which means you can't see a thing on a router. If you could, turn around NOW. The guys in your back are from the NSA, or Mossad, or worse. Two things can happen in the next minute : you'll be extreme rich (and you killed the entire Internet), or you're dead.
Care about the planet, stop burning these billions of CPU cycles for nothing, and remove 'antivir' software. If you want one, use it on the 'end device' like your PC.
And no, a (router || firrewall) != ( NAS || mail server || anything else)
-
@gertjan Thanks for that. I also fully understand already that ClamAV isn't going to see encrypted traffic. This is way off topic and I haven't come here for this.
Guys... What I'd really wanted to understand how these zero bytes config files happen and how to stop them. Its a PITA. Has anybody got ideas with that?
-
@chamilton_ccn You are my hero. I was ready to jump off the balcony, until I saw your post and it solved my problem.
Thank you!
