Pfsense keeps crashing

tjabas

@tjabas this is just a copy of the photo I took of the log on my screen, I really can't tell whats wrong

tjabas

@tjabas

Gertjan

@tjabas

When you see 'CAM' and the device is 'ada.....' then you know the 'disk', or communication with the disk has issues.

To rule out the disk, or confirm the issue : use another disk.

tjabas

i managed to get a photo of my screen, i have no clue whats happening.
i have deactivated icap interface,clam antivirus,squid and squidguard, just to start somewhere.

tjabas

@tjabas so you mean that the ssd disk inside the router?
the router is brand new, even though its a china box, i guess that the quality can vary alot.

tjabas

@Gertjan

the disk i have monted in the router now is a msata 256gb, i found another msata at home but only 16gb, i also found a 250gb 2,5" ssd sata disk, but what happens when i remove the faulty disk, and replace it with one of the new ones, will i also loose the boot meny settings or are they stored in the motherboard?
will 16gb be enough?

tjabas

i found an old ssd 256gb harddrive , this is now installed and the system is up running, it will be interesting to see if the problem is still there.

stephenw10

16GB is probably fine. The boot settings will be stored in flash not on the disk. You will have to reinstall pfSense obviously but you must have done that already.
That error is definitely a drive or drive controller problem, almost certainly a bad drive.

Steve

Gertjan

@tjabas said in Pfsense keeps crashing:

will i also loose the boot

No, of course not.
That why you make these regular daily Diagnostics > Backup & Restore Backup & Restore
And that's why you've set up Services > Auto Configuration Backup > Settings

A new disk means of course : you need a install media, an USB drive with the 'latest'.
After install, import de config. Reboot. Done.

@tjabas said in Pfsense keeps crashing:

icap interface,clam antivirus,squid and squidguard

These can produce a lot of disk activity.
If the drive is 'small', a lot of sectors will be rewritten often. That's what kills even the best SSD.

tjabas

thanks for your help, maybee ill have to stop using these programs i use in pfsense, the last pfsense hardware i used lasted for 12years without any problems at all, and this brand new device lasted 2 weeks.

stephenw10

Even the worst current SSD should last waaaaay longer than that. There were some early SSDs that had bad firmware or controllers in which the wear leveling was completely broken. But if that was 256GB it probably wasn't one of them. Just a bad device.

Steve

tjabas

@stephenw10 yes that sucks, someone in this tread mentioned autoconfig backup, i usually do backups and save them on my laptop, but what does the autoconfig backup do? and where do pfsense store that backup?

stephenw10

ACB encrypts the config file and sends it to our servers. The key remains in the firewall, we are unable to see those configs.

It can be configured to backup periodically or at every config change.

https://docs.netgate.com/pfsense/en/latest/backup/autoconfigbackup.html

Steve

Gertjan

@tjabas

If you have some 'always or often on' Windows, Linux, iOS device, look here.

It will login in over SSH, get a copy of the config.xml, keepts the latest xx days

mentisdominus

I was running into the same issue with my SSD. I decided to run pfSense with RAM disk enabled with hourly writes of system logs, RRD, and DHCP leases. The only package that I have running that requires an SSD/HDD is pfBlockerNG. It's been stable for over a week now (8 Days 06 Hours). Whereas before, I would have to hard reset every day.

It might be worth trying doing the same just to see if it works for you. It might save you some time.

I'm hesitant to buy another SSD (I live where returning items isn't so easy) because I found another person on reddit ahcich1 timeout, CAM command timeout who experienced the same issue on the same device running pfSense that I do (HP T620 Plus). That person experienced the same issue with a new SSD in two separate devices of the same model (HP T620 Plus). Before enabling RAM drive, I tried various settings in the BIOS and sysctl no avail.

Gertjan

@mentisdominus said in Pfsense keeps crashing:

I would have to hard reset every day

That's the perfect way to break your drive 'logically' : the file system becomes a mess, the drive partition becomes read-only, which triggers run time failures, as the OS can't write files anymore.
Easy to understand : take your own PC : remove power / remove battery, repeat several times and your system goes into eternal blue screen mode. It's re install time, version 'from the ground up' which mans : partition hard drive, destructive re format, the big total.

A normal (== good ?) pfSense system never reboots or fails.
Netgate devices tend to deliver on that criteria.

mentisdominus

@Gertjan The system was suspended showing the error message along the lines of "Solaris: zpool 'pfSense' has encountered an uncorrectable I/O error the system is suspended". There was no way to login/ssh into it to restart it or enter a keyboard command to restart. ~~Hard resetting~~ Hard reboot was the only way I know how to restart it.

Edit: I previously wrote "hard reset" in my previous comments as instead of "hard reboot"; as in, physically holding down the power button to turn it off and then pressing it again to power it back on. I went back to correct that part.

stephenw10

Yes, from there you can really only reset.

That sounds like a possible drive controller issue though if it happens repeatedly even after reinstalling. Or across multiple drives.