PfSense crashes ever few weeks - log is blank

FarmerB3d

The board hovers around 44C so don't think heat is a problem. While it is fanless it is in a very perforated case: http://linitx.com/images/products/M350_Universal_Mini-ITX_Enclosure_main_large.jpg

Yes, the SSD is an old repurposed one but was healthy (SMART) when I took it out of the previous machine. I'll check SMART again and see what it says.

I had another look at the log file - it's not missing sections. It has whole block of information out of order - as if it wrote in the middle of the file, then the end and then back in the middle. I can only assume the partition table is dodge… Will focus on that.

kejianshi

CLOGs… Perhaps?

https://doc.pfsense.org/index.php/Why_can%27t_I_view_view_log_files_with_cat/grep/etc%3F_%28clog%29

Don't break it thinking you have an issue there. At first glance, this seems normal to me.

I'd leave it alone and wait for more crashes since you removed that USB thingy. Give it a chance to be stable. Unless its already crashing again?

FarmerB3d

ok, it may well be that. The "writing in the middle" continues.

Mar 17 08:10:06 pfSense php-fpm[2384]: /index.php: Successful login for user 'admin' from: 10.10.50.X
Mar 17 08:10:06 pfSense php-fpm[2384]: /index.php: Successful login for user 'admin' from: 10.10.50.X
Mar 17 09:06:43 pfSense sshd[14419]: error: PAM: authentication error for root from 10.10.50.X
Mar 17 09:06:43 pfSense sshd[14419]: error: PAM: authentication error for root from 10.10.50.X
Mar 17 09:06:49 pfSense sshd[14419]: Accepted keyboard-interactive/pam for root from 10.10.50.X port XXXX ssh2
ad_status: Syncing firewall
Feb 1 12:10:30 pfSense kernel: ovpns2: link state changed to DOWN
Feb 1 12:10:30 pfSense check_reload_status: Reloading filter
Feb 1 12:10:30 pfSense kernel: ovpns2: changing name to 'tun2'
Feb 1 12:10:31 pfSense check_reload_status: Syncing firewall

I see most of my log files are exactly 500KB so it stops at that point and writes from the top again.

Thanks for that - removes my biggest worry.

It's not crashing and I don't expect it to crash for a long time. This is the second or third time it has crashed since early June - most of which was running Alpha nightlies.

Also, the crash, as far as I can see, is not a panic (as I know it). The system is still up and working but just really badly.

thanks
FB

kejianshi

Well - with nightlies I'd be expecting some glitches anyway. Basically you are beta testing, which is nice but its certainly not the way I would start out. I think stable releases are a better bet for someone just getting to know pfsense.

Did I say beta testing… I should have said alpha testing :o

FarmerB3d

TBH, I was surprised how good the Alphas were. Only issue was this one I have now. I needed Alpha because the hardware I bought was not supported by the previous release of BSD.

kejianshi

Are the alphas using a different version of BSD than 2.2? (I'm not sure - I haven't tried)

heper

afaik the early ones used 10.0 , while now they are on 10.1

FarmerB3d

Right, well, didn't this go badly!

I woke up this morning at 3am (sick 18 month old). Walked downstairs and notices text scrolling past at a rapid rate on the pfSense monitor. "vfs error <something-or-other>"

Well, I guess this is it then. Internet is still working. Try log into pfSense and nudda. Error writing to /tmp/session bla bla.
I can however ssh into it. Supidly, I did not get the config file because I religiously backup after I make any changes. Of course I did…

Right, reboot - cannot. Pull power - does not reboot. FreeBSD hangs just after bring the NICs up. Oh well, we're toast.

Grab spare SSD out the drawer (as you do)
Swap it with the now-dead one.
Download the latest installer (USB live image)
Burn it
Boot from USB
ta-da. We're back. ;D

Now to restore the last backup I have... hmm, looks a bit old. :-
Not to worry, it will have the bulk of the config. It's out by about a month.

Restore - what a nightmare :'( The restore was just not happy with the NICs. Each reboot it would ask me again which is the WAN/LAN etc. Eventually it stops asking.

None of the VPNs are working correctly, the gateways are a mess, the firewall rules and NAT rules are a shambles. I start patching them together again but cannot get them working. Dreading a redo of them all I go get a coffee. What has changed... why would the restore not work? I've done it a few times without trouble...

Damn you kejianshi / cmb, damn you to hell :P
Take out the USB NIC they said. It's all the USB NIC's fault they said.
I had forgotten about it.
Realising that the missing NIC is the issue with the restore I put it back in, restored again and ta-daaa. Perfect restore :D Happy bunny.

So, disk was on it's way out. I would still like to try and get the latest config off it because my backup is one set of changes short. (Bad me)
The restore works perfectly if the hardware is identical. If not, it's a bit of a headache.

Having said all that - to rebuild a busted firewall in an hour and be back up and running is remarkable. Full credit to the devs and community of pfSense for making such an awesome bit of kit.

Cheers,
FB</something-or-other>

kejianshi

haha

FarmerB3d

Well, this is odd. Second SSD is now complaining the same as the first.
Both SSDs used to sit in my NAS (ZFS) as cache drives so either they both got porked while in there or this motherboard is killing them or psSense is killing them.
Both SSDs are 4 years old (found the invoice, was hoping on warranty).

Guess a new on is needed and will see from there.

pfSense is still running so no rebooting until spare drive arrives…