PfSense Crashed

limecat

In my case, attempting to upgrade thru the GUI triggered reboots 90% of the time. This should fix the problem if anyone has run into it (run from SSH / console)

For i386:

8
fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/i386/pfSense_HEAD/updates/pfSense-Full-Update-2.1-BETA1-i386-20130423-1530.tgz
exit
13
2
/root/pfSense-Full-Update-2.1-BETA1-i386-20130423-1530.tgz
y

For amd64:

8
fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/amd64/pfSense_HEAD/updates/pfSense-Full-Update-2.1-BETA1-amd64-20130423-0841.tgz
exit
13
2
/root/pfSense-Full-Update-2.1-BETA1-amd64-20130423-0841.tgz
y

This has been tested on a remote system over SSH, and works fine.

I should add that if you are on the affected version from thursday, you need to downgrade ASAP; on one of my (virtual) systems the problem progressed until the vm no longer booted up. It seems to get worse, probably due to repeated dirty unmounts of the filesystem.

rcfa

@limecat

I can confirm the "getting worse" part, and that the GUI access makes it even less stable.
However there seem to be other things, maybe VPN related, that made my system not stable enough to be recoverable from the CLI, because by the time it was up and I did an slogin, it was about to crash.

So the only way I could do this was to disconnect all network cables (physically), so there were no network events (packets, pings, VPN down, etc.) and then do a restore of a full backup from the physical console, which was a bit tricky due to the kind of hardware I use. (No keyboard and video port on the outside of the case).

So your suggestion may not work for everyone.

limecat

If you are in that state where it wont boot, this should work. I have used it to remotely restore 3 boxes so far, and it seems to work well.

Get an ISO of a "good" version (2.0.3, 2.1 as of april 23). Boot up to it, and select "recovery". Pick your drive, and continue.
You will need to re-assign your interfaces to their adapters, dont worry about getting all of them correct as we will restore the config.
Once you are at the standard "menu", run the following:

8
cp /tmp/hdrescue/cf/conf/config.xml /cf/conf/config.xml
cp /tmp/hdrescue/cf/conf/config.xml /conf/config.xml
rm /tmp/config.cache
exit

Your config should now be loaded. Manually assign the proper IPs to your interfaces, and you should have proper web-gui access again. Log in, and make a backup of your config.

Continue with the installation, which should preserve your now in-memory configuration.

I HIGHLY recommend that you A) confirm that the downloaded configuration is correct and that B) cat /cf/conf/config.xml shows your configuration. Make SURE you have backups before proceeding with the install, which will involve unmounting and wiping your existing partition.

monkfish

@markuhde - i know you didnt. was somebody else mangling my earlier comment about pfsense rocks. which it does!

Moving swiftly on - all my 32-bit builds are happy running Thu Apr 25 09:08:19 EDT 2013, which I think was the last snapshot prior to the broken one. All good this end!

lucky

Looking at today's build log, seems like the error from the weekend is gone (I think it was a sig 15 during a make world). A build seems to be running now.

jits

Ahh..whew!! That's a good crash!

At least it wasn't someone on the network trying to play master hacker.

markuhde

Anyone dare load today's build yet?

mdima

@markuhde:

Anyone dare load today's build yet?

Why, the 24th April build is running soooooo well… :D

eri--

The changes that caused the crashes have been reverted.
So the new snapshots are safe.

I would have been interested on your backtraces shown in the info when you rebooted pfSense, if you can get those.

mdima

I confirm… after a remote update the box is still there...

Thank Ermal to you and all the staff!

Michele

rcfa

@ermal:

The changes that caused the crashes have been reverted.
So the new snapshots are safe.

I would have been interested on your backtraces shown in the info when you rebooted pfSense, if you can get those.

So was it the build process crashing that caused the issue, or a code change?
If code change, what was it? Just curious as to what has such strange effects.

Oh, yes, and also, so far no issues with the new build…

markuhde

Thank you everybody! Updating now (no one's on the campground today so it's perfect for this).

jits

Hi.

How do you do a backtrace?

I'm doubtful if I can do it now. I re-installed from scratch and my latest backup were from December 2012. Serves me right!

In any case, on that particular system running an intel atom 1.3Ghz processor with Realtek daughterboard NIC card, all is well when rebooted with no network cables connected. The second I connect the LAN cable, there is a huge amount of data that starts scrolling up the screen, then it stops for half-a-sec, and reboots. Will do this repeatedly.

When cable is already connected, the screen halts at either NTP Configured or WAN Configured, then you hear the sound it makes when there is supposedly a successful reboot or restart.

lucky

Just tried today's build - 2.1-BETA1 (amd64) built on Mon Apr 29 09:14:28 EDT 2013.

So far, so good.

freebee

sbflush_internal cc 0 || mb 0xc2651c00 || mbcnt 4352

Clear-Pixel

@lucky:

Just tried today's build - 2.1-BETA1 (amd64) built on Mon Apr 29 09:14:28 EDT 2013.

So far, so good.

Not showing a Apr 29 09:14:28 build ???
http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/i386/pfSense_HEAD/livecd_installer/?C=M;O=D

It's only showing ???
pfSense-LiveCD-2.1-BETA1-i386-20130429-0914.iso.gz 29-Apr-2013 09:43 85M

Was planning on doing a fresh install but I'm a bit Scared. :-\

Maybe tomorrow do a fresh install off the latest snap shot ???

lucky

@Clear-Pixel:

It's only showing ???
pfSense-LiveCD-2.1-BETA1-i386-20130429-0914.iso.gz 29-Apr-2013 09:43 85M

Was planning on doing a fresh install but I'm a bit Scared. :-\

Seems like you looked at the i386 build vs amd64, which is what I'm running. Not sure how i386 is working out.

Clear-Pixel

Oops my bad…

@rcfa:

If code change, what was it? Just curious as to what has such strange effects.

My bet would be on this one:
https://github.com/pfsense/pfsense-tools/commit/3fe8a9cd9297d64ab243fe3f38cea8b4ef147899

cmb

@alexh:

My bet would be on this one:
https://github.com/pfsense/pfsense-tools/commit/3fe8a9cd9297d64ab243fe3f38cea8b4ef147899

That's it, disabling those kernel patches gets us back to where we were last week before the bad snapshot.