PfSense Crashed

limecat

I just upgraded 3 instances to the Thursday build, and I've started experiencing this as well.

Theyre all configured differently, and the only things in common are

OpenVPN client export
iPerf
They have OpenVPN configured (but disabled on one of them)

It seems that messing with the firmware upgrade page is a sure-fire way to trigger a crash. One of them seems stable as long as I dont log into it.

The problem only started happening after a few hours of working on (2 of) the boxes, then all of a sudden all 3 started acting unstable. I dont think its just the webGUI, I had a number of crashes while launching daemons from SSH as well. Right now i have a ping -t running against one and I have ceased attempting to log into the GUI, and it seems stable.

Not sure how helpful that is.

monkfish

Um, me again…

I think anybody chosing to run the DEVELOPMENT version should be doing so in the strict knowledge and understanding that there could be issues. I do.

Ensure your backup/DR/contingency plan is robust, thats basic. Carry out own testing before releasing to a production or working rig, that's basic as well.

I dont think its right however "light-hearted" it seems to criticise the project for a broken DEVELOPMENT version. But there we go. Thats my opinion.

Constructively - would the developers perhaps consider PULLING the broken release to negate more people being affected?

Nulani

In my case it doesn't reboot: it simply freezes and refuses to accept any input. Strangely, it was working fine yesterday. Not that it matters much: I've been using it on my secondary (backup) firewall to provide IPv6 connectivity. My primary one is still on 2.0.X.

It probably would be an idea to pull the broken release.

sgw

Re-installed image from Mon Apr 22 04:52:47 EDT 2013 … to get back online for now.
That works so far ;)

I will follow this thread to see when it is safe to do another update to the latest snapshot.

Thanks, Stefan

xbipin

24th april snaps r safe, im using it from past few days

markuhde

@monkfish:

I dont think its right however "light-hearted" it seems to criticise the project for a broken DEVELOPMENT version. But there we go. Thats my opinion.

Constructively - would the developers perhaps consider PULLING the broken release to negate more people being affected?

Umm, I DIDN'T criticise it (since I was the one who said what I said was light-hearted). I was saying how great pfSense was. I was light-heartedly criticising those of us who got burned by this snap :) (I'm in that group, the snaps have been so close to production-ready that I thought nothing of clicking the upgrade button. My fault, it's development)

limecat

In my case, attempting to upgrade thru the GUI triggered reboots 90% of the time. This should fix the problem if anyone has run into it (run from SSH / console)

For i386:

8
fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/i386/pfSense_HEAD/updates/pfSense-Full-Update-2.1-BETA1-i386-20130423-1530.tgz
exit
13
2
/root/pfSense-Full-Update-2.1-BETA1-i386-20130423-1530.tgz
y

For amd64:

8
fetch http://snapshots.pfsense.org/FreeBSD_RELENG_8_3/amd64/pfSense_HEAD/updates/pfSense-Full-Update-2.1-BETA1-amd64-20130423-0841.tgz
exit
13
2
/root/pfSense-Full-Update-2.1-BETA1-amd64-20130423-0841.tgz
y

This has been tested on a remote system over SSH, and works fine.

I should add that if you are on the affected version from thursday, you need to downgrade ASAP; on one of my (virtual) systems the problem progressed until the vm no longer booted up. It seems to get worse, probably due to repeated dirty unmounts of the filesystem.

rcfa

@limecat

I can confirm the "getting worse" part, and that the GUI access makes it even less stable.
However there seem to be other things, maybe VPN related, that made my system not stable enough to be recoverable from the CLI, because by the time it was up and I did an slogin, it was about to crash.

So the only way I could do this was to disconnect all network cables (physically), so there were no network events (packets, pings, VPN down, etc.) and then do a restore of a full backup from the physical console, which was a bit tricky due to the kind of hardware I use. (No keyboard and video port on the outside of the case).

So your suggestion may not work for everyone.

limecat

If you are in that state where it wont boot, this should work. I have used it to remotely restore 3 boxes so far, and it seems to work well.

Get an ISO of a "good" version (2.0.3, 2.1 as of april 23). Boot up to it, and select "recovery". Pick your drive, and continue.
You will need to re-assign your interfaces to their adapters, dont worry about getting all of them correct as we will restore the config.
Once you are at the standard "menu", run the following:

8
cp /tmp/hdrescue/cf/conf/config.xml /cf/conf/config.xml
cp /tmp/hdrescue/cf/conf/config.xml /conf/config.xml
rm /tmp/config.cache
exit

Your config should now be loaded. Manually assign the proper IPs to your interfaces, and you should have proper web-gui access again. Log in, and make a backup of your config.

Continue with the installation, which should preserve your now in-memory configuration.

I HIGHLY recommend that you A) confirm that the downloaded configuration is correct and that B) cat /cf/conf/config.xml shows your configuration. Make SURE you have backups before proceeding with the install, which will involve unmounting and wiping your existing partition.

monkfish

@markuhde - i know you didnt. was somebody else mangling my earlier comment about pfsense rocks. which it does!

Moving swiftly on - all my 32-bit builds are happy running Thu Apr 25 09:08:19 EDT 2013, which I think was the last snapshot prior to the broken one. All good this end!

lucky

Looking at today's build log, seems like the error from the weekend is gone (I think it was a sig 15 during a make world). A build seems to be running now.

jits

Ahh..whew!! That's a good crash!

At least it wasn't someone on the network trying to play master hacker.

markuhde

Anyone dare load today's build yet?

mdima

@markuhde:

Anyone dare load today's build yet?

Why, the 24th April build is running soooooo well… :D

eri--

The changes that caused the crashes have been reverted.
So the new snapshots are safe.

I would have been interested on your backtraces shown in the info when you rebooted pfSense, if you can get those.

mdima

I confirm… after a remote update the box is still there...

Thank Ermal to you and all the staff!

Michele

rcfa

@ermal:

The changes that caused the crashes have been reverted.
So the new snapshots are safe.

I would have been interested on your backtraces shown in the info when you rebooted pfSense, if you can get those.

So was it the build process crashing that caused the issue, or a code change?
If code change, what was it? Just curious as to what has such strange effects.

Oh, yes, and also, so far no issues with the new build…

markuhde

Thank you everybody! Updating now (no one's on the campground today so it's perfect for this).

jits

Hi.

How do you do a backtrace?

I'm doubtful if I can do it now. I re-installed from scratch and my latest backup were from December 2012. Serves me right!

In any case, on that particular system running an intel atom 1.3Ghz processor with Realtek daughterboard NIC card, all is well when rebooted with no network cables connected. The second I connect the LAN cable, there is a huge amount of data that starts scrolling up the screen, then it stops for half-a-sec, and reboots. Will do this repeatedly.

When cable is already connected, the screen halts at either NTP Configured or WAN Configured, then you hear the sound it makes when there is supposedly a successful reboot or restart.

lucky

Just tried today's build - 2.1-BETA1 (amd64) built on Mon Apr 29 09:14:28 EDT 2013.

So far, so good.