SOLVED: 1.2.3-RC3 embedded not booting on alix 1c/ alix 1d/ alix3c3

PeeZee

After some hours of burning images to cf cards, flashing alix bios versions, testing, retesting and testing again, I can conclude the following:

pfSense 1.2.3-RC3 embedded does NOT boot on the alix 1c / alix 1d / alix 3c3
boot loader works, freebsd system starts booting, system gets stuck at the line "Starting device manager (devd)"
(The "done" is not appearing after this command)

If I pull out the cf card and put it in an alix2c3 (also most recent bios version), it boots without problems and asks me whether I want to assign VLANs.

Any idea what could be causing this ?

To be complete :
alix1c is running latest bios version at time of writing (the version called 'beta', dated januari 2009)
alix1d is running latest bios version at time of writing (the version called 'beta', dated januari 2009)
alix3c3 is running latest bios version at time of writing (the version called 'beta', dated februari 2009)
UPDATE: I also tried with the other available BIOS versions, specifically the ones dated from 10/06/08, 6/16/08, 8/21/07, no difference
UPDATE2: Same results with one of the latest snapshots (pfSense-1.2.3-1g-20091105-0026-nanobsd.img.gz)

pfsense image is gunzipped, then written to cf disk using dd if=xxx.img of=/dev/sdb bs=1M (I've tried 16k, 1M and 2M blocksizes to rule out image write errors)

If you need more info, just ask, I can test and try whatever you want

AhnHEL

pfSense requires 2 network interfaces to run, all the devices that you are having problems with only have one.

Never tried it personally but if you have a vlan capable switch then you can get away with using vlans and one network interface. Read the forums and I'm sure you'll find more info on running pfsense with a one network interface device and vlans.

PeeZee

@onhel:

pfSense requires 2 network interfaces to run, all the devices that you are having problems with only have one.

Actually, I'm using a mini pci wireless card as the second nic, which connects to a wireless netwerk for the WAN. I have this setup running already for a while now, but wanted to test 1.2.3-RC3 on the alix boxes
UPDATE: wireless card is a Ubiquiti cm9, atheros chipset, was already supported way back in 1.2-RC3 so should be detected without problems. Can't verify that because I never get to a usable shell, but can't see any reason why it should not work as second nic

Not that it matters though, because the boot process on the affected alix boxes doesn't even get far enough to ask me to create VLANs

@onhel:

Never tried it personally but if you have a vlan capable switch then you can get away with using vlans and one network interface. Read the forums and I'm sure you'll find more info on running pfsense with a one network interface device and vlans.

I've done that a few times already. My cisco catalyst 2950XL switch here at home is running 6 VLAN's linked to 6 VLAN interfaces in my pfsense box, works like a charm. Still, that won't help me here since the boot process gets stuck sooner than that.

AhnHEL

Is it possibly not being detected due to the change in the ath driver when 1.2.3 RC3 was released?

http://forum.pfsense.org/index.php/topic,19691.msg101391.html#msg101391

If that is the case then you'd be back to one IF

Try another wifi card, I'm using a WLM54G-HP which has been working good for me.

jimp

@onhel:

Is it possibly not being detected due to the change in the ath driver when 1.2.3 RC3 was released?

Actually if you try a snapshot from the last day or so, wireless is back to the stock drivers from 7.2. It was hoped that knocking the wireless down to an earlier revision would cure some bugs, and it did, but it also brought other problems along.

PeeZee

@onhel:

Is it possibly not being detected due to the change in the ath driver when 1.2.3 RC3 was released?

http://forum.pfsense.org/index.php/topic,19691.msg101391.html#msg101391

If that is the case then you'd be back to one IF

Try another wifi card, I'm using a WLM54G-HP which has been working good for me.

The wireless card I'm using was already supported in v1.2-RC3 over a year ago.
If I understood correctly the drivers that were reverted in 1.2.3-RC3 were only introduced in 1.2.3-RC2 and as such should have no impact on that.

EDIT:
I started this post on my mobile phone on screen keyboard yesterday, didn't manage to finish it properly. Here is what I was trying to say:

First of all, thanks for trying to help, I really appreciate it.
I am however 99% sure that this has nothing to do with the mentioned alix boards having only 1 nic because the wireless card is not recognized.
If that was the problem, pfsense would still boot up fully and then tell me during the interface assignments that I need 2 NIC's.
Since the pfsense 1.2.3-RC3 doesn't boot completely, I don't even get to the VLAN or interface assignment screen.

I would like to stress that this appears to be a show stopper for 1.2.3-RC3 on all the alix boards that have a vga connector. pfsense nanobsd version DOES NOT work on them.
Since pfsense embedded 1.2.2 works without problems on these alix boards, this is definitely a regression.

I'm hoping someone else or maybe one of the devs can at least confirm that this is a problem with pfsense nanobsd version and the above mentioned alix boards, so that I'm sure it is not something I did wrong. As soon as this issue is confirmed, I feel more comfortable creating a bug report for it.

cmb

We have a ton of ALIX boards, but none of the ones that have VGA so I don't know whether those work. They should. It works fine on all of the 15-20 different embedded platforms we have.

PeeZee

Is there anything else I can test/run/break/download/install/… to give you more details about this ?
It would really be a pity if the new embedded pfsense version would not be compatible anymore with these alix boards

jimp

When they stop, are they completely unresponsive on serial?

Does pressing ctrl-T show anything on the screen?

PeeZee

Sorry for the delayed answer, there was another project that required my attention the last 2 days.

Just tested what you asked, when the "Starting device manager (devd)…" appears and the boot process stops, I can still press RETURN on the serial console and line feeds appear. So the OS is not frozen, it is apparently waiting for that boot step to finish.

When I press Ctrl-T, it prints the line
load: 0.11 cmd: php 131 [nanslp] 0.89u 1.10s 0% 20028k

Hope this helps :-)

PeeZee

Some more info, based on input from jimp on irc:

I booted the cf card in an alix 2c3, and edited /etc/rc.bootup commenting out start_devd:

        echo "Starting device manager (devd)...";
        mute_kernel_msgs();
//        start_devd();
        set_device_perms();
        unmute_kernel_msgs();
        echo "done.\n";

After this, I put the cf card back in my alix3c3 and it boots up just fine.

However, I have no idea what I broke by doing this, so for all readers out there DO NOT DO THAT YOURSELF if you don't know what you're doing (like me :-) )

Could anybody tell me what start_devd() does and why I would need it ?

jimp

It starts devd

/rimshot

Really if you look in /etc/inc/config.inc it really just exec's /sbin/devd

PeeZee

This is getting stranger by the minute

I wanted to find out which step in start_devd it was getting stuck on, so I edited /etc/rc.bootup and /etc/inc/config.inc to echo some output after each line of code executed in start_devd()

The last command which is started is sleep(1); in start_devd() in /etc/inc/config.inc
After that the boot process is stuck, it seems the sleep command never returns…

???

PeeZee

So there is really not a single person out there running pfsense on an alix1c / alix1d / alix3c3 ?

Cool, I always wanted to be special and unique in some way… :-)

PeeZee

2 weeks and no further response… And RC3 is in prerelease now apparently

Can I stress again that pfSense 1.2.3 nanobsd DOES NOT work on the above mentioned alix boards.

1.2.2 worked perfectly on them so this is definately a HUGE step backwards !

What do I have to do to get some attention to this issue ?
I tried debugging myself but got nowhere with my limited BSD knowledge

Please, could someone at least test this themselves so the issue is at least confirmed and just maybe someone can take a look at why this is happening ?

My apologies if I sound irritated, I know this is a forum ran by volunteers, and I know pfSense is a free product. I've been very happy with pfSense over the last few years that I've been using it.
It just astonishes me that this bug seems to get no attention when I discover such a big problem, a real show stopper for using pfSense on these alix boards, and definately a regression from the previous version.
If the forum is not the place to discuss this, please let me know where to go.

Thanks!

Perry

You could fill out a bug report http://redmine.pfsense.org/

Also if it's possible to get remote access to your box, you could join ##pfsense on IRC and ask for help.

jimp

There was a thread on the FreeBSD-stable list a week or two ago about the issue.

http://www.mail-archive.com/freebsd-stable@freebsd.org/msg106186.html

(We already have the patch mentioned in the thread, by the way)

Part of the problem is that this appears to be a FreeBSD issue, not necessarily a pfSense issue. There may not be anything that we can do about it, it may need to be fixed upstream:

http://doc.pfsense.org/index.php/Policy_on_FreeBSD_issues

keylevel

I've installed 1.2.3-RC3 on an Alix 3D3 (updated 3C3) and have had exactly the same problem.

I found that selecting the "no acpi" boot allowed the system to boot.

I then set the power management mode in the BIOS to APM and then re-installed (not sure if that was needed, but I changed the install media as well). This also boots ok.

PeeZee

@keylevel:

I've installed 1.2.3-RC3 on an Alix 3D3 (updated 3C3) and have had exactly the same problem.

I found that selecting the "no acpi" boot allowed the system to boot.

I then set the power management mode in the BIOS to APM and then re-installed (not sure if that was needed, but I changed the install media as well). This also boots ok.

Confirmed ! This solves the issue, alix3c3 boots right up with BIOS power management set to APM.

(I'll test on the 1c1 and 1d1 later and confirm whether fix works on those boards as well)

Thanks !