Installation Error, Fatal Trap 12 - devfs_populate_loop

[twibowo]

Hi Guys,

I have an issue during pfsense 2.1.3 installation with the error: "Fatal trap 12: page fault while in kernel mode"

The error mention something about process stopped at "devfs_populate_loop+0x21d"

I have read and browse around the forum, and I understand it seems that this error is general message (does not specifically point out the issue), more of FreeBSD error than pfsense's, and therefore there are still a few possibilities of the root causes.

I have attached the screenshot of the error including the "trace" result.

The installation is using USB flashdisk, because my server does not have CD-rom.
The system specification:

• Dell PowerEdge R300
• Intel Xeon CPU E3113 @3.00GHz
• 64-bit
• Memory 2GB ECC DDR2
• System Memory Speed 667 MHz

Below are the steps that I have tried:
1. Tried different image below, but still getting same error:

• pfSense-memstick-2.1.3-RELEASE-i386-20140501-1552
• pfSense-memstick-2.1.3-RELEASE-amd64-20140501-1552
• pfSense-memstick-2.1.2-RELEASE-i386-20140410-0523
• pfSense-memstick-2.1.2-RELEASE-amd64-20140410-0541

2. Tried the workaround in below links but still getting same error:
https://doc.pfsense.org/index.php/Tuning_and_Troubleshooting_Network_Cards

kern.ipc.nmbclusters="131072"
hw.bce.tso_enable=0
hw.pci.enable_msix=0

3. Tried to unplug the Broadcom Network Cards (2 pieces) from my server but still getting same error.
FYI my server has 4 network interfaces. 2 are built in with the motherboard, another 2 are broadcom network cards.

Please find below for the attachments which shows the screenshot.
Kindly let me know if you guys have any idea how to solve this, or ever experience similar issue.

Thank you,
Best regards


[divsys]

Have you tried booting from different USB slots on the system?

I seem to recall someone having a similar issue and resolving with that.  There may even be an internal USB slot you could try, although sometimes they need to be enabled in the BIOS.

The other option is to try and temporarily hook up a CDROM drive, I've sometimes plugged in an old IDE CD on a cable when necessary.  I don't know if that model of Dell has any free SATA or IDE ports you could try.  Again, the BIOS may need a tweak or two.

In general I've used quite a number of old Dell units as pfsense routers and they work reasonably well, if a bit noisy and power hungry.
But usually the price is right for just that reason  :)

[twibowo]

Hi Divsys,

Thanks for your suggestion,
Today I have tried to do below:

1. Try different different USB stick
   –-> able to boot, but getting the same error as above
2. Try all USB ports, external 4 ports, and internal 1 port
   ---> able to boot, but getting same error as above
3. Try install using USB portable CD-rom drive, use Live-CD install image
   ---> able to boot, but getting same error above
4. Try install pfsense option 2, "Boot pfsense with ACPI disabled"
   ---> the installation stopped at "IPsec: Initializated Security Association Processing" . [Please find attached screenshot for more detail.]
5. Try disabling ACPI by adding hint.acpi.0.disabled="1" in the boot loader.conf
   –-> the installation stopped at "IPsec: Initializated Security Association Processing" . [Please find attached screenshot for more detail.]

For all above tests, I have tried to use version 2.1.3, each i386 version and amd64, with same result.

Do you have any more idea what could be the problem?
Do you ever see these kind of errors before?

Thank you,
Best regards.

[divsys]

Well that's fairly ugly  :P

Do you know the history of this unit, was it working properly etc?

It may be worth running some diagnostics, at least a memtest and/or a burnin to make sure you're not fighting a hardware problem.

I haven't personally run into this scenario, but someone else here might be able to chime in.

[stephenw10]

It's not clear here what you are installing to. You say you are using USB because the cdrom doesn't work but are you then installing to a hard drive? Does it crash out when you try to boot from the hd after installing or during the initial boot from the install media?

I would guess this is an issue with the drive controller in the server. Is it a hardware raid option of some type?

Steve

[twibowo]

@divsys:

Well that's fairly ugly  :P

Do you know the history of this unit, was it working properly etc?

It may be worth running some diagnostics, at least a memtest and/or a burnin to make sure you're not fighting a hardware problem.

I haven't personally run into this scenario, but someone else here might be able to chime in.

Yes, the hardware is fine, as it is running Antlabs as wireless gateway.
Actually I am trying to convert it into pfsense.
When I stop the installation,  it will return to boot from HDD and is still running on Antlabs without any issue.

Nevertheless, it is a good idea, I will try to run some diagnostics just to make sure.

@stephenw10:

It's not clear here what you are installing to. You say you are using USB because the cdrom doesn't work but are you then installing to a hard drive? Does it crash out when you try to boot from the hd after installing or during the initial boot from the install media?

I would guess this is an issue with the drive controller in the server. Is it a hardware raid option of some type?

Steve

Sorry it if my explanation is not clear.

Yes, it is an old server which does not have any CD-ROM drive.
I am trying to install using USB flashdisk, as well as portable USB CD-rom drive.

It can boot from the USB successfully.
On the "Welcome to Pfsense!" page, and I choose "3. Boot Pfsense Using USB Device".
The boot process would continue, and stopped with "Fatal trap 12: page fault while in kernel mode"
The error mention something about process stopped at "devfs_populate_loop+0x21d"
Please find attached screenshot (P1020857)

Meanwhile,
When I choose "2. Boot pfsense with ACPI disabled",
the boot process would continue and stopped at  "IPsec: Initializated Security Association Processing"
Please find attached screenshot (P1020866)

I am not very sure about your question :
"I would guess this is an issue with the drive controller in the server. Is it a hardware raid option of some type?"

Could you elaborate more?
Thank you,

Best regards.


[stephenw10]

The R300 was available in many configurations including several different RAID options. Yours may not be supported. You can probably just disable the RAID mode to treat it as a SATA controller.
However if that is the case I wouldn't expect it to crash during the boot.

There is this:
@https://doc.pfsense.org/index.php/Boot_Troubleshooting#Vendor-Specific_Issues:

Certain Dell Blade servers may hang at boot if you have the system's virtual USB media enabled. If you disable the virtual media in the BIOS, it should boot normally.

Steve

[twibowo]

Hi Steve,

Please find attachment for the screenshots.
I think in my server there is no RAID, also the virtualization feature is disabled. I cannot find any other settings in BIOS to disable virtual USB media.
In the screenshot also is showing the SATA controller.

Do you have any other ideas?

Meanwhile, currently testing this method and I will update you again:
http://djlab.com/2010/06/installing-pfsense-directly-to-hard-drive-with-vga-and-no-cd-rom/

Thank you,
Best regards,

[stephenw10]

That's the wrong virtual device setting, that's for direct CPU access in a virtuial machine environment.

The setting you are looking for is for remote management where you can attach a USB device to the management machine and have it appear as a local device to the server. I don't have a server I can look in for reference but it will be in the remote management options somewhere.

That blog post is old but still mostly holds true. You can swap out the HD install to it on another machine and swap it back. You will probably get a mountroot> error when you first boot but it's easy to workaround.

Steve

[charliem]

The 'Fatal trap 12' is a very broad message.  Some previous pfSense investigation of it is here:
https://forum.pfsense.org/index.php?topic=68590.0

My guess would be RAM (have you run a memtest for a few hours or so?), or virtual media as Steve suggested.

[twibowo]

Hi Guys,

Just want to update regarding this thread. I tried many things, including replacing HD but still could not solve the issue.

Then I add another 2G RAM (become 4G RAM) into this server, then I install VM (ESXi), and pfsense on top of that VM.

It works fine.

So just FYI guys if you see similar issue, you may want to consider using VM instead.

Thanks all for help.