Hard locks, panics, and fatal trap 12s unless APIC disabled at bootup

Thunderhacker

I'm at a loss as to where to look to solve this problem. It appears to be a bug in the underlying FreeBSD APIC code. After several hours of hair pulling frustration and troubleshooting I was able to track down that disabling the APIC made the system stable, but at the cost of about 60% of the CPU performance (it's a dual core system, no apic == only 1 core.)

Hardware:

MSI Wind Nettop 100 (Intel Atom 330 dual core CPU)
1GB DDR2 533 RAM stick pulled from a laptop when the laptop was upgraded
320GB Western Digital SATA hard drive
Lite-ON SATA DVDROM drive
Cheap Logitech USB Keyboard

The lockups and crashes happen almost randomly under normal circumstances, happening sometimes before I can even complete a normal installation. This happens on both the 1.2.3 stable and 2.0 BETA pfsense, as well as the 9.0-CURRENT-200912 livefs of FreeBSD (although hard lockups are more common on the latter.) Dropping to the bootloader prompt during bootup and entering 'set hint.apic.0.disabled="1"' makes the system stable on all three.

I can cause an almost immediate Fault Trap 12 using the following method:
1. Boot pfSense 1.2.3-stable
2. Configure VLANs, etc to get to console menu
3. Drop to a shell and enter the following command to put a load on the processor:

dd if=/dev/zero of=/dev/null

4. After a couple of seconds the system crashes with Fault Trap 12 (or sometimes, but rarely, a panic of some other kind)

I have tried a different stick of RAM and the problems continue. (The RAM in it has worked flawlessly in a laptop for over a year.) Memtest86+ passes with no errors, and the system will successfully complete the linpack benchmark (run from the PelicanHPC LiveCD Linux Distro.) The system ran stable long enough to DBAN 2 1TB hard disks, so I tend to think this problem is related to FreeBSD in some way.

I should note that this system lacks a serial port so getting a backtrace isn't possible, unless a USB to serial converter can be used, in which case I'll go and buy one.

The motherboard is at the latest BIOS version released from MSI so a BIOS upgrade is not an option (I haven't tried a downgrade yet.)

If there's something I've missed let me know. I'm more of a Linux person; my only experience with FreeBSD is from using pfSense (which I've been using as my firewall for the past 3 years, may thanks for outstanding software!) As I'm used to troubleshooting Linux it's quite possible I've neglected to check something that should be checked in FreeBSD land.

Thanks in advance to anyone who has any ideas on what to do. I think I'll be able to make do with the system in a crippled state with the APIC disabled so I'm going to go ahead and install and run the 1.2.3 stable release like that. The highest load I can imagine the system seeing is transferring 20-30Mbps over VPN between a wireless client and a file server.

–-- EDIT ----

I forgot to mention, I'm using VLANs and a Gig-E "smart" switch (Netgear GS108T) hence the reason only 1 Ethernet port.

kc8apf

It's been quite a while since I've had to deal with APIC issues, but they almost always boil down to a hardware, firmware, or kernel bug. Have you tried disabling ACPI instead of APIC? Frequently the ACPI tables are only tested with Windows and provide very buggy information to the kernel. You may also want to follow up with the FreeBSD folks as this is certainly an issue with FreeBSD and your hardware, not anything specific to pfSense.

Thunderhacker

I've tried disabling ACPI, that only leads to fewer Fatal Trap 12s but more of the cryptic panics. Still just as unstable in the end. All of my searches suggested disabling ACPI to solve the problem as well. The only reason I found out it was APIC was because when booted in failsafe it was stable. Process of elimination finally sorted out the problem.

I plan on posting to the FreeBSD forums, but thought I'd start a thread here both to see if anyone here has seen/fixed this issue and to serve as a warning to anyone thinking about buying this particular hardware for use with pfSense.

jimp

I haven't seen anything similar but the usual suggestions are to try BIOS updates and disable ACPI, which I see you have tried.

Given the widespread nature of your issues my first guess would be faulty hardware of some kind, starting with RAM.

Thunderhacker

I was thinking faulty RAM as well but it does the same thing with 2 different sticks of RAM and it passes Memtest86+ fine (although I have read that sometimes a bad stick can get past it.) It also has no problems passing the linpack benchmark (very memory intensive.) Both sticks of RAM that I tested it with run stable in a laptop. Linux doesn't seem to have any problems running on it either. Haven't tried installing Windows, but I have a feeling that would run fine as well.

For the record, my cross-post of this thread over at the FreeBSD forums is here: http://forums.freebsd.org/showthread.php?t=9798

Tikimotel

Well from what I did come across reading about atom cpu's and freebsd was that most users have issues with hyperthreading and freebsd spinlock.
Disabling hyperhreading would allow some more stability. (longer uptime)
But the end result is almost always kernel (spinlock) panic (without crash report), sometimes after some hours other times after a month of uptime.
Reading these issues keeps me away from changing my motherboard for a atom based one.

(googled for: "atom hyperthreading panic freebsd")
http://lists.freebsd.org/pipermail/freebsd-stable/2009-September/051983.html
http://hype-o-thetic.com/2009/07/09/freenas-d945gclf2-configuration/comment-page-1/

Thunderhacker

Thanks for the info. I did try disabling hyperthreading in the bios but it didn't help stability any for me. I just went from fault trap 12s to random panics (I don't recall seeing anything about spinlock.) I guess this is what I get for not researching a product extensively before I buy it. (Insert cluestick here…)

For what it's worth I installed Ubuntu + VMware to run a pfSense VM on the box. It's been stable for the past week or so. VPN speeds suffer due to the slow processor (only get about 1MB/s transfer over VPN) but everything else seems fine.

Are VIA C-7 based boards decent? I know they have the hardware AES support so OpenVPN should work a bit better on them. I know I'm straying offtopic, but does anyone know of a place to get low density memory? Every VIA C-7 barebones system I've come across seems to require it.

Again, thanks to everyone for the help.

wallabybob

@Thunderhacker:

I know I'm straying offtopic, but does anyone know of a place to get low density memory? Every VIA C-7 barebones system I've come across seems to require it.

http://www.ebay.com
http://www.crucial.com if you don't mind paying a bit more.

Are you sure about the low density? My C-3 systems (which use an older chipset than the C-7s) will quite happily take 512MB or 1GB RAM sticks. The specs said DDR-2700 RAM but I'm using something like DDR-3200 without apparent problem.

Thunderhacker

You're thinking of speed. Look here: http://reviews.ebay.com/Myth-Low-Density-vs-High-Density-memory-modules_W0QQugidZ10000000001236178 That link does a better job explaining it than I could.

I've been looking at the C-7 barebone systems on NewEgg (example: http://www.newegg.com/Product/Product.aspx?Item=N82E16856107059) and the ones that have comments say that the boards require low density memory. Crucial doesn't list the density of the RAM they sell (if they do I can't find it, I'd be more than happy to buy from them if they listed density; they charge more but have better quality IMHO) and the systems aren't listed in their system configuration tool.

eBay isn't an option for me because I refuse to have anything do to with PayPal. WAY too many horror stories about them for my liking.

wallabybob

Thanks for the pointer to the explanation about high density vs low density. (I think its a poor choice of words to call a 64Mx8 chip low density and a 128Mx4 chip high density but maybe thats just a personal quirk.)

If you would be happy to purchase memory from Crucial (they are retail arm of Micron which is one of the suppliers recommended by the eBay site to which you linked) why not ask them for a memory recommendation for the motherboards/systems in which you are interested. Kingston might be another company you could ask for a recommendation (http://www.ec.kingston.com/ecom/configurator_new/mfrmod.asp?root=anz&LinkBack=http://www.kingston.com/anz

Thunderhacker

I might try that next week if I have some spare time. I just tried the memory search on the page you linked (the site redirected back to the front page) and Jetway isn't even listed.

I agree with you on the density thing. It seems that "density" is a relative term. What's high density today will be low density tomorrow (on a computer upgrade time frame.) I also highly doubt that only 10% of the boards out there will support this cheap high density memory he talks about. I think it's more likely to be closer to 50% or more that support high density. I mainly linked for the explanation of the difference between high and low density memory.

wallabybob

@Thunderhacker:

I just tried the memory search on the page you linked (the site redirected back to the front page) and Jetway isn't even listed.

I suggested that page because there was a link there in this text: _While our list of supported systems is the largest in the memory industry, even we might miss a system or manufacturer along the way. We probably have what you need, though.

To find out, just click here and tell us what system or device you have and we'll find out._
and the link is to http://www.kingston.com/support/newsalesform.asp
(I also didn't see Jetway in their list of supported manufacturers.)

Thunderhacker

Thanks for the link. I filled out the form, it'll probably be tomorrow some time before I get a reply back. It would be nice if Kingston had RAM that would work as they're less expensive than Crucial.

Thanks for all your help. It'll probably be a couple of weeks before I order the VIA box as I'd like to sell off this Atom box first.