PfSense keeps crashing (Fatal trap 12: page fault while in kernel mode)

telendt

Sorry for imprecise title but I don't know where the problem comes from and I'm not able to parse the crash logs myself.
This is my first pfSense build, I bought this hardware specifically for pfSense, it runs pfSense 2.4.1-RELEASE installed directly on the SSD nothing else:

Supermicro A2SAV-L (MBD-A2SAV-L-O with Atom E3940 on board)
8GB of RAM (Kingston KVR16LS11/8)
64GB M.2 SSD (Transcend TS64GMTS800)

System crashes multiple times a day (I have not yet seen an uptime higher that 24h) since it first run (I switched it on about a week ago)
Here's the crash log

I've already ran memtest86 and it went fine (no memory errors detected).
The installation is pretty standard, the only things I changed from default install options were filesystem (ZFS), network interfaces order (igb0 for LAN and igb1 for WAN), enabling AES-NI for crypto and PowerD for burst mode (I played with other stuff but at this point everything else should be pretty standard).

As mentioned before, this is my first build and I have no prior experience with pfSense. I have not tried running different builds (maybe I should?).

Any suggestions how to fix the problem are welcome.

Birke

i see in the crashlog that you use intel network chips with msix.
try the tipps mentioned here: https://forum.pfsense.org/index.php?topic=69486.0

chidgear

It might be your installation media or your hardware too (it could be defective from factory). Try an installation using another fresh downloaded copy of pfsense (dont use the copy you already downloaded), if that fails too, try doing it in a healthy mechanical HDD, if keeps crashing, try another ram, if the issue keeps going, you could say that the issue is in you motherboard or CPU.

Greetings!

telendt

@Birke:

i see in the crashlog that you use intel network chips with msix.
try the tipps mentioned here: https://forum.pfsense.org/index.php?topic=69486.0

I tried the following changes (saved to /boot/loader.conf.local + reboot):


kern.ipc.nmbclusters="1000000"
hw.pci.enable_msix=0

But it didn't help.

Here's the latest crashlog.

Also, [this doc page[\url] says:

If the firewall does panic, and the panic message contains a backtrace that mentions things like memory allocation, mbuf, uma_zalloc_arg, or similar, then it may be crashing due to mbuf exhaustion.

But I don't see these things being mentioned :(

What I see though are those 2 mentions:

Fatal trap 12: page fault while in kernel mode

and

Fatal trap 1: privileged instruction fault while in kernel mode

CPU is not overheated (temp below 40°C), RAM passed memtest86 test.

@chidgear:

It might be your installation media or your hardware too (it could be defective from factory). Try an installation using another fresh downloaded copy of pfsense (dont use the copy you already downloaded), if that fails too, try doing it in a healthy mechanical HDD, if keeps crashing, try another ram, if the issue keeps going, you could say that the issue is in you motherboard or CPU.

Ouch… that would be a very lengthy and hard process (especially that I don't have any spare parts).

Cheers.](https://doc.pfsense.org/index.php/Unexpected_Reboot_Troubleshooting)

w0w

https://forum.pfsense.org/index.php?topic=136939.msg749218#msg749218
Try 2.4.2

Birke

@telendt:

I tried the following changes (saved to /boot/loader.conf.local + reboot):
kern.ipc.nmbclusters="1000000"
hw.pci.enable_msix=0
But it didn't help.

according to the crashlog the devices still use msix.

try these options:

kern.cam.boot_delay=10000
kern.ipc.nmbclusters=1000000
hw.igb.num_queues=1
legal.intel_ipw.license_ack=1
legal.intel_iwi.license_ack=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0

these work with our igb-devices.

telendt

@Birke:

according to the crashlog the devices still use msix.

try these options:
kern.cam.boot_delay=10000
kern.ipc.nmbclusters=1000000
hw.igb.num_queues=1
legal.intel_ipw.license_ack=1
legal.intel_iwi.license_ack=1
hw.pci.enable_msix=0
hw.igb.enable_msix=0
these work with our igb-devices.

I just copied your config and rebooted the machine. An hour later I had another crash. No MSIX interrupts this time but plain MSI. Still the same panic though:

Fatal trap 12: page fault while in kernel mode

@w0w:

https://forum.pfsense.org/index.php?topic=136939.msg749218#msg749218
Try 2.4.2

You mean 2.4.2-DEVELOPMENT (snapshot)? Yeah, I'll try that. Playing with different builds is the only thing that's left I guess (except of trying with different hardware, which I don't really have).

w0w

@telendt:

Yeah, I'll try that. Playing with different builds is the only thing that's left I guess (except of trying with different hardware, which I don't really have).

Not really only one, but definitely better to try it first. You can also try to disable TSO AND LRO on SYSTEM>ADVANCED>NETWORKING, but if it's a freebsd bug 213903 then there is no other way, only to try the new build.

telendt

So I reinstalled pfSense from 2.4.2-RC snapshot (20171117) and used a different boot drive (some spare WD Red spinning NAS drive). After some time I experienced a first panic, "Fatal trap 1: privileged instruction fault while in kernel mode".
I applied options mentioned by @Birke, and rebooted the system.
Sysctl complained that all hw.igb.* tunables are read only, but the system ran without a crash for over 30h. Not sure if these options helped of if I had luck.

To get a better use of my systems CPU I enabled PowerD (Hidaptive) and AES-NI acceleration for crypto. Some time after that I experienced another crash, similar to previous one + mentioned "Fatal trap 12: page fault while in kernel mode".

At this point I have no idea where does the problem come from. I contacted another forum member that owns this motherboard and he told me that his system runs stable without any tunable changes :(