[v2.3 & v2.4] Kernel crash with Fatal trap 12: page fault while in kernel mode

CDuv

That's why I am completely lost here…

Here is a map of the network installation:

/–----►( Internet )◄-------------
| ▲ |
| | |
┌───•──┐ ┌───•──┐ ┌•─────┐
│ISP B •--\ │ISP C │ │ISP A │
│router│ | │router│ │router│
└──────┘ | └──•───┘ └────•─┘
| | |
(WAN_B) (WAN_C) |
| | |
┌─────────•────────•────────────────┐ (WAN_A)
│ igb2 igb3 │ |
│ │ |
│ Lanner FW-7551 running pfSense │ |
│ │ |
│ VID3&VID4 │ |
│igb0 igb1 igb4 igb5 │ |
└─•───────•───────────•─────────•───┘ |
| | | | |
(LAN) (WAN_A) (LAN_GUEST1) (SYNC) |
| | & | |
| | (LAN_GUEST2) | |
| | | | |
| | | x |
| | | unused yet: left |
| | | for future HA |
| | | pfSense setup |
| | | |
┌─•───────•───────────•────────────────────┐ |
│ p7 p13 p12 │ |
│VID1 VID2 VID3&VID4 │ |
│ │ |
│ D-Link DGS managed switch p9•--/
│ VID2│
│ VID1 VID3 VID4 │
│ p8 p10 p11 │
└──•────────────────────────•─────•────────┘
| | |
| | ---► (LAN_GUEST2)
▼ |
(LAN) ---► (LAN_GUEST1)

Particularities:

I am using a VLAN for WAN_A (VID2) because ISP A's router only have one Ethernet port to connect my pfSense router to and I want to install a backup pfSense box: Thus I am using a managed switch and VLANs to virtually multiply Ethernet ports so that I can plug the backup server (this setup was working fine for years on previous pfSense server)
ISP A's router requires interface to be forced at "100 Mbps Full duplex", so igb1 and switch ports p13 and p9 are set at 100 Mbps Full duplex.
igb4 interface is the "host" of two virtual interfaces that uses VLANs : VID3 and VID4 for LAN_GUEST1and LAN_GUEST2
This is a multi-WAN with load balancing scenario
But WAN_B and WAN_C are currently unused/disconnected (interfaces are forced to "down")

Networks:

LAN: Network used by the clients (10.0.0.0/8)
- pfSense has 10.0.0.5 (igb0 interface)
- pfSense also uses a Virtual IP alias (igb0 interface) 10.0.0.254 that the LAN clients uses as their gateway
WAN_A: Network between pfSense and the router from my ISP "A" (80.26.35.12/30)
- ISP A's router has 80.26.35.13
- pfSense has 80.26.35.14 (igb1 interface), using 80.26.35.13 as the gateway
WAN_B: Network between pfSense and the router from ISP "B" (192.168.2.0/24)
WAN_C: Network between pfSense and the router from ISP "C" (192.168.3.0/24)
LAN_GUEST1: Network used by guests clients (192.168.10.0/24)
- pfSense has 192.168.10.5 (igb0 interface)
- pfSense also uses a Virtual IP alias (igb4 interface) 192.168.10.1 that the LAN_GUEST1 clients uses as their gateway
LAN_GUEST2: Network used by other guests clients (192.168.20.0/24)
- pfSense has 192.168.20.5 (igb0 interface)
- pfSense also uses a Virtual IP alias (igb4 interface) 192.168.20.1 that the LAN_GUEST2 clients uses as their gateway
SYNC: Future HA synchronization network

VLANs (VID<->Network mapping):

VID1: LAN
VID2: WAN_A
VID3: LAN_GUEST1
VID4: LAN_GUEST2

Today (non working day) it crashed about every two or three hours.

beppo

I got also your described problem with the supermicro board. The openvpn service is not enabled but I got daily reboots after crashs. For my thinking the issue is somehow related to the 2.3.x version as versions before ran months without problems.

kpa

If the crash is always the same or looks very similar to the others it's likely that the problem is a software one, random crashes with wildly varying types of reports is an indication of a hardware problem instead.

CDuv

I don't know how to read crash reports.
Sometime the file "/var/crash/info.0" has:

Panic String: sbflush_internal: cc 0 || mb 0xfffff800643e2800 || mbcnt 2304

sometime it does not.
But crash report always has:

Fatal trap 12: page fault while in kernel mode

igb driver is Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k

w0w

Try different pfSense version, for example 2.2.6 or even 2.4-BETA, if problem persists its looks more like hardware issue, the fastest way is to try same config and version on other hardware, but it is possible only if you have another one.

CDuv

OK, I'll try other versions: I have other hardware.

I guess testing v2.3.3 wouldn't do any better (since I doubt this unknown bug would get fixed).

I rather try v2.4 instead of v2.2 (to avoid loosing any feature v2.3 brought): would my v2.3.2-p1 configuration file be accepted on v2.4?

CDuv

Should I disable "Flow Control" (as the Wiki says)

w0w

Yes, 2.4 can use backup config from previous versions. You can also try any other settings you find, not only flow control.

CDuv

So, I tried 2.4.0-BETA v20161113-2326 (pfSense-CE-memstick-serial-2.4.0-BETA-amd64-20161113-2326), in 15 hours it failed twice (9h and 15h later).

When I logged in in the WebConfigurator to get the first crash report I got it fine:

Crash report begins. Anonymous machine information:

amd64
11.0-RELEASE-p3
FreeBSD 11.0-RELEASE-p3 #180 8fb831d(RELENG_2_4): Sun Nov 13 23:31:20 CST 2016 root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense

Crash report details:

Filename: /var/crash/bounds
1

Filename: /var/crash/info.0
Dump header from device: /dev/ada0s1b
Architecture: amd64
Architecture Version: 2
Dump Length: 580517888
Blocksize: 512
Dumptime: Tue Nov 15 04:00:16 2016
Hostname: pfsensebox.example.com
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 11.0-RELEASE-p3 #180 8fb831d(RELENG_2_4): Sun Nov 13 23:31:20 CST 2016
root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
Panic String: page fault
Dump Parity: 1903556642
Bounds: 0
Dump Status: good

Filename: /var/crash/info.last
Dump header from device: /dev/ada0s1b
Architecture: amd64
Architecture Version: 2
Dump Length: 580517888
Blocksize: 512
Dumptime: Tue Nov 15 04:00:16 2016
Hostname: pfsensebox.example.com
Magic: FreeBSD Kernel Dump
Version String: FreeBSD 11.0-RELEASE-p3 #180 8fb831d(RELENG_2_4): Sun Nov 13 23:31:20 CST 2016
root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense
Panic String: page fault
Dump Parity: 1903556642
Bounds: 0
Dump Status: good

Filename: /var/crash/minfree
2048

but when sending report to developers, it got it's second crash which I don't want to send (because it would maybe re-crash the system) but I got this on the serial console:

Enter an option:
Message from syslogd@pfsensebox at Nov 15 10:01:15 …
pfsensebox php-fpm[84587]: /index.php: Successful login for user 'admin' from: 10.0.1.53
panic: sbsndptr: sockbuf 0xfffff8010d811518 and mbuf 0xfffff8010ddc6000 clashing
cpuid = 6
Uptime: 6h0m53s
Dumping 567 out of 8135 MB: (CTRL-C to abort) ..3%..12%..23%..32%..43%..51%..63%..71%..82%..91%
Dump complete
99
TAB Key on Remote Keyboard To Entry Setup Menu
MB-7551 Ver.AE0 03/28/2014
Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc.
Press ~~or <esc>to enter setup.

(.. many empty lines ..)

|oading /boot/defaults/loader.conf serial port
/IOS drive C: is disk0 /boot/config: -S115200 -D
BIOS 619kB/2081240kB available memory

FreeBSD/x86 bootstrap loader, Revision 1.1
(root@buildbot2.netgate.com, Wed Aug 3 08:04:25 CDT 2016)

(.. many empty lines ..)

/boot/entropy size=0x100017b93e]a0 |
Booting… _/ _| ___ _ __ ___ ___
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
| ./The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p3 #180 8fb831d(RELENG_2_4): Sun Nov 13 23:31:20 CST 2016
root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64</esc>~~

I have a 567MB file "/var/crash/vmcore.0".

CDuv

Other crash:

Fatal trap 12: page fault while in kernel mode
cpuid = 5; apic id = 0a
fault virtual address = 0x78
fault code = supervisor read data, page not present
instruction pointer = 0x20:0xffffffff80d6632c
stack pointer = 0x28:0xfffffe01ec7d9930
frame pointer = 0x28:0xfffffe01ec7d9990
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (irq289: igb4:que 5)
trap number = 12
panic: page fault
cpuid = 5
Uptime: 1h33m1s
Dumping 568 out of 8135 MB:..3%..12%..23%..31%..43%..51%..62%..71%..82%..91%
Dump complete
99
TAB Key on Remote Keyboard To Entry Setup Menu
MB-7551 Ver.AE0 03/28/2014
Version 2.16.1242. Copyright (C) 2013 American Megatrends, Inc.
Press ~~or <esc>to enter setup.

(.. many empty lines ..)

|oading /boot/defaults/loader.conf serial port
/IOS drive C: is disk0 BIOS 619kB/2081240kB available memory

FreeBSD/x86 bootstrap loader, Revision 1.1
(root@buildbot2.netgate.com, Wed Aug 3 08:04:25 CDT 2016)
\

(.. many empty lines ..)

syms=[0x8+0x17b620+0x8+0x17b93e]a0 data=0xaad7b8+0x4c60e8
/boot/entropy size=0x1000 __ ___ ___
Booting…|___ \ / _ \ ' / __|/ _ \
Copyright (c) 1992-2016 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
|_| The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-RELEASE-p3 #180 8fb831d(RELENG_2_4): Sun Nov 13 23:31:20 CST 2016
root@buildbot2.netgate.com:/builder/ce/tmp/obj/builder/ce/tmp/FreeBSD-src/sys/pfSense amd64</esc>~~

I will now try v2.3.1 (I think I recall that it was not crashing that much at that time).

2.4.0-BETA crashed few seconds after I changed Virtual IP settings (to disable box running v2.4.0-BETA and switch production to the box running v2.3.1).

w0w

I can be wrong but it mostly looks like ECC memory failure. Please replace your memory and test again.

CDuv

It occurs on 2 identical brand-new box so I doubt faulty hardware is the cause (it is still possible indeed but lowly possible).

w0w

Two boxes? Does it mean you don't use same pieces of hardware when testing on both machines? For example harddrive/CF or USB stick?

CDuv

It means I have 2 identical servers with exact same model of component (1X SSD, 1x RAM memory stick) in each one.

v2.3.2 was tested on both servers.
v2.40-BETA (which performs a bit better : only 2-3 crashes per day) was only tested on box 2.
v2.3.1 was only tested on box 1.

I never swapped any piece (would it be RAM or SSD)
Used a couple of USB memory stick for the installations : it is actually the only piece of hardware that was shared between servers.

CDuv

Funny fact, when unplugging network cables (when I want to swap production from one server to another: for the tests): the server crashes…

w0w

Are there some BIOS/UEFI options regarding OS installation compatibility?
Did you try to install 2.4 on ZFS with GPT-UEFI (it should work on latest builds)? I am not sure may be its related to some power savings or anything else you can find in BIOS or UEFI settings related to power savings.
It would be good to disable all CPU power saving modes except common C1 mode for testing purposes.

W4RH34D

bad cable? faulty plug on the other end with out of spec voltages?

CDuv

@w0w:

Are there some BIOS/UEFI options regarding OS installation compatibility?
Did you try to install 2.4 on ZFS with GPT-UEFI (it should work on latest builds)? I am not sure may be its related to some power savings or anything else you can find in BIOS or UEFI settings related to power savings.
It would be good to disable all CPU power saving modes except common C1 mode for testing purposes.

pfSense 2.4 was installed on MBR: I'll try GPT…

I have no access to BIOS on this appliance which is pre-prepared for pfSense (bought to a local appliance reseller): serial console does not allow me to enter BIOS (I see the "Press ~~to enter…" but pressing the [Del] key does nothing).~~: Got it working.
The BIOS says the following about CPU:

EIST (GV3) : Disable
P-stat Coordination : Package (cannot be modified)
TM1 : Enable
TM2 Mode : Adaptative Throttling (cannot be modified)
CPU C State : Disable
Enhanced Halt State : Disable (cannot be modified)
ACP C2 : Diable (cannot be modified)
Monitor/Mwait : Enable (cannot be modified)
L1 Prefetcher : Enable
L2 Prefetcher : Enable
Max CPUID Value Limit : Disable
Execute Disable Bit : Enable
AES-NI : Enable
Turbo : Enable (cannot be modified)
Active Processor Core : All

But, on the pfSense config, PowerD is disabled and AC Power, Battery Power and Unknown Power settings are all set to "Hiadaptive" (did not touched theses after 2.4.0-BETA installation).

It is advised to disable power saving modes in the BIOS/UEFI?

beppo

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 06
fault virtual address	= 0x5e00000000
fault code		= supervisor read data, page not present
instruction pointer	= 0x20:0xffffffff80d80b00
stack pointer	        = 0x28:0xfffffe00a1644b60
frame pointer	        = 0x28:0xfffffe00a1644b80
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 75463 (pfctl)

Got the same error once a day with the same motherboard in 2.3.x. Bios settings weren't changed since the first install of 2.x. I don't think it is a hardware issue but I also have no clue how to solve the issue but installing 2.2 again.

CDuv

Yeah! I am not alone!

Is your network architecture partially similar to mine? Do you have a lot of users?

If your workaround is to downgrade to 2.2 it is a lead for some debugging and a possible fix…

I see bug #4689 (Panic/Crash "sbflush_internal: cc 4294967166 || mb 0 || mbcnt 0") is similar but is marked as resolved for 2.3…
Original bug report on FreeBSD bug tracker is still open and someone reported it ran into the issue it a month ago.