pfsense 2.7.2-RELEASE (amd64 VM) crashes once in a while

wingyiulam

Hi all, my vm pfsense 2.7.2 runs fine, and I recently encountered crashes once in a while after installing snort and pfblockingNG. The plugins run okay, but it maybe the cause of crashing. below is a partial error log, I have no clue how to uploading the original log to forum.

Thanks
Wing

<118>2024-06-14T14:34:27.776444-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776517-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776569-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776637-07:00 pfSense.unwing.arpa snort <118>2024-06-14T14:34:27.776688-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776755-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776807-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776857-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776921-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.776996-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777048-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777099-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777163-07:00 pfSense.unwing.arpa snort <118>2024-06-14T14:34:27.777236-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777288-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777366-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777437-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777498-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777574-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777623-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777675-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777735-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.777821-07:00 pfSense.unwing.arpa snort <118>2024-06-14T14:34:27.777875-07:00 pfSense.unwing.arpa snort <118>2024-06-14T14:34:27.777939-07:00 pfSense.unwing.arpa snort <118>2024-06-14T14:34:27.777979-07:00 pfSense.unwing.arpa snort <118>2024-06-14T14:34:27.778041-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.778102-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.778153-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.778204-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.778269-07:00 pfSense.unwing.arpa snort 3808 - - <118>2024-06-14T14:34:27.778321-07:00 pfSense.unwing.arpa snort <6>igc1: promiscuous mode disabled Memory in use: 17488 bytes
Config Statistics:
No of allocs: 44
3808 - - POP Preprocessor Statistics
Max concurrent sessions : 0
Used Memory : 0
No of Frees : 0
Used Memory : 17568
Total memory used : 17568
Memory in use: 17568 bytes
No of frees: 0
Memory in use: 17568 bytes
3808 - - ===============================================================================
Max concurrent sessions : 0
IMAP Session
No of Allocs : 0
IMAP Config
No of Frees : 0
Total Statistics:
No of allocs: 6
Config Statistics:
No of frees: 0
3808 - - Total buffers allocated: 0
3808 - - Total buffers released: 0
3808 - - Total freed file mempool: 0
3808 - - Total released file mempool: 0
Total Statistics:
No of frees: 0
Memory in use: 48 bytes
No of frees: 0
No of allocs: 4
3808 - - ===============================================================================

Fatal trap 12: page fault while in kernel mode
cpuid = 1; apic id = 01
fault virtual address = 0xfffffe088fcce190
fault code = supervisor write data, page not present
instruction pointer = 0x20:0xffffffff80fa3f8b
stack pointer = 0x28:0xfffffe0084528e50
frame pointer = 0x28:0xfffffe0084528e80
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 7 (pf purge)
rdi: fffff8008f541dc0 rsi: fffffe008f59e000 rdx: 000000000002e00a
rcx: fffffe088fcce190 r8: 000000008e6ea24d r9: 0000000020510000
rax: 0000000000000000 rbx: fffffe008de02020 rbp: fffffe0084528e80
r10: 000000005c798c5b r11: 00000000816ed0e3 r12: fffffe008fcce1b0
r13: fffff8008f541dc0 r14: fffff8008f541dc0 r15: 000000000002e00a
trap number = 12
panic: page fault
cpuid = 1
time = 1718428896
KDB: enter: panic

stephenw10

Do you have the full crash report?

wingyiulam

@stephenw10 sure thing, my bad, I didn't realize that there is an upload file option.

textdump.tar.0 info.0

stephenw10

Backtrace:

db:0:kdb.enter.default>  bt
Tracing pid 7 tid 100109 td 0xfffffe008de02020
kdb_enter() at kdb_enter+0x32/frame 0xfffffe0084528b30
vpanic() at vpanic+0x163/frame 0xfffffe0084528c60
panic() at panic+0x43/frame 0xfffffe0084528cc0
trap_fatal() at trap_fatal+0x40c/frame 0xfffffe0084528d20
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe0084528d80
calltrap() at calltrap+0x8/frame 0xfffffe0084528d80
--- trap 0xc, rip = 0xffffffff80fa3f8b, rsp = 0xfffffe0084528e50, rbp = 0xfffffe0084528e80 ---
pf_unlink_state() at pf_unlink_state+0x17b/frame 0xfffffe0084528e80
pf_purge_expired_states() at pf_purge_expired_states+0x188/frame 0xfffffe0084528ec0
pf_purge_thread() at pf_purge_thread+0x13b/frame 0xfffffe0084528ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe0084528f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0084528f30
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

But clearly in pf_purge.

Unfortunately the Snort logging has obliterated any other logs in the message buffer.

Seen once before here: https://redmine.pfsense.org/issues/13417

You are seeing this multiple times though? Identical crash? Do you have any other crash reports?

driver8

Similar problem here just this morning. v2.7.2 has been running for quite a while but crashes infrequently at random times. It's running on Intel Atom D2500, Intel NICs, no hardware crypto support, lightly loaded. Also have Snort and pfBlockerNG running. I've been poking around trying to find a cause and would appreciate if someone could suggest which logs would be most revealing.

I did catch this sequence back in March, for what it's worth:
Mar 18 17:58:37 kernel Copyright (c) 1992-2023 The FreeBSD Project.
Mar 18 17:58:37 kernel ---<<BOOT>>---
Mar 18 17:58:37 syslogd kernel boot file is /boot/kernel/kernel
Mar 18 17:56:37 snort 17674 [1:2402000:6945] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.242.226.45:33450 -> xxx.xxx.xxx.xxx:49172
Mar 18 17:56:08 snort 17674 [1:2402000:6945] ET DROP Dshield Block Listed Source group 1 [Classification: Misc Attack] [Priority: 2] {TCP} 185.242.226.40:58588 -> xxx.xxx.xxx.xxx:48443

wingyiulam

@stephenw10 I see, I don’t have other crash reports; I can save it out if it happens again. My “Pfsense + config” VM works fine in my main servers. I move and host the “Pfsense + config” on a dedicated mini pc since May, and I also try and setup snort and pfblockingNG, random crash occurs. It has been crashed 3 times in a month if I recall correctly. I don't mind removing snort or pfblockingNG if they are the cause of crashes.

stephenw10

Well the first thing would be to collect at least 2 crash reports and compare them. If they are identical (or close to) then it's almost certainly a software issue that should be fixed.

It's possible it could be Snort triggering something if you're running in blocking mode.

bmeeks

The only thing that immediately comes to my mind here is the fact Snort calls the "expire table" option of pfctl when the option to automatically clear blocked hosts on an interval is enabled. The crash appears to happen in the pf code that seems part of that "expire table" option (based on the crashing function names). That would point to a potential issue in either pf or pfctl itself and Snort's only involvement is to simply be the process calling pfctl for the operation. Snort does also call libpfctl to kill states when it blocks an IP and the "kill states" option is enabled in Snort.

driver8

@bmeeks Blocked hosts set to clear in 1 day, Snort blocking kill states is ON. Will keep monitoring for more crashes.