Pfsense crashing randomly pfsnese plus 24.03

[ssjucrono]

Running pfsense plus 24.03 on an intel i5 3700k 16GB of RAM, installed on an SSD.

within the last few weeks I am seeing crash on it.

here are the logs I believe

textdump.tar(1).0
info(1).0 .

Thank you

[stephenw10]

Backtrace:

db:1:pfs> bt
Tracing pid 2 tid 100039 td 0xfffff80001950740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00c2af18f0
panic() at panic+0x43/frame 0xfffffe00c2af1950
trap_fatal() at trap_fatal+0x40f/frame 0xfffffe00c2af19b0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c2af1a10
calltrap() at calltrap+0x8/frame 0xfffffe00c2af1a10
--- trap 0xc, rip = 0xffffffff80f246e2, rsp = 0xfffffe00c2af1ae0, rbp = 0xfffffe00c2af1b70 ---
tcp_m_copym() at tcp_m_copym+0x62/frame 0xfffffe00c2af1b70
tcp_default_output() at tcp_default_output+0x1294/frame 0xfffffe00c2af1d60
tcp_timer_rexmt() at tcp_timer_rexmt+0x53c/frame 0xfffffe00c2af1dc0
tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe00c2af1e00
softclock_call_cc() at softclock_call_cc+0x12e/frame 0xfffffe00c2af1ec0
softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00c2af1ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c2af1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2af1f30
--- trap 0, rip = 0x720065, rsp = 0x7200650064, rbp = 0x6f004200200073 ---

This is this bug which we are working currently:
https://redmine.pfsense.org/issues/15684

It looks like it was triggered when igb0 flapped. What is that interface? The Snort spam in your logs has obscured anything else.

Steve

[ssjucrono]

@stephenw10
Thank you! Igb0 is my lan interface.

I did have snort installed but have uninstalled it prior to this crash.

Thank you

[stephenw10]

Is this the first time it has crashed? Are you able to trigger it on demand in any way?

[cboenning]

@ssjucrono As we're experiencing the same (same signature, same Redmine bug to track, another forum Topic though) I'd be interested .. do you run any of the following packages:

• acme
• aws-wizard (pre-installed on pfsense+)
• frr
• ipse-profile-wizard (pre-installed on pfsense+)
• netgate_firmware_upgrade (pre-installed on pfsense+)
• node_exporter
• openvpn-client-export (pre-installed, I think)
• zabbix-agent64

[ssjucrono]

@cboenning said in Pfsense crashing randomly pfsnese plus 24.03:

Thank you! yes I run these 2. though I can remove openvpn as I do not use it anymore. I have switched to tailscale
acme
openvpn-client-export

[cboenning]

@ssjucrono no no. Don’t remove anything. I was just interested if there might be some similarities to our setup.

I think those 2 packages are pretty unspectacular given they’re not really doing „anything network“

[ssjucrono]

@cboenning
yeah, I don't need them. I removed acme and openvpn exporter as I have never used them.

thank you

[cboenning]

@ssjucrono you may want to opt in to enabling „full core dumps“ as outlined here (https://forum.netgate.com/topic/188861/24-03-crashing-again/19) and provide them to @stephenw10 and/or Redmine to get this debugged eventually though.

[stephenw10]

Yup, that. If you're able to enable full core dumps that will help a lot here. However be aware that you need to have enough SWAP available for the dump file which will be the size of the used RAM.

An alternative that may also help would be to run the debug kernel:

https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.html

That may show additional errors before the panic.

[cboenning]

@ssjucrono you may want to check the Redmine issue for a workaround (https://redmine.pfsense.org/issues/15684#note-14)

[stephenw10]

Yup let us know if disabling net.inet.tcp.sack.enable works to prevent it.

For reference that looks like:

[ssjucrono]

@stephenw10 Thank you for the update. I don't have net.inet.tcp.sack.enable in my system tunables? should I add it? or just leave it as is?

[stephenw10]

Yes you will need to add that. It's not a default tunable.

[ssjucrono]

@stephenw10 I have not seen this crash in awhile. I will set this though.

Maybe it was caused by my Unraid Docker Containers being backed up each night. So they are all stopped and then started within about 12minutes. I do get a flapping warning from arpwatch each night when this occurs. Perhaps that was the cause of the initial crash?

[stephenw10]

I doubt it. But it's unclear what actually triggers it since most users never hit it.

[enthu19]

@stephenw10 said in Pfsense crashing randomly pfsnese plus 24.03:

Yup let us know if disabling net.inet.tcp.sack.enable works to prevent it.

For reference that looks like:

It works. I had random crashes, but once I added "net.inet.tcp.sack.enable=0", I haven't experienced any crashes.

[stephenw10]

Great. That should be patched in the next release.