Pfsense crashing randomly pfsnese plus 24.03

stephenw10

Backtrace:

db:1:pfs> bt
Tracing pid 2 tid 100039 td 0xfffff80001950740
kdb_enter() at kdb_enter+0x33/frame 0xfffffe00c2af18f0
panic() at panic+0x43/frame 0xfffffe00c2af1950
trap_fatal() at trap_fatal+0x40f/frame 0xfffffe00c2af19b0
trap_pfault() at trap_pfault+0x4f/frame 0xfffffe00c2af1a10
calltrap() at calltrap+0x8/frame 0xfffffe00c2af1a10
--- trap 0xc, rip = 0xffffffff80f246e2, rsp = 0xfffffe00c2af1ae0, rbp = 0xfffffe00c2af1b70 ---
tcp_m_copym() at tcp_m_copym+0x62/frame 0xfffffe00c2af1b70
tcp_default_output() at tcp_default_output+0x1294/frame 0xfffffe00c2af1d60
tcp_timer_rexmt() at tcp_timer_rexmt+0x53c/frame 0xfffffe00c2af1dc0
tcp_timer_enter() at tcp_timer_enter+0x101/frame 0xfffffe00c2af1e00
softclock_call_cc() at softclock_call_cc+0x12e/frame 0xfffffe00c2af1ec0
softclock_thread() at softclock_thread+0xe9/frame 0xfffffe00c2af1ef0
fork_exit() at fork_exit+0x7f/frame 0xfffffe00c2af1f30
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe00c2af1f30
--- trap 0, rip = 0x720065, rsp = 0x7200650064, rbp = 0x6f004200200073 ---

This is this bug which we are working currently:
https://redmine.pfsense.org/issues/15684

It looks like it was triggered when igb0 flapped. What is that interface? The Snort spam in your logs has obscured anything else.

Steve

ssjucrono

@stephenw10
Thank you! Igb0 is my lan interface.

I did have snort installed but have uninstalled it prior to this crash.

Thank you

stephenw10

Is this the first time it has crashed? Are you able to trigger it on demand in any way?

cboenning

@ssjucrono As we're experiencing the same (same signature, same Redmine bug to track, another forum Topic though) I'd be interested .. do you run any of the following packages:

acme
aws-wizard (pre-installed on pfsense+)
frr
ipse-profile-wizard (pre-installed on pfsense+)
netgate_firmware_upgrade (pre-installed on pfsense+)
node_exporter
openvpn-client-export (pre-installed, I think)
zabbix-agent64

ssjucrono

@cboenning said in Pfsense crashing randomly pfsnese plus 24.03:

Thank you! yes I run these 2. though I can remove openvpn as I do not use it anymore. I have switched to tailscale
acme
openvpn-client-export

cboenning

@ssjucrono no no. Don’t remove anything. I was just interested if there might be some similarities to our setup.

I think those 2 packages are pretty unspectacular given they’re not really doing „anything network“

ssjucrono

@cboenning
yeah, I don't need them. I removed acme and openvpn exporter as I have never used them.

thank you

cboenning

@ssjucrono you may want to opt in to enabling „full core dumps“ as outlined here (https://forum.netgate.com/topic/188861/24-03-crashing-again/19) and provide them to @stephenw10 and/or Redmine to get this debugged eventually though.

stephenw10

Yup, that. If you're able to enable full core dumps that will help a lot here. However be aware that you need to have enough SWAP available for the dump file which will be the size of the used RAM.

An alternative that may also help would be to run the debug kernel:

https://docs.netgate.com/pfsense/en/latest/troubleshooting/debug-kernel.html

That may show additional errors before the panic.

cboenning

@ssjucrono you may want to check the Redmine issue for a workaround (https://redmine.pfsense.org/issues/15684#note-14)

stephenw10

Yup let us know if disabling net.inet.tcp.sack.enable works to prevent it.

For reference that looks like:
Screenshot from 2024-08-23 15-13-54.png

ssjucrono

@stephenw10 Thank you for the update. I don't have net.inet.tcp.sack.enable in my system tunables? should I add it? or just leave it as is?

stephenw10

Yes you will need to add that. It's not a default tunable.

ssjucrono

@stephenw10 I have not seen this crash in awhile. I will set this though.

Maybe it was caused by my Unraid Docker Containers being backed up each night. So they are all stopped and then started within about 12minutes. I do get a flapping warning from arpwatch each night when this occurs. Perhaps that was the cause of the initial crash?

stephenw10

I doubt it. But it's unclear what actually triggers it since most users never hit it.

enthu19

@stephenw10 said in Pfsense crashing randomly pfsnese plus 24.03:

Yup let us know if disabling net.inet.tcp.sack.enable works to prevent it.

For reference that looks like:

It works. I had random crashes, but once I added "net.inet.tcp.sack.enable=0", I haven't experienced any crashes.

stephenw10

Great. That should be patched in the next release.

hulleyrob

I wonder if this is what hit me the other day.

Will post the dumps to see if they are of any use.

Will upgrading to 24.11 fix this? I normally just update the system patches (currently 2.2.11_17)

Thanks
Rob
info.0

enthu19

@hulleyrob

Upgrading to latest version always recommended.

or you can try adding this entry in System Tunnable : "net.inet.tcp.sack.enable=0"

I am running 24.11 - Pretty solid

hulleyrob

@enthu19 well 24.03 was until it wasn’t. 6 months ish uptime from memory and then I suspect the ISP changed my WAN IP (PPPOE) and I got a page fault. Thought updating the system packages was an alternative to doing a full upgrade.
Was this system tunable added to 24.11 otherwise I don’t see how upgrading will help with my problem.