-
Ahh ok, I remember jacking up the fragments on my old hardware and that solved it then. I bet that's what it is. I guess the advanced stuff like that doesn't get transferred over in the config dumps when you move it to new hardware.
I know what the fragments are. There is an NTP pool node behind this pfSense and it sees some heavy UDP connections, among other things...
Thanks for tracking that down!
-
Ok I got another crash today, and today it's not showing the pf frag entries limit reached, but same kernel crash message.
MBUs set to a million
Frag packets set to 25,000.Any other ideas?
-
For kicks I've disabled MSI-X and went to MSI interrupts. We'll see if that has any effect.
/boot/loader.conf.local
hw.pci.enable_msix=0 -
Hmm, that's different crash. Still in the network stack though:
db:0:kdb.enter.default> show pcpu cpuid = 1 dynamic pcpu = 0xfffffe03df1c3380 curthread = 0xfffff8001d50e620: pid 20 "pf purge" curpcb = 0xfffffe03d59d2b80 fpcurthread = none idlethread = 0xfffff80005f40620: tid 100004 "idle: cpu1" curpmap = 0xffffffff82b85998 tssp = 0xffffffff82bb6878 commontssp = 0xffffffff82bb6878 rsp0 = 0xfffffe03d59d2b80 gs32p = 0xffffffff82bbd0d0 ldt = 0xffffffff82bbd110 tss = 0xffffffff82bbd100 db:0:kdb.enter.default> bt Tracing pid 20 tid 100130 td 0xfffff8001d50e620 __mtx_lock_sleep() at __mtx_lock_sleep+0xcd/frame 0xfffffe03d59d2990 pf_detach_state() at pf_detach_state+0x311/frame 0xfffffe03d59d29c0 pf_unlink_state() at pf_unlink_state+0x1ee/frame 0xfffffe03d59d29f0 pf_purge_expired_states() at pf_purge_expired_states+0x6d/frame 0xfffffe03d59d2a40 pf_purge_thread() at pf_purge_thread+0x71/frame 0xfffffe03d59d2a70 fork_exit() at fork_exit+0x83/frame 0xfffffe03d59d2ab0 fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe03d59d2ab0 --- trap 0, rip = 0, rsp = 0, rbp = 0 --- db:0:kdb.enter.default> psNothing in the message buffer to help.....
-
Yeah, I've been reading about how Intel cards suck with MSI-X, so hopefully that's what it is. I guess time will tell.
-
That has not been my experience. Especially with em(4) which is very mature at this point.
Steve
-
https://www.intel.com/content/www/us/en/support/articles/000017506/network-and-i-o.html
Maybe it's the amount of packets I'm pushing through my Gigabit Intel WAN? I'm pushing it pretty hard. 12 hours on MSI so far and no crashes and no degraded throughput or anything. Probably a slight increase on CPU, barely 3% if that.
-
Well, 1 Day 11 Hours 08 Minutes 36 Seconds uptime since going to MSI. Knocks on wood
-
This surprises me as well after reading through this thread - Intel cards are generally top tier and perform very well with pfSense. What specific Intel card / chipset are you using?
-
WAN - Intel Gigabit CT PCI-E Network Adapter EXPI9301CTBLK
LAN - Intel X520-DA2 (The real Intel, not the 10GTek knock off, Multiple VLANs using only one port) -
@Zermus said in Ryzen 3 2200G randomly crashing:
WAN - Intel Gigabit CT PCI-E Network Adapter EXPI9301CTBLK
LAN - Intel X520-DA2 (The real Intel, not the 10GTek knock off, Multiple VLANs using only one port)Which one of those two was / is giving you trouble? Looking at data sheets and product briefs, both of those cards support MSI-X. Also, just to doublecheck, are you running pfSense on bare metal or inside a VM? Have you tried adjusting any of other tunables on the cards or is everything basically running with stock settings?
Come to think of it now, I did have one EXPI9301CTBLK adapter seemingly go bad on me a couple years back. Was running in a Windows desktop machine though. For some reason one day it stopped passing gigabit speeds and nothing I did would fix it (reinstall, change cables, etc.). Replaced with another card and all was well.
I think the chipset in the Intel CT adapter is targeted more towards desktop/laptop applications. If you're interested in replacing it with something a bit higher end, these HP Intel i210 based cards should work well, are fairly inexpensive, and use the igb driver in FreeBSD:
https://www.amazon.com/Intel-I210-T1-Network-Adapter-E0X95AA/dp/B00ESFF2JC/
https://www8.hp.com/us/en/products/oas/product-detail.html?oid=5383389Hope this helps.
-
Hmm, I'll try to swap out the Intel CT with the I210. The weird part is that the system was stable on older hardware with the same NICs, the system was based on an AMD A8-7600 Kavari (Steamroller). Unfortunately the motherboard started flaking out on that system though, which lead to the Ryzen rebuild. Right now with MSI interrupts the system seems stable for about 2 to 3 days but now experiences hard crashes where I get no crash reports.
Extended 12 hour memtest run came up with no errors, so I'm fairly certain hardware is good in this case. I'm starting to wonder if it's just Ryzen and FreeBSD bugs creeping up that I'm hitting.
-
@Zermus said in Ryzen 3 2200G randomly crashing:
Hmm, I'll try to swap out the Intel CT with the I210. The weird part is that the system was stable on older hardware with the same NICs, the system was based on an AMD A8-7600 Kavari (Steamroller). Unfortunately the motherboard started flaking out on that system though, which lead to the Ryzen rebuild. Right now with MSI interrupts the system seems stable for about 2 to 3 days but now experiences hard crashes where I get no crash reports.
Extended 12 hour memtest run came up with no errors, so I'm fairly certain hardware is good in this case. I'm starting to wonder if it's just Ryzen and FreeBSD bugs creeping up that I'm hitting.
That's a good point - I wonder if support / stability will improve with FreeBSD 12.0 (which pfSense 2.5 will use).
-
Try a snapshot and see if it's still crashing in 2.4.4. I have several boxes running 2.5 snaps here without issue currently. Though they are still just code snapshots so things might break.
Steve
-
So I think I've fixed this by trial and error network tuning. I might make a separate thread for fine tuning NTP node or heavy UDP traffic behind pfSense. I'm using an Intel I210 (igb driver) as my WAN (Referred to me as a previous fix in this thread, but I could probably have kept the Gigabit CT in) and 1 port on an Intel X520-DA2 (ix driver) as my LAN. So far with this setup I've been running 3 days stable. I'll update with any further tuning updates if the need arises.
Under System -> Advanced -> Networking I have:
Hardware Checksum Offloading unchecked. (Enabled)
Hardware TCP Segmentation Offloading checked. (Disabled)
Hardware Large Receive Offloading checked. (Disabled)kern.ipc.nmbclusters="1000000" kern.ipc.nmbjumbop="524288" hw.igb.max_interrupt_rate="32000" hw.ix.max_interrupt_rate="32000" net.isr.maxthreads="-1" net.isr.bindthreads="1" net.pf.source_nodes_hashsize="1048576"NOTE: hw.igb and hw.ix are specific to my NIC drivers, so you might have to tune these to your own NIC hardware.
-
@Zermus - Glad things are more stable now. I do think the I210 is a superior card / chipset compared to the CT, but did you ever try plugging the CT back in to see if the instability issues returned?
-
so not a cpu issue?
-
Nope, well sorta. I just had to fine tune the hell out of the kernel to deal with the NTP traffic. I left the i210 in the box, but I'm guessing either card would have worked. Going on 90 days uptime now.
Here is what I ended up with on /boot/loader.conf.local
hw.igb.max_interrupt_rate="216000" hw.ix.max_interrupt_rate="216000" hw.igb.rxd=2048 hw.igb.txd=2048 hw.ix.rxd=2048 hw.ix.txd=2048 net.isr.maxthreads="-1" net.isr.bindthreads="1" net.pf.states_hashsize="2097152" net.pf.source_nodes_hashsize="1421000" -
have a 3200g system up and running with intel lan cards. so far so good,
-
You should be fine with a standard amount of connections with default settings, like below 15,000. You're currently using 1,895. I average 100k-250k UDP connections every second, on top of my normal traffic around like 1-5k, which is why I needed to fine tune so much.
Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.