Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Ryzen 3 2200G randomly crashing

    Scheduled Pinned Locked Moved Hardware
    28 Posts 7 Posters 3.7k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Z
      Zermus
      last edited by Zermus

      pfSense 2.4.4-p3 on B450 chipset on latest BIOS. Anyone had a similar issue? Just randomly crashes and reboots. It's not under a terrible amount of load or anything, but I'm lucky to have a 3+ day uptime...

      I've got a pfSense crash report if necessary that I can provide.

      1 Reply Last reply Reply Quote 0
      • jimpJ
        jimp Rebel Alliance Developer Netgate
        last edited by

        Post the crash report.

        Remember: Upvote with the 👍 button for any user/post you find to be helpful, informative, or deserving of recognition!

        Need help fast? Netgate Global Support!

        Do not Chat/PM for help!

        1 Reply Last reply Reply Quote 0
        • Z
          Zermus
          last edited by

          Here you go.
          pfsense-crash.txt

          1 Reply Last reply Reply Quote 0
          • kiokomanK
            kiokoman LAYER 8
            last edited by

            do you have snort + barnyar2 as inline mode ? it seems to me a network card issue, Realtek RTL8111H could be the culprit

            ̿' ̿'\̵͇̿̿\з=(◕_◕)=ε/̵͇̿̿/'̿'̿ ̿
            Please do not use chat/PM to ask for help
            we must focus on silencing this @guest character. we must make up lies and alter the copyrights !
            Don't forget to Upvote with the 👍 button for any post you find to be helpful.

            1 Reply Last reply Reply Quote 0
            • DerelictD
              Derelict LAYER 8 Netgate
              last edited by

              Moved all posts here. Please do not cross post the same topic to multiple categories.

              Chattanooga, Tennessee, USA
              A comprehensive network diagram is worth 10,000 words and 15 conference calls.
              DO NOT set a source address/port in a port forward or firewall rule unless you KNOW you need it!
              Do Not Chat For Help! NO_WAN_EGRESS(TM)

              1 Reply Last reply Reply Quote 0
              • Z
                Zermus
                last edited by Zermus

                My bad Derelict, I meant to delete that other one in General but got distracted.

                I used to have barnyard configured on old hardware but I have it disabled now. I transferred the config over.

                The Realtek card is not in use for network traffic. I could try to disable that in the bios. I've had issues with Realtek cards in pfsense before and that usually just crashes the NIC driver, not the whole server. I think this is more of a FreeBSD kernel/cpu/chipset thing based on the crash report. Just my 2 cents though.

                There is this bug, but it was supposed to be fixed in FreeBSD 10.1 - https://forums.freebsd.org/threads/fatal-trap-12-page-fault-while-in-kernel-mode-on-new-server-running-freebsd-10-1-release-p10.51737/

                1 Reply Last reply Reply Quote 0
                • stephenw10S
                  stephenw10 Netgate Administrator
                  last edited by stephenw10

                  Nope that bug is completely different. The key part is the backtrace here:

                  db:0:kdb.enter.default>  show pcpu
                  cpuid        = 0
                  dynamic pcpu = 0xc22380
                  curthread    = 0xfffff8000632d620: pid 12 "irq271: em0:rx0"
                  curpcb       = 0xfffffe03d58a1b80
                  fpcurthread  = none
                  idlethread   = 0xfffff80005f41000: tid 100003 "idle: cpu0"
                  curpmap      = 0xffffffff82b85998
                  tssp         = 0xffffffff82bb6810
                  commontssp   = 0xffffffff82bb6810
                  rsp0         = 0xfffffe03d58a1b80
                  gs32p        = 0xffffffff82bbd068
                  ldt          = 0xffffffff82bbd0a8
                  tss          = 0xffffffff82bbd098
                  db:0:kdb.enter.default>  bt
                  Tracing pid 12 tid 100076 td 0xfffff8000632d620
                  bcmp() at bcmp+0xb/frame 0xfffffe03d58a1200
                  pf_find_state() at pf_find_state+0xad/frame 0xfffffe03d58a1240
                  pf_test_state_udp() at pf_test_state_udp+0x11b/frame 0xfffffe03d58a12c0
                  pf_test() at pf_test+0x1b64/frame 0xfffffe03d58a1500
                  pf_check_out() at pf_check_out+0x1d/frame 0xfffffe03d58a1520
                  pfil_run_hooks() at pfil_run_hooks+0x90/frame 0xfffffe03d58a15b0
                  ip_output() at ip_output+0xb1d/frame 0xfffffe03d58a16e0
                  ip_forward() at ip_forward+0x2b5/frame 0xfffffe03d58a1780
                  ip_input() at ip_input+0x72a/frame 0xfffffe03d58a17e0
                  netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe03d58a1830
                  ether_demux() at ether_demux+0x173/frame 0xfffffe03d58a1860
                  ether_nh_input() at ether_nh_input+0x32b/frame 0xfffffe03d58a18c0
                  netisr_dispatch_src() at netisr_dispatch_src+0xa8/frame 0xfffffe03d58a1910
                  ether_input() at ether_input+0x26/frame 0xfffffe03d58a1930
                  if_input() at if_input+0xa/frame 0xfffffe03d58a1940
                  em_rxeof() at em_rxeof+0x2c0/frame 0xfffffe03d58a19b0
                  em_msix_rx() at em_msix_rx+0x34/frame 0xfffffe03d58a19e0
                  intr_event_execute_handlers() at intr_event_execute_handlers+0xe9/frame 0xfffffe03d58a1a20
                  ithread_loop() at ithread_loop+0xe7/frame 0xfffffe03d58a1a70
                  fork_exit() at fork_exit+0x83/frame 0xfffffe03d58a1ab0
                  fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe03d58a1ab0
                  --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
                  db:0:kdb.enter.default>  ps
                  

                  But this is not good:
                  [zone: pf frag entries] PF frag entries limit reached
                  You could increase the frags limit in System > Advanced > Firewall & NAT
                  But really you are hitting that because of an unusually high number of fragmented packets you should try to find why that's happening and fix it.

                  Steve

                  1 Reply Last reply Reply Quote 1
                  • Z
                    Zermus
                    last edited by Zermus

                    Ahh ok, I remember jacking up the fragments on my old hardware and that solved it then. I bet that's what it is. I guess the advanced stuff like that doesn't get transferred over in the config dumps when you move it to new hardware.

                    I know what the fragments are. There is an NTP pool node behind this pfSense and it sees some heavy UDP connections, among other things...

                    Thanks for tracking that down!

                    1 Reply Last reply Reply Quote 0
                    • Z
                      Zermus
                      last edited by

                      Ok I got another crash today, and today it's not showing the pf frag entries limit reached, but same kernel crash message.

                      MBUs set to a million
                      Frag packets set to 25,000.

                      pfsense-crash.txt

                      Any other ideas?

                      Z 1 Reply Last reply Reply Quote 0
                      • Z
                        Zermus @Zermus
                        last edited by

                        For kicks I've disabled MSI-X and went to MSI interrupts. We'll see if that has any effect.

                        /boot/loader.conf.local

                        hw.pci.enable_msix=0
                        
                        1 Reply Last reply Reply Quote 0
                        • stephenw10S
                          stephenw10 Netgate Administrator
                          last edited by

                          Hmm, that's different crash. Still in the network stack though:

                          db:0:kdb.enter.default>  show pcpu
                          cpuid        = 1
                          dynamic pcpu = 0xfffffe03df1c3380
                          curthread    = 0xfffff8001d50e620: pid 20 "pf purge"
                          curpcb       = 0xfffffe03d59d2b80
                          fpcurthread  = none
                          idlethread   = 0xfffff80005f40620: tid 100004 "idle: cpu1"
                          curpmap      = 0xffffffff82b85998
                          tssp         = 0xffffffff82bb6878
                          commontssp   = 0xffffffff82bb6878
                          rsp0         = 0xfffffe03d59d2b80
                          gs32p        = 0xffffffff82bbd0d0
                          ldt          = 0xffffffff82bbd110
                          tss          = 0xffffffff82bbd100
                          db:0:kdb.enter.default>  bt
                          Tracing pid 20 tid 100130 td 0xfffff8001d50e620
                          __mtx_lock_sleep() at __mtx_lock_sleep+0xcd/frame 0xfffffe03d59d2990
                          pf_detach_state() at pf_detach_state+0x311/frame 0xfffffe03d59d29c0
                          pf_unlink_state() at pf_unlink_state+0x1ee/frame 0xfffffe03d59d29f0
                          pf_purge_expired_states() at pf_purge_expired_states+0x6d/frame 0xfffffe03d59d2a40
                          pf_purge_thread() at pf_purge_thread+0x71/frame 0xfffffe03d59d2a70
                          fork_exit() at fork_exit+0x83/frame 0xfffffe03d59d2ab0
                          fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe03d59d2ab0
                          --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
                          db:0:kdb.enter.default>  ps
                          

                          Nothing in the message buffer to help.....

                          1 Reply Last reply Reply Quote 1
                          • Z
                            Zermus
                            last edited by

                            Yeah, I've been reading about how Intel cards suck with MSI-X, so hopefully that's what it is. I guess time will tell.

                            1 Reply Last reply Reply Quote 0
                            • stephenw10S
                              stephenw10 Netgate Administrator
                              last edited by

                              That has not been my experience. Especially with em(4) which is very mature at this point.

                              Steve

                              Z 1 Reply Last reply Reply Quote 0
                              • Z
                                Zermus @stephenw10
                                last edited by Zermus

                                @stephenw10

                                https://www.intel.com/content/www/us/en/support/articles/000017506/network-and-i-o.html

                                Maybe it's the amount of packets I'm pushing through my Gigabit Intel WAN? I'm pushing it pretty hard. 12 hours on MSI so far and no crashes and no degraded throughput or anything. Probably a slight increase on CPU, barely 3% if that.

                                1 Reply Last reply Reply Quote 0
                                • Z
                                  Zermus
                                  last edited by

                                  Well, 1 Day 11 Hours 08 Minutes 36 Seconds uptime since going to MSI. Knocks on wood

                                  T 1 Reply Last reply Reply Quote 1
                                  • T
                                    tman222 @Zermus
                                    last edited by

                                    @Zermus

                                    This surprises me as well after reading through this thread - Intel cards are generally top tier and perform very well with pfSense. What specific Intel card / chipset are you using?

                                    Z 1 Reply Last reply Reply Quote 0
                                    • Z
                                      Zermus @tman222
                                      last edited by

                                      @tman222

                                      WAN - Intel Gigabit CT PCI-E Network Adapter EXPI9301CTBLK
                                      LAN - Intel X520-DA2 (The real Intel, not the 10GTek knock off, Multiple VLANs using only one port)

                                      T 1 Reply Last reply Reply Quote 0
                                      • T
                                        tman222 @Zermus
                                        last edited by

                                        @Zermus said in Ryzen 3 2200G randomly crashing:

                                        @tman222

                                        WAN - Intel Gigabit CT PCI-E Network Adapter EXPI9301CTBLK
                                        LAN - Intel X520-DA2 (The real Intel, not the 10GTek knock off, Multiple VLANs using only one port)

                                        Which one of those two was / is giving you trouble? Looking at data sheets and product briefs, both of those cards support MSI-X. Also, just to doublecheck, are you running pfSense on bare metal or inside a VM? Have you tried adjusting any of other tunables on the cards or is everything basically running with stock settings?

                                        Come to think of it now, I did have one EXPI9301CTBLK adapter seemingly go bad on me a couple years back. Was running in a Windows desktop machine though. For some reason one day it stopped passing gigabit speeds and nothing I did would fix it (reinstall, change cables, etc.). Replaced with another card and all was well.

                                        I think the chipset in the Intel CT adapter is targeted more towards desktop/laptop applications. If you're interested in replacing it with something a bit higher end, these HP Intel i210 based cards should work well, are fairly inexpensive, and use the igb driver in FreeBSD:

                                        https://www.amazon.com/Intel-I210-T1-Network-Adapter-E0X95AA/dp/B00ESFF2JC/
                                        https://www8.hp.com/us/en/products/oas/product-detail.html?oid=5383389

                                        Hope this helps.

                                        1 Reply Last reply Reply Quote 0
                                        • Z
                                          Zermus
                                          last edited by Zermus

                                          Hmm, I'll try to swap out the Intel CT with the I210. The weird part is that the system was stable on older hardware with the same NICs, the system was based on an AMD A8-7600 Kavari (Steamroller). Unfortunately the motherboard started flaking out on that system though, which lead to the Ryzen rebuild. Right now with MSI interrupts the system seems stable for about 2 to 3 days but now experiences hard crashes where I get no crash reports.

                                          Extended 12 hour memtest run came up with no errors, so I'm fairly certain hardware is good in this case. I'm starting to wonder if it's just Ryzen and FreeBSD bugs creeping up that I'm hitting.

                                          T 1 Reply Last reply Reply Quote 0
                                          • T
                                            tman222 @Zermus
                                            last edited by

                                            @Zermus said in Ryzen 3 2200G randomly crashing:

                                            Hmm, I'll try to swap out the Intel CT with the I210. The weird part is that the system was stable on older hardware with the same NICs, the system was based on an AMD A8-7600 Kavari (Steamroller). Unfortunately the motherboard started flaking out on that system though, which lead to the Ryzen rebuild. Right now with MSI interrupts the system seems stable for about 2 to 3 days but now experiences hard crashes where I get no crash reports.

                                            Extended 12 hour memtest run came up with no errors, so I'm fairly certain hardware is good in this case. I'm starting to wonder if it's just Ryzen and FreeBSD bugs creeping up that I'm hitting.

                                            That's a good point - I wonder if support / stability will improve with FreeBSD 12.0 (which pfSense 2.5 will use).

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.