Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Super Micro C2758 crashes

    Scheduled Pinned Locked Moved 2.4 Development Snapshots
    17 Posts 4 Posters 5.0k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • A
      athurdent
      last edited by

      OK, no luck with 2 queues.

      I can reboot the board with

      hping3 -c 100 -d 120 -S -w 64 -p 443 --flood 192.168.x.10
      

      from LAN to my DMZ (produces about 130000 states).
      I just send in the Crash Report for that.

      With only one queue I can even use "–rand-source" until the state table is full, no reboot.

      1 Reply Last reply Reply Quote 0
      • w0wW
        w0w
        last edited by

        I'll do the test on similar hardware later this week.

        1 Reply Last reply Reply Quote 0
        • A
          athurdent
          last edited by

          Thanks w0w, looking forward to your results.

          BTW the hping3 crash looked different:

          panic: bpf_mcopy
          cpuid = 6
          KDB: enter: panic

          Edit: Just found this on Reddit, so I guess only one queue per interface should be OK for everyone just passing traffic trough?

          https://www.reddit.com/r/PFSENSE/comments/5obhlm/what_are_the_ramifications_of_less_nic_queues/dci77px/

          1 Reply Last reply Reply Quote 0
          • w0wW
            w0w
            last edited by

            Tested, but I could not replicate the issue.
            How much physical RAM do you have installed?
            Do you have any custom tunables enabled, other than you have provided in this topic?
            Below you will see network stack tunes I have enabled:
            #some magic numbers :)
            kern.ipc.nmbjumbo9="20000"
            kern.ipc.nmbclusters="1000000"
            kern.ipc.maxsockbuf="256000000"
            #some more igb tune for GIG links
            hw.igb.rxd="4096"
            hw.igb.txd="4096"
            net.inet.tcp.syncache.hashsize=1024
            net.inet.tcp.syncache.bucketlimit=100
            net.isr.defaultqlimit=4096
            net.link.ifqmaxlen=10240
            hw.igb.rx_process_limit="-1"
            hw.igb.num_queues=2
            #disable flow control on all igb interfaces
            dev.igb.0.fc=0
            dev.igb.1.fc=0
            dev.igb.2.fc=0
            dev.igb.3.fc=0
            I'll do some test with all tunables disabled and will try to utilize all bandwidth I have on WAN…

            1 Reply Last reply Reply Quote 0
            • A
              athurdent
              last edited by

              Thanks for looking into this. I tend to stick with the defaults until I hit a problem. So I only have nmbclusters defined. I have also disabled Hardware Checksum Offloading, can't remember why, though.
              Amongst others I'm using CARP, VLANs, the on-board SNMP and Snort. Those could also be resonsible for problems I guess.

              1 Reply Last reply Reply Quote 0
              • C
                chrcoluk
                last edited by

                did you try with 1 igb queue? if not please test with it.

                –edit--

                I see you did, thanks for reporting the findings, it does seem igb currently on FreeBSD 11 has issues with multi queue.

                pfSense CE 2.8.0

                1 Reply Last reply Reply Quote 0
                • A
                  athurdent
                  last edited by

                  @chrcoluk:

                  I see you did, thanks for reporting the findings, it does seem igb currently on FreeBSD 11 has issues with multi queue.

                  Thanks, do you have any FreeBSD Bugtracker or forum reference for this?

                  1 Reply Last reply Reply Quote 0
                  • C
                    chrcoluk
                    last edited by

                    sadly I did not bookmark it but will have a look later and if I find it will post here.

                    pfSense CE 2.8.0

                    1 Reply Last reply Reply Quote 0
                    • w0wW
                      w0w
                      last edited by

                      Tested with states over 196015, but nothing.  But I have found that what I am doing is a little bit wrong. The problem is that my testing machine have PPPoE link enabled on WAN testing interface and that means that WAN uses only one queue by freebsd design.
                      Sorry, I can not test it in other way, but it looks like igb(4) drivers are not so good, but at least are not so bad as stock realtek freebsd drivers;)
                      I did not researched yet, but there are some custom patches can be found over freebsd community.
                      But if your SuperMicro C2758 doing its job and you have not performance issues than just leave it  alone :)

                      1 Reply Last reply Reply Quote 0
                      • C
                        chrcoluk
                        last edited by

                        The problem seems to be triggered by one of these two things or perhaps even both.

                        FreeBSD 11 changes ref igb driver.

                        RSS awareness has been added to the igb(4) driver (r268028)
                        Automatic disabling of multi queue if ALTQ is enabled in kernel.

                        The latter I think is the most likely as it seems its not fully disabling multi queue, as evident by the fact the loader tunable is defaulting to match number of cpu cores, and then the problem goes away when its forced to a 1 value.

                        https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212413

                        That problem report also links to another couple as well.

                        pfSense CE 2.8.0

                        1 Reply Last reply Reply Quote 0
                        • A
                          athurdent
                          last edited by

                          @chrcoluk: Thank you!

                          1 Reply Last reply Reply Quote 0
                          • A
                            athurdent
                            last edited by

                            Might be fixed now: https://redmine.pfsense.org/issues/7149

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.