Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Super Micro C2758 crashes

    Scheduled Pinned Locked Moved
    2.4 Development Snapshots
    4
    17
    4.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • w0wW
      w0w
      last edited by

      I'll do the test on similar hardware later this week.

      1 Reply Last reply Reply Quote 0
      • A
        athurdent
        last edited by

        Thanks w0w, looking forward to your results.

        BTW the hping3 crash looked different:

        panic: bpf_mcopy
        cpuid = 6
        KDB: enter: panic

        Edit: Just found this on Reddit, so I guess only one queue per interface should be OK for everyone just passing traffic trough?

        https://www.reddit.com/r/PFSENSE/comments/5obhlm/what_are_the_ramifications_of_less_nic_queues/dci77px/

        1 Reply Last reply Reply Quote 0
        • w0wW
          w0w
          last edited by

          Tested, but I could not replicate the issue.
          How much physical RAM do you have installed?
          Do you have any custom tunables enabled, other than you have provided in this topic?
          Below you will see network stack tunes I have enabled:
          #some magic numbers :)
          kern.ipc.nmbjumbo9="20000"
          kern.ipc.nmbclusters="1000000"
          kern.ipc.maxsockbuf="256000000"
          #some more igb tune for GIG links
          hw.igb.rxd="4096"
          hw.igb.txd="4096"
          net.inet.tcp.syncache.hashsize=1024
          net.inet.tcp.syncache.bucketlimit=100
          net.isr.defaultqlimit=4096
          net.link.ifqmaxlen=10240
          hw.igb.rx_process_limit="-1"
          hw.igb.num_queues=2
          #disable flow control on all igb interfaces
          dev.igb.0.fc=0
          dev.igb.1.fc=0
          dev.igb.2.fc=0
          dev.igb.3.fc=0
          I'll do some test with all tunables disabled and will try to utilize all bandwidth I have on WAN…

          1 Reply Last reply Reply Quote 0
          • A
            athurdent
            last edited by

            Thanks for looking into this. I tend to stick with the defaults until I hit a problem. So I only have nmbclusters defined. I have also disabled Hardware Checksum Offloading, can't remember why, though.
            Amongst others I'm using CARP, VLANs, the on-board SNMP and Snort. Those could also be resonsible for problems I guess.

            1 Reply Last reply Reply Quote 0
            • C
              chrcoluk
              last edited by

              did you try with 1 igb queue? if not please test with it.

              –edit--

              I see you did, thanks for reporting the findings, it does seem igb currently on FreeBSD 11 has issues with multi queue.

              pfSense CE 2.7.2

              1 Reply Last reply Reply Quote 0
              • A
                athurdent
                last edited by

                @chrcoluk:

                I see you did, thanks for reporting the findings, it does seem igb currently on FreeBSD 11 has issues with multi queue.

                Thanks, do you have any FreeBSD Bugtracker or forum reference for this?

                1 Reply Last reply Reply Quote 0
                • C
                  chrcoluk
                  last edited by

                  sadly I did not bookmark it but will have a look later and if I find it will post here.

                  pfSense CE 2.7.2

                  1 Reply Last reply Reply Quote 0
                  • w0wW
                    w0w
                    last edited by

                    Tested with states over 196015, but nothing.  But I have found that what I am doing is a little bit wrong. The problem is that my testing machine have PPPoE link enabled on WAN testing interface and that means that WAN uses only one queue by freebsd design.
                    Sorry, I can not test it in other way, but it looks like igb(4) drivers are not so good, but at least are not so bad as stock realtek freebsd drivers;)
                    I did not researched yet, but there are some custom patches can be found over freebsd community.
                    But if your SuperMicro C2758 doing its job and you have not performance issues than just leave it  alone :)

                    1 Reply Last reply Reply Quote 0
                    • C
                      chrcoluk
                      last edited by

                      The problem seems to be triggered by one of these two things or perhaps even both.

                      FreeBSD 11 changes ref igb driver.

                      RSS awareness has been added to the igb(4) driver (r268028)
                      Automatic disabling of multi queue if ALTQ is enabled in kernel.

                      The latter I think is the most likely as it seems its not fully disabling multi queue, as evident by the fact the loader tunable is defaulting to match number of cpu cores, and then the problem goes away when its forced to a 1 value.

                      https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=212413

                      That problem report also links to another couple as well.

                      pfSense CE 2.7.2

                      1 Reply Last reply Reply Quote 0
                      • A
                        athurdent
                        last edited by

                        @chrcoluk: Thank you!

                        1 Reply Last reply Reply Quote 0
                        • A
                          athurdent
                          last edited by

                          Might be fixed now: https://redmine.pfsense.org/issues/7149

                          1 Reply Last reply Reply Quote 0
                          • First post
                            Last post