Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    6100 SLOW in comparison to Protectli FW6E

    Scheduled Pinned Locked Moved Official Netgate® Hardware
    119 Posts 5 Posters 66.6k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bmeeksB
      bmeeks @manilx
      last edited by

      @manilx said in 6100 SLOW in comparison to Protectli FW6E:

      @bmeeks If I remember well I don't have it installed, otherwise I'll uninstall first and then install the test package manually.

      One more point, too. If you are running either pfSense or OPNsense as a virtual machine, the choice of virtual NIC will matter in how well the bug fix works (unless you are doing passthrough of a real hardware NIC). As I mentioned, when you use a NIC that does not support native netmap operation, the netmap kernel code creates something called a generic netmap adapter. That is a kludge and is much slower performance-wise than native-mode netmap. There are also currently some known issues with the generic netmap adapter when stressed with high throughput. A team of developers is actively working on those issues and submitting their proposed fixes to FreeBSD for inclusion.

      So, saying all the above to note that you may install the package and still experience a stall, but I'm hoping things run longer at least with the patched Suricata. And it is possible things may run flawlessly as well (that's my hope).

      M 1 Reply Last reply Reply Quote 0
      • M
        manilx @bmeeks
        last edited by

        @bmeeks ScreenShot 2023-02-16 at 20.46.25.png

        2 Virtio network cards. No passthrough.

        I'll run a few speedtests but will not keep it running for a longer time... Hope this helps.

        Netgate 8200max

        M bmeeksB 2 Replies Last reply Reply Quote 0
        • M
          manilx @manilx
          last edited by

          @manilx P.S. I'll do it later tonight in a couple of hours.

          Netgate 8200max

          1 Reply Last reply Reply Quote 0
          • bmeeksB
            bmeeks @manilx
            last edited by bmeeks

            @manilx said in 6100 SLOW in comparison to Protectli FW6E:

            @bmeeks ScreenShot 2023-02-16 at 20.46.25.png

            2 Virtio network cards. No passthrough.

            I'll run a few speedtests but will not keep it running for a longer time... Hope this helps.

            The main thing I want to see is if there is a change in the positive direction. Meaning either it works without stalling, or it works noticeably longer before stalling.

            Did this setup work okay with Suricata prior to November of last year? If so, I'm hoping it returns the same level of reliability you had prior to the end of last year when the new version of Suricata 6.0.9 was rolled out with OPNsense.

            M 1 Reply Last reply Reply Quote 0
            • M
              manilx @bmeeks
              last edited by

              @bmeeks Well, I have posted my issue with OPNsense in a link above.
              I lost internet connection out of the blue. But not consistently and I couldn't force it!
              I don't know if it was suricata back then. What I know it was when I upgraded OPNsense to a newer release (all in the post above).

              SO I don't think this test can help. I won't have the VM running for longer than a few speedtests.

              If this is OK I'll do it, if not then I'll spare me the hassle.

              I moved for good to pfsense (because of the above and the "attitude" I got from the developer!!).
              I now have pfsense hardware and all.

              Netgate 8200max

              bmeeksB 1 Reply Last reply Reply Quote 0
              • bmeeksB
                bmeeks @manilx
                last edited by

                @manilx said in 6100 SLOW in comparison to Protectli FW6E:

                @bmeeks Well, I have posted my issue with OPNsense in a link above.
                I lost internet connection out of the blue. But not consistently and I couldn't force it!
                I don't know if it was suricata back then. What I know it was when I upgraded OPNsense to a newer release (all in the post above).

                SO I don't think this test can help. I won't have the VM running for longer than a few speedtests.

                If this is OK I'll do it, if not then I'll spare me the hassle.

                I moved for good to pfsense (because of the above and the "attitude" I got from the developer!!).
                I now have pfsense hardware and all.

                I think I recall checking out your post in the link, but the details I have forgotten. Been juggling a lot of balls in the air with various Suricata changes recently, and I'm not a spring chicken anymore and things can escape my brain into the ether 😊.

                Is your VM setup of pfSense using CE or Plus? If using either 23.01 Plus, or 2.7 CE DEVEL, I can send you a "fixed" Suricata binary for pfSense you can test with. I can also make you one for 2.6 CE, but it will take a bit to build.

                M 1 Reply Last reply Reply Quote 0
                • M
                  manilx @bmeeks
                  last edited by

                  @bmeeks VM is only the "old" OPNsense.
                  I'm running pfsense+ 23.01 on a brand new 8200max now.

                  Netgate 8200max

                  bmeeksB 1 Reply Last reply Reply Quote 0
                  • bmeeksB
                    bmeeks @manilx
                    last edited by bmeeks

                    @manilx said in 6100 SLOW in comparison to Protectli FW6E:

                    @bmeeks VM is only the "old" OPNsense.
                    I'm running pfsense+ 23.01 on a brand new 8200max now.

                    That's fine. If you can fire up the old VM and test with the Suricata package Franco linked in the Suricata Redmine Issue thread that would be helpful. Just bring it up and run a few speed tests. Based on some of the earlier reports from other users, I would expect a speed test to trigger the stall fairly quickly: or at least several back-to-back speed tests on Gigabit or higher links should trigger it. If the speed tests work well, then that increases my confidence in the patch.

                    My problem with testing locally is that currently I don't have a suitable setup in place. I hope to free up some hardware next week that will help. The bug needs to be exercised with really heavy traffic loads in order to show up. I've never been able to get it to show with my feeble VMware Workstation testing setup (not surprised at all there, though).

                    M 1 Reply Last reply Reply Quote 0
                    • M
                      manilx @bmeeks
                      last edited by

                      @bmeeks OK!

                      Updated OPNsense to the latest version.
                      Installed the suricata test package.
                      ScreenShot 2023-02-16 at 22.44.08.png

                      Run half a dozen speedtests saturating the line at 930+

                      ScreenShot 2023-02-16 at 22.48.50.png

                      No problem so far.

                      Reverted to 8200.

                      Hope this helps

                      Netgate 8200max

                      bmeeksB 1 Reply Last reply Reply Quote 0
                      • bmeeksB
                        bmeeks @manilx
                        last edited by bmeeks

                        @manilx said in 6100 SLOW in comparison to Protectli FW6E:

                        @bmeeks OK!

                        Updated OPNsense to the latest version.
                        Installed the suricata test package.
                        ScreenShot 2023-02-16 at 22.44.08.png

                        Run half a dozen speedtests saturating the line at 930+

                        ScreenShot 2023-02-16 at 22.48.50.png

                        No problem so far.

                        Reverted to 8200.

                        Hope this helps

                        Yes! This is very helpful. It proves the patch works. Sorry to be a little late replying. Had to leave home to attend a meeting and just returned.

                        Thank you very much for testing. I will pass the word on to the Suricata team.

                        M 1 Reply Last reply Reply Quote 0
                        • M
                          manilx @bmeeks
                          last edited by

                          @bmeeks You’re welcomed. Hope to see the results in pfsense soon 😉

                          Netgate 8200max

                          bmeeksB 2 Replies Last reply Reply Quote 0
                          • bmeeksB
                            bmeeks @manilx
                            last edited by bmeeks

                            @manilx said in 6100 SLOW in comparison to Protectli FW6E:

                            @bmeeks You’re welcomed. Hope to see the results in pfsense soon 😉

                            The pull request is posted now and awaiting review and approval by the Netgate team. They have some more important fires to beat out relative to the just released 23.01 update, but I expect them to get the new package version merged soon. It will show as Suricata-6.0.10 when released. The pull request for updating the binary portion is here: https://github.com/pfsense/FreeBSD-ports/pull/1226.

                            1 Reply Last reply Reply Quote 2
                            • bmeeksB
                              bmeeks @manilx
                              last edited by

                              @manilx said in 6100 SLOW in comparison to Protectli FW6E:

                              @bmeeks You’re welcomed. Hope to see the results in pfsense soon 😉

                              The pull requests for updating Suricata have been accepted by the Netgate team. For the moment, automatic package builds are disabled in the 23.01 branch so the new version will not show up yet. The automatic builds should be re-enabled soon, and after that you will see a Suricata-6.0.10 package available containing the latest fixes.

                              Once you have that installed, go to the INTERFACE SETTINGS tab and change the Run Mode under the Performance and Detection Engine Settings section to "workers", save the change, and then restart Suricata on the interface. You should see a marked increase in throughput using Inline IPS Mode after that. And with the new fix, the stall/hang condition is hopefully gone!

                              M 1 Reply Last reply Reply Quote 2
                              • M
                                manilx @bmeeks
                                last edited by

                                @bmeeks Awesome. Once it's there I'll try it immediately and post back here.

                                Netgate 8200max

                                bmeeksB 1 Reply Last reply Reply Quote 0
                                • bmeeksB
                                  bmeeks @manilx
                                  last edited by

                                  @manilx:
                                  The new updated Suricata package containing the netmap stall fix has been posted now for pfSense 23.01. The new Suricata package version is 6.0.10_1.

                                  I am interested in how this new package performs for you. For peak throughput when using Inline IPS Mode, remember to change the Run Mode parameter from its default of autofp to workers. Save that change and then restart Suricata on the interface.

                                  M 1 Reply Last reply Reply Quote 0
                                  • M
                                    manilx @bmeeks
                                    last edited by

                                    @bmeeks Hi. Updated suricata. Changed run mode to workers.
                                    Restarted on interface. Doen't start. Tried several times to restart, doesn't.
                                    Rebooted. Suricata not started.
                                    reverted to snaphot (updated suricata but not workers mode). Working.

                                    Netgate 8200max

                                    bmeeksB 1 Reply Last reply Reply Quote 0
                                    • bmeeksB
                                      bmeeks @manilx
                                      last edited by

                                      @manilx said in 6100 SLOW in comparison to Protectli FW6E:

                                      @bmeeks Hi. Updated suricata. Changed run mode to workers.
                                      Restarted on interface. Doen't start. Tried several times to restart, doesn't.
                                      Rebooted. Suricata not started.
                                      reverted to snaphot (updated suricata but not workers mode). Working.

                                      Dang! Post the output of the suricata.log file for a failed startup in workers run mode.

                                      M 1 Reply Last reply Reply Quote 0
                                      • M
                                        manilx @bmeeks
                                        last edited by

                                        @bmeeks Will have to boot back to other snapshot and try.

                                        Netgate 8200max

                                        M 1 Reply Last reply Reply Quote 0
                                        • M
                                          manilx @manilx
                                          last edited by stephenw10

                                          @manilx

                                          21/2/2023 -- 20:14:21 - <Notice> -- This is Suricata version 6.0.10 RELEASE running in SYSTEM mode
21/2/2023 -- 20:14:21 - <Info> -- CPUs/cores online: 8
21/2/2023 -- 20:14:22 - <Info> -- HTTP memcap: 67108864
21/2/2023 -- 20:14:22 - <Info> -- Netmap: Setting IPS mode
21/2/2023 -- 20:14:22 - <Info> -- fast output device (regular) initialized: alerts.log
21/2/2023 -- 20:14:22 - <Info> -- http-log output device (regular) initialized: http.log
21/2/2023 -- 20:14:24 - <Info> -- 1 rule files processed. 6453 rules successfully loaded, 0 rules failed
21/2/2023 -- 20:14:24 - <Warning> -- [ERRCODE: SC_ERR_EVENT_ENGINE(210)] - can't suppress sid 2260002, gid 1: unknown rule
21/2/2023 -- 20:14:24 - <Info> -- Threshold config parsed: 2 rule(s) found
21/2/2023 -- 20:14:24 - <Info> -- 6453 signatures processed. 910 are IP-only rules, 0 are inspecting packet payload, 5376 inspect application layer, 108 are decoder event only
21/2/2023 -- 20:14:25 - <Info> -- Going to use 8 thread(s)
21/2/2023 -- 20:14:26 - <Info> -- devname [fd: 6] netmap:ix3-0/R@conf:host-rings=8 ix3 opened
21/2/2023 -- 20:14:26 - <Info> -- devname [fd: 7] netmap:ix3^0/T@conf:host-rings=8 ix3^ opened
21/2/2023 -- 20:14:26 - <Info> -- devname [fd: 11] netmap:ix3-1/R ix3 opened
21/2/2023 -- 20:14:26 - <Info> -- devname [fd: 12] netmap:ix3^1/T ix3^ opened
21/2/2023 -- 20:14:26 - <Info> -- devname [fd: 14] netmap:ix3-2/R ix3 opened
21/2/2023 -- 20:14:27 - <Info> -- devname [fd: 15] netmap:ix3^2/T ix3^ opened
21/2/2023 -- 20:14:27 - <Info> -- devname [fd: 16] netmap:ix3-3/R ix3 opened
21/2/2023 -- 20:14:27 - <Info> -- devname [fd: 17] netmap:ix3^3/T ix3^ opened
21/2/2023 -- 20:14:27 - <Info> -- devname [fd: 18] netmap:ix3-4/R ix3 opened
21/2/2023 -- 20:14:28 - <Info> -- devname [fd: 19] netmap:ix3^4/T ix3^ opened
21/2/2023 -- 20:14:28 - <Info> -- devname [fd: 20] netmap:ix3-5/R ix3 opened
21/2/2023 -- 20:14:28 - <Info> -- devname [fd: 21] netmap:ix3^5/T ix3^ opened
21/2/2023 -- 20:14:28 - <Info> -- devname [fd: 22] netmap:ix3-6/R ix3 opened
21/2/2023 -- 20:14:28 - <Info> -- devname [fd: 23] netmap:ix3^6/T ix3^ opened
21/2/2023 -- 20:14:29 - <Info> -- devname [fd: 24] netmap:ix3-7/R ix3 opened
21/2/2023 -- 20:14:29 - <Info> -- devname [fd: 25] netmap:ix3^7/T ix3^ opened
21/2/2023 -- 20:14:29 - <Info> -- Going to use 8 thread(s)
21/2/2023 -- 20:14:29 - <Info> -- devname [fd: 26] netmap:ix3^0/R@conf:host-rings=8 ix3^ opened
21/2/2023 -- 20:14:29 - <Info> -- devname [fd: 27] netmap:ix3-0/T@conf:host-rings=8 ix3 opened
21/2/2023 -- 20:14:30 - <Info> -- devname [fd: 28] netmap:ix3^1/R ix3^ opened
21/2/2023 -- 20:14:30 - <Info> -- devname [fd: 29] netmap:ix3-1/T ix3 opened
21/2/2023 -- 20:14:30 - <Info> -- devname [fd: 30] netmap:ix3^2/R ix3^ opened
21/2/2023 -- 20:14:31 - <Info> -- devname [fd: 31] netmap:ix3-2/T ix3 opened
21/2/2023 -- 20:14:31 - <Info> -- devname [fd: 32] netmap:ix3^3/R ix3^ opened
21/2/2023 -- 20:14:31 - <Info> -- devname [fd: 33] netmap:ix3-3/T ix3 opened
21/2/2023 -- 20:14:31 - <Info> -- devname [fd: 34] netmap:ix3^4/R ix3^ opened
21/2/2023 -- 20:14:31 - <Info> -- devname [fd: 35] netmap:ix3-4/T ix3 opened
21/2/2023 -- 20:14:32 - <Info> -- devname [fd: 36] netmap:ix3^5/R ix3^ opened
21/2/2023 -- 20:14:32 - <Info> -- devname [fd: 37] netmap:ix3-5/T ix3 opened
21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?
21/2/2023 -- 20:14:32 - <Info> -- devname [fd: 38] netmap:ix3^6/R ix3^ opened
21/2/2023 -- 20:14:32 - <Info> -- devname [fd: 39] netmap:ix3-6/T ix3 opened
21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?
21/2/2023 -- 20:14:33 - <Info> -- devname [fd: 40] netmap:ix3^7/R ix3^ opened
21/2/2023 -- 20:14:33 - <Info> -- devname [fd: 41] netmap:ix3-7/T ix3 opened
21/2/2023 -- 20:14:33 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - alloc error
21/2/2023 -- 20:14:33 - <Error> -- [ERRCODE: SC_ERR_POOL_INIT(66)] - pool grow failed
21/2/2023 -- 20:14:33 - <Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?
21/2/2023 -- 20:14:33 - <Error> -- [ERRCODE: SC_ERR_THREAD_INIT(49)] - thread "W#06-ix3^" failed to initialize: flags 0145
21/2/2023 -- 20:14:33 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - Engine initialization failed, aborting...

                                          Netgate 8200max

                                          M 1 Reply Last reply Reply Quote 0
                                          • M
                                            manilx @manilx
                                            last edited by

                                            @manilx Also changing back to autofp (when suricata doesn't start any longer) and trying to start on the interface results in it NOT starting.
                                            Will revert to working snapshot! Not further testing for the time being.

                                            Netgate 8200max

                                            bmeeksB 2 Replies Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.