6100 SLOW in comparison to Protectli FW6E
-
@bmeeks Looking forward to!
-
@manilx:
Do you still have your Protectli appliance and can you configure it with OPNsense again? If so, I would really be interested in how this test Suricata package from the OPNsense developer works for you in terms of eliminating the stall/hang when using Suricata in IPS mode.Here is a link to the link for the package as posted on the Suricata Redmine site: https://redmine.openinfosecfoundation.org/issues/5744#note-69. The package was created by Franco for use on OPNsense, so it will not work on pfSense. A different user in that Redmine thread had been working with us for testing, but he seems to have dropped off the radar over the last week or so.
The test package at the link above contains the latest iteration of a fix for the stall/hang condition in netmap that happens during heavy traffic transfers such as speed tests when using IPS mode in Suricata. I have also included this fix in the next release of Suricata 6.0.10 on pfSense, but some early feedback would be helpful.
-
@bmeeks Hi
Yes have it with pfsense configured as backup.
Still have my OPNsense proxmox VM, which run for 9 months until....
I can test it there.
Tell me how I can install this patch and how you want me to test this.
Netgate 8200max
-
@manilx P.S: I tried running OPNsense on the Protectli before but it got sporadic crashes with lost internet access and could not afford to debug that as VM was running fine.
-
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks Hi
Yes have it with pfsense configured as backup.
Still have my OPNsense proxmox VM, which run for 9 months until....
I can test it there.
Tell me how I can install this patch and how you want me to test this.
Suricata in OPNsense would be suffering from the bug this proposed patch corrects. It gets technical, but Suricata on OPNsense uses netmap with only a single pair of host stack rings while we have been using multiple host stack rings with Suricata on pfSense. The bug identified affects both, but due to having only a single pair of host stack rings OPNsense Suricata is more vulnerable to the stall.
Typical usage with sort of low data rates will not cause the bug to manifest when using Suricata on pfSense. But heavy traffic loading can lead to the stall as you experienced when I suggested earlier in this thread to switch to
runmode = workers.I am hoping the patch I provided in the Suricata Redmine thread solves the stall issue. The OPNsense developer prepared the test Suricata package to see, and that's the package I was hoping you might could test. I have never used OPNsense, so I can't tell you exactly how to install the test package. My guess is do it this way --
Get OPNsense up and running on the device with Suricata installed/enabled/configured (or whatever you normally do to use Suricata in OPNsense). Then, once it is running (I realize it quickly crashes), go to a shell prompt and install the test package using the
pkgutility like so:pkg install <package_name.pkg>I would download the package using the URL in the link I provided and then store it locally on the firewall (maybe in /tmp or /root), then install it from there. What it should do is simply overwrite the existing Suricata binary with this new version.
After installing the new binary, repeat your speed tests and see if things are more stable. I'm hoping they will be. If you are using a virtual NIC (or unsupported hardware that results in netmap using its generic adapter), then it's possible you still get a hang. Work is continuing on some bugs in the way the generic netmap adapter works. We don't have that on pfSense because for now we limit within the GUI which network adapters are allowed when running Suricata in IPS mode. You can't select that mode in the GUI if your NIC does not support native-mode netmap operation.
-
@bmeeks OK. I'll try to do this saturday morning (GMT). Already restored the VM.
-
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks OK. I'll try to do this saturday morning (GMT). Already restored the VM.
Thanks! I will keep my fingers crossed that the patch helps.
-
@bmeeks I will install suricata, add a lot of rules and do Speedtest.net a few times.
Guess that should do it.
I'll report asap. If I can do it earlier I will.
-
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks I will install suricata, add a lot of rules and do Speedtest.net a few times.
Guess that should do it.
I'll report asap. If I can do it earlier I will.
Thanks -- I did not mention it, but after installing the updated Suricata binary you will need to restart Suricata. In fact, it would be even better to stop it, install the updated package, then restart it.
On OPNsense I don't know what kind of manual control you have in the GUI for stopping and starting. Might take a reboot after updating the binary, but I just don't know having never used OPNsense.
-
@bmeeks If I remember well I don't have it installed, otherwise I'll uninstall first and then install the test package manually.
-
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks If I remember well I don't have it installed, otherwise I'll uninstall first and then install the test package manually.
One more point, too. If you are running either pfSense or OPNsense as a virtual machine, the choice of virtual NIC will matter in how well the bug fix works (unless you are doing passthrough of a real hardware NIC). As I mentioned, when you use a NIC that does not support native netmap operation, the netmap kernel code creates something called a generic netmap adapter. That is a kludge and is much slower performance-wise than native-mode netmap. There are also currently some known issues with the generic netmap adapter when stressed with high throughput. A team of developers is actively working on those issues and submitting their proposed fixes to FreeBSD for inclusion.
So, saying all the above to note that you may install the package and still experience a stall, but I'm hoping things run longer at least with the patched Suricata. And it is possible things may run flawlessly as well (that's my hope).
-
2 Virtio network cards. No passthrough.
I'll run a few speedtests but will not keep it running for a longer time... Hope this helps.
Netgate 8200max
-
@manilx P.S. I'll do it later tonight in a couple of hours.
-
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
2 Virtio network cards. No passthrough.
I'll run a few speedtests but will not keep it running for a longer time... Hope this helps.
The main thing I want to see is if there is a change in the positive direction. Meaning either it works without stalling, or it works noticeably longer before stalling.
Did this setup work okay with Suricata prior to November of last year? If so, I'm hoping it returns the same level of reliability you had prior to the end of last year when the new version of Suricata 6.0.9 was rolled out with OPNsense.
-
@bmeeks Well, I have posted my issue with OPNsense in a link above.
I lost internet connection out of the blue. But not consistently and I couldn't force it!
I don't know if it was suricata back then. What I know it was when I upgraded OPNsense to a newer release (all in the post above).SO I don't think this test can help. I won't have the VM running for longer than a few speedtests.
If this is OK I'll do it, if not then I'll spare me the hassle.
I moved for good to pfsense (because of the above and the "attitude" I got from the developer!!).
I now have pfsense hardware and all. -
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks Well, I have posted my issue with OPNsense in a link above.
I lost internet connection out of the blue. But not consistently and I couldn't force it!
I don't know if it was suricata back then. What I know it was when I upgraded OPNsense to a newer release (all in the post above).SO I don't think this test can help. I won't have the VM running for longer than a few speedtests.
If this is OK I'll do it, if not then I'll spare me the hassle.
I moved for good to pfsense (because of the above and the "attitude" I got from the developer!!).
I now have pfsense hardware and all.I think I recall checking out your post in the link, but the details I have forgotten. Been juggling a lot of balls in the air with various Suricata changes recently, and I'm not a spring chicken anymore and things can escape my brain into the ether
.Is your VM setup of pfSense using CE or Plus? If using either 23.01 Plus, or 2.7 CE DEVEL, I can send you a "fixed" Suricata binary for pfSense you can test with. I can also make you one for 2.6 CE, but it will take a bit to build.
-
@bmeeks VM is only the "old" OPNsense.
I'm running pfsense+ 23.01 on a brand new 8200max now. -
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks VM is only the "old" OPNsense.
I'm running pfsense+ 23.01 on a brand new 8200max now.That's fine. If you can fire up the old VM and test with the Suricata package Franco linked in the Suricata Redmine Issue thread that would be helpful. Just bring it up and run a few speed tests. Based on some of the earlier reports from other users, I would expect a speed test to trigger the stall fairly quickly: or at least several back-to-back speed tests on Gigabit or higher links should trigger it. If the speed tests work well, then that increases my confidence in the patch.
My problem with testing locally is that currently I don't have a suitable setup in place. I hope to free up some hardware next week that will help. The bug needs to be exercised with really heavy traffic loads in order to show up. I've never been able to get it to show with my feeble VMware Workstation testing setup (not surprised at all there, though).
-
@bmeeks OK!
Updated OPNsense to the latest version.
Installed the suricata test package.
Run half a dozen speedtests saturating the line at 930+
No problem so far.
Reverted to 8200.
Hope this helps
-
@manilx said in 6100 SLOW in comparison to Protectli FW6E:
@bmeeks OK!
Updated OPNsense to the latest version.
Installed the suricata test package.
Run half a dozen speedtests saturating the line at 930+
No problem so far.
Reverted to 8200.
Hope this helps
Yes! This is very helpful. It proves the patch works. Sorry to be a little late replying. Had to leave home to attend a meeting and just returned.
Thank you very much for testing. I will pass the word on to the Suricata team.
