Suricata 6.0.0 - After updating from 5.0.3 to 6.0.0, the interfaces will not pass any traffic

NRgia

Hello @bmeeks
Thank you for Suricata 6.0, I don't see it on the freshports site, so maybe you compile this yourself?

Anyways after updating from 5.0.3 to 6.0.0 the interfaces will not pass traffic anymore. The only way I can restore traffic is to disable Suricata.

I have tried to increase various buffers, but to no avail.

I run Netmap in
dev.netmap.admode: 1
1 forces native mode and fails if not available;

So even if the Netmap native mode will fail, Suricata should not start.

I don't see any errors in the system logs, or via dmesg

The WAN and LAN will drop any traffic after 1 minute after Suricata initializes
Wan Interface.png

The update was done by uninstalling the old package and reinstalling the new version. No changes were made by the user, and Netmap inline mode was supported in all previous Suricata versions.

If you need more information please let me know.

Also, I don't think there is a way to revert to Suricata 5.0.3. Or maybe there is a workaround in order not be be blocked by this?

Thank you

kiokoman

did you try legacy mode instead?

NRgia

@kiokoman Yep, it's working ok in Legacy mode.
But the point here is, it worked in Inline mode before 6.0.0, never had an issue.
Just lower throughput, but no other problems.

Also the way I run Netmap, as I described in my initial post:

dev.netmap.admode: 1

will force Netmap to start in Native mode. If Native mode is not supported(compatibility issue), then Netmap will fail with an error like: "Native mode not supported". Also if Netmap will not start Suricata shouldn't start also.

But in my case Netmap and Suricata start without any errors.

Cool_Corona

Youre probably seeing the same

https://forum.opnsense.org/index.php?topic=19851.0

NRgia

@Cool_Corona Good for you that you have guts to post about "that" project here.

I knew that they were experimenting with some custom kernels due to low throughput in FreeBSD 12. I also have an issue with low throughput, after updating to 2.5.0, but I don't think pfSense experimented with custom kernels, in order to fix the issue, so I don't see the connection.
Do you think it's FreeBSD code related from upstream?

Cool_Corona

@NRgia

Can you downgrade to 2.4.5p1 and test again?? Then we will have a clue if its FreeBSD or Suricata related.

NRgia

@Cool_Corona It worked on FreeBSD 12.2 also, because I am on 2.5.0-snapshots for a long time. All I did was updating the snapshot last night, and Suricata package from 5.0.3 to 6.0.0.
Suricata 6.0.0 is not available to 2.4.5p1, it's available only for development 2.5.0 version.

What I need is to be able to have 2 versions of Suricata in pfSense repo, just like in FreeBSD, for testing purposes, and be able to revert, not to reinstall the whole system.

I saw in FreeBSD with Unifi packages
You can install unifi-controller5 or unifi-controller6

bmeeks

The Suricata GUI package, unlike Snort, is not necessarily tied to specific binary versions. There is a vast difference in Suricata as used on Netagate ARM hardware versus AMD64/Intel hardware, so you can't mix versions there as Suricata versions newer than 4.x will not work at all on ARM hardware.

I warned in the Release Notes that there may be issues with Suricata-6.0.0. I tested on my VMware virtual machines using the e1000 virtual NIC which uses the em driver. It worked for me there.

The whole netmap device thing is turning into a small disaster (IMHO) in FreeBSD due to the recent changes to the NIC driver subsystem via the iflib wrapper. I know that some diehard FreeBSD fans may disagree with me, but there are several reported issues with throughput and even basic stability with regards to the netmap device and different hardware.

I will say it again to be sure everyone understands. This is not necessarily problems within Suricata or Snort. All these programs do is open the kernel netmap device via API calls and then attempt to use it via the published interface. The issue is there are multiple API versions that differ in different FreeBSD versions. And the different API versions of the netmap interface are not all backwards compatible. Netmap initially sounded like a great thing, but it has disappointed me greatly both in terms of reliability and performance.

NRgia

@bmeeks I didn't say you did not post a warning, I'm happy to test when it comes to Suricata, but when something happens, I must be able to work around that situation, in order to be able to test again later on.

My hint was to have 2 versions, in order to be able to revert back. Thank you for providing the solution.

Regarding Netmap, FreeBSD and Suricata we've discussed the situation in other topics like the following:
https://forum.netgate.com/topic/144979/porting-bge-driver-to-iflib/21?_=1604173006203

https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0/16?_=1604173006206

We've tested something together also, so I understand the situation.

I also prefer Linux, and I think slowly some of the die hard supporters of FreeBSD like IXsystems ( please see TrueNas Scale), Netgate (TNSR) will ditch FreeBSD.

But until then I will try to report my findings, maybe I can prevent others from hitting the same issues as I.

Before reverting, can I test something else, to get to the root cause?

Thank you

bmeeks

@NRgia said in Suricata 6.0.0 - After updating from 5.0.3 to 6.0.0, the interfaces will not pass any traffic:

@bmeeks I didn't say you did not post a warning, I'm happy to test when it comes to Suricata, but when something happens, I must be able to work around that situation, in order to be able to test again later on.

My hint was to have 2 versions, in order to be able to revert back. Thank you for providing the solution.

Regarding Netmap, FreeBSD and Suricata we've discussed the situation in other topics like the following:
https://forum.netgate.com/topic/144979/porting-bge-driver-to-iflib/21?_=1604173006203

https://forum.netgate.com/topic/154014/netmap-not-supported-for-intel-x553-driver-in-pfsense-2-5-0/16?_=1604173006206

We've tested something together also, so I understand the situation.

I also prefer Linux, and I think slowly some of the die hard supporters of FreeBSD like IXsystems ( please see TrueNas Scale), Netgate (TNSR) will ditch FreeBSD.

But until then I will try to report my findings, maybe I can prevent other from hitting the same issues as I.

Before reverting, can I test something else, to get to the root cause?

Thank you

I was not targeting you with my reply about the "warning" -- just repeating for others that may read this thread.

The way pkg works on FreeBSD makes it impossible to have two packages with the same name but different versions. It would be possible to have a Suricata package with a different name (say Suricata6 instead of Suricata, for instance). But that also has a downside because you have to include extra complexity to let pkg know what other versions can't coexist.

Honestly I have not yet identified what may be different in Suricata 6.0.0 versus 5.0.3 in terms of the netmap module. A quick look earlier today turned up no differences. If one version works but the other does not, then obviously something changed in the binary. As I keep saying, I do absolutely nothing to the upstream Suricata binary code when it comes to netmap. No changes or patches at all. Been that way from the beginning, too (for netmap and Inline IPS Mode).

I don't know what else you can test.

NRgia

@bmeeks Ok, then I will roll back to 5.0.3
Thanks again

bmeeks

@NRgia said in Suricata 6.0.0 - After updating from 5.0.3 to 6.0.0, the interfaces will not pass any traffic:

@bmeeks Ok, then I will roll back to 5.0.3
Thanks again

You can use the 5.0.3 binary with the 6.0.0 GUI code with no issues at all. Snort does not work that way, but Suricata can.

bmeeks

Another user has reported a different fairly severe issue with Suricata-6.0.0. The problems are within the binary and not the PHP GUI code. The binary will stop generating alerts after a period of time. The only way to get alerts after that is to restart Suricata.

There is also an issue with shutdowns. It can take forever for a Suricata binary process to shutdown.

I am most likely going to ask the pfSense team to pull the 6.0.0 binary package down and revert the binary to 5.0.3. There are bug reports on Suricata Redmine about the "no alerts" issue plus some others. IMHO it appears some recent changes upstream have not gone well.

NRgia

@bmeeks Just as a followup
I've compiled Suricata 5.0.4 myself, using your hints.
I can confirm that all the interfaces are passing traffic, and I see Drops in the Alerts tab.
I have tested both in Inline and Legacy mode. Until now all is good.

Thanks again Bill for your time

bmeeks

An updated Suricata package has now been posted that officially reverts the Suricata binary to the 5.0.4 version from 6.0.0. The new GUI package is Suricata-6.0.0_1.