23.01RC - Suricata stops working after Wireguard installed
-
@bmeeks Ok first time doing that, but I'll manage...
Actually, there's another Suricata issue that I found, I think...
If I have selected the 'ABUSE.ch SSL Blacklist rules' in Suricata Global settings, package installation hangs at the point where it reports downloading of abuse.ch.
Perhaps another bug? Is it also known or does it need reporting?
-
@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:
If I have selected the 'ABUSE.ch SSL Blacklist rules' in Suricata Global settings, package installation hangs at the point where it reports downloading of abuse.ch.
Perhaps another bug? Is it also known or does it need reporting?That one is weird. I have investigated but been unable to determine the cause. It will eventually timeout and proceed (after 10 minutes if I recall correctly). I've not given up, but also have had no success in finding a root cause.
There is no need to report that one. There is already an existing bug report for it with a different URL for the rules, but the symptoms are the same and the underlying cause is likely the same.
The code in the Suricata package around that functionality has not changed, but the snapshots include a new FreeBSD operating system version and most of the other supporting packages have updated as well. I'm thinking an update there is the cause.
-
@bmeeks I had a problem like this before. I think, an Upgrade should never depend on some third party lists hosted somewhere, lists etc should be updated after the whole upgrade procedure.
-
@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:
@bmeeks I had a problem like this before. I think, an Upgrade should never depend on some third party lists hosted somewhere, lists etc should be updated after the whole upgrade procedure.
What do you mean? The lists in question were selected/added by the user as additional rules to download and install. There is nothing "upgrade" about them.
Are you saying the upgrade process should selectively ignore things the user has explicitly configured? If so, which things?
-
@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:
which things?
Everything third party.
-
@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:
@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:
which things?
Everything third party.
I still don't understand. What are you calling "third party"?
If you mean supporting packages such as
curlor other shared libraries, then it is impossible not to upgrade those. If you don't, the new package you are installing will likely not work at all if it was compiled expecting updated shared libraries.If you mean things like enabled rules downloads in the IDS/IPS package, then if those are not updated/downloaded the package is useless as it will have no rules to inspect traffic with.
-
@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:
If you mean things like enabled rules downloads in the IDS/IPS package, then if those are not updated/downloaded the package is useless as it will have no rules to inspect traffic with.
But they can be updated later, after the upgrade. This behavior had bitten me before, when my ISP back in the day Deutsche Telekom decided to slow down many downloads from AWS to the degree, that they would beginn but never end...
-
@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:
@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:
If you mean things like enabled rules downloads in the IDS/IPS package, then if those are not updated/downloaded the package is useless as it will have no rules to inspect traffic with.
But they can be updated later, after the upgrade. This behavior had bitten me before, when my ISP back in the day Deutsche Telekom decided to slow down many downloads from AWS to the degree, that they would beginn but never end...
So updated when, exactly? Wait for the user to do it manually? Or just never? How will the upgrade code know what to do when? What will it base a decision on?
Without updating/downloading new rules after installation, the IDS/IPS is non-functional. Do you think that is what the security admin expects after updating it?
I'm not sure you have thoroughly thought about all the adverse consequences of your assertion.
-
@bmeeks Although Suricata is not running because I prioritize Wireguard, I made a test doing a force update now. First without abuse.ch selected. It ran through as expected in less than a minute according to the System log which shows [Suricata] The Rules update has finished..
However, there is an issue with the popup window which never auto-closes.
Did force update with abuse.ch selected and the system log shows that it starts to download and update and within seconds it gets to Feodo (next being ABUSE) and then nothing...
Left it like that for 8 minutes and then deselected abuse.ch and saved in Global Settings.
System log now shows
[Suricata] Hide Deprecated Rules is enabled. Removing obsoleted rules [Suricata] The Rules update has finished.
Selecting abuse.ch again, and it shows up as updated within the same time frame as the other rule sets...?! -
@gblenn:
My current guess is something is weird about the returned md5 file contents whencurlattempts the download. It's like it fails to recognize the end of file and thus does not know the download is completed.The PHP code uses the built-in PHP
curlfunction to download the rules files. The exact same PHP code in the Suricata package is used for every downloaded rules file. The fact only certain ones fail points to an issue with how that particular site is doing something (thatcurlis not liking or not interpreting properly). But I have been unable to pin it down.The only thing I have determined is that it's the download of the md5 checksum file that "hangs", but only for certain URLs.
-
@bmeeks Makes sense, so are there ways to "catch" this if it happens? A time out and then continue in this case since Suricata does not "depend" on it. At least not to the point that it doesn't work as long as other rule sets are installed or the previous one is still in place.
-
@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:
@bmeeks Makes sense, so are there ways to "catch" this if it happens? A time out and then continue in this case since Suricata does not "depend" on it. At least not to the point that it doesn't work as long as other rule sets are installed or the previous one is still in place.
There is a timeout in place in the PHP code. That's where my 10-minute value came from. Specifically these lines in the code:
curl_setopt($ch, CURLOPT_TIMEOUT, 0); curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); curl_setopt($ch, CURLOPT_TCP_KEEPALIVE, 1);I am digging into it. It is something with the new PHP 8.1 or perhaps an updated
curllibrary. This issue does not exist in the current pfSense RELEASE branch and the Suricata PHP code here is pretty much identical to that code. The only change is how the code reads some parameters fromconfig.xmlwith the new PHP 8.1 rules, but none of that is part of the code area that is acting up. -
@bmeeks I can't remember what version of Suricata I had in 22.05, but since things were was working there, would it be possible to downgrade somehow?
-
@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:
@bmeeks I can't remember what version of Suricata I had in 22.05, but since things were was working there, would it be possible to downgrade somehow?
No, you can't downgrade Suricata because the PHP code is radically changed to work with the new PHP 8.1 that is part of 23.01 and 2.7.0 CE. Older Suricata PHP code will not work at all in the upcoming pfSense release due the new 8.1 PHP environment.
I have found the problem with the "hang" during installation when installing certain rules files. It is related to an update in some shared libraries and has to do with HTTP2 support in cURL. I've tested my fix in a virtual machine, and the stalling is fixed. Will be submitting that pull request later this evening for the Netgate team to review and merge.
As for the other problem with Wireguard, I need help from the Netgate wirequard guru. He said he will investigate, but may not get to it until next week. So that bug may linger for a bit longer.
-
@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:
I think, an Upgrade should never depend on some third party lists hosted somewhere, lists etc should be updated after the whole upgrade procedure.
As always, the safest way to perform a pfSense upgrade is to uninstall packages, upgrade, then reinstall packages. Settings are retained.
Not convenient but...
-
@bmeeks Ah yes php was changed, well I just changed to the 22.05 version and let the 23.01 sit and wait for a while.
BTW, on the Wireguard issue, the passlist itself is complete as far as I can see.
/usr/local/etc/suricata/suricata_28603_igb1/passlistThat's how I knew the two WG IP's were next to be processed, looking at the error reported. After that I have 127.0.0.1/32 and then my LAN and VLAN subnets.
The only thing in that list which makes the WG IP's different, is that they start with 10 and have a /31 mask, where it's /32 and /24 for the others. -
@stephenw10 said in 23.01RC - Suricata stops working after Wireguard installed:
Not convenient but...
True.
But Netgate could enforce a different behavior on the package maintainers (or maybe not). As a prosumer I say it is more of a standard to upgrade just the app and load "definitions" later. But most of the people probably don't have a hostile ISP. And pfSense is special in so may ways. -
I'm having the same issue and suricata service won't start.
3/2/2023 -- 09:45:45 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL -
@mrsunfire And do you also have the WG tunnel IP's directly after that error?
If you go into /usr/local/etc/suricata/suricata_some_directory_name/
Check the passlist in that folder, do you see the IP's and can identify your WG IP's? -
@gblenn Yes the WG IP are in that list. If I uncheck the passlist (set to none) the service is starting.
Until I set it back to normal passlist I get this:
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_29846_pppoe1/passlist. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 1.0.0.1 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 1.1.1.1 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.4.4 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.8.8 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.8.8/32 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 9.9.9.9 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.0.0/24 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.1.0/24 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.2.0/24 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.40.0/24 from assigned Pass List. 3/2/2023 -- 10:15:24 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL 3/2/2023 -- 10:15:25 - <Info> -- (RX#01-pppoe1) Packets 0, bytes 0 3/2/2023 -- 10:15:25 - <Info> -- (RX#01-pppoe1) Pcap Total:67 Recv:67 Drop:0 (0.0%). 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks 3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts 3/2/2023 -- 10:15:25 - <Info> -- cleaning up signature grouping structure... complete 3/2/2023 -- 10:15:25 - <Notice> -- Stats for 'pppoe1': pkts: 0, drop: 0 (nan%), invalid chksum: 0Right after 10.0.40.0/24 is the WG ip in the passlist file.
