23.01RC - Suricata stops working after Wireguard installed

bmeeks

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks I had a problem like this before. I think, an Upgrade should never depend on some third party lists hosted somewhere, lists etc should be updated after the whole upgrade procedure.

What do you mean? The lists in question were selected/added by the user as additional rules to download and install. There is nothing "upgrade" about them.

Are you saying the upgrade process should selectively ignore things the user has explicitly configured? If so, which things?

Bob.Dig

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

which things?

Everything third party.

bmeeks

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

which things?

Everything third party.

I still don't understand. What are you calling "third party"?

If you mean supporting packages such as curl or other shared libraries, then it is impossible not to upgrade those. If you don't, the new package you are installing will likely not work at all if it was compiled expecting updated shared libraries.

If you mean things like enabled rules downloads in the IDS/IPS package, then if those are not updated/downloaded the package is useless as it will have no rules to inspect traffic with.

Bob.Dig

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

If you mean things like enabled rules downloads in the IDS/IPS package, then if those are not updated/downloaded the package is useless as it will have no rules to inspect traffic with.

But they can be updated later, after the upgrade. This behavior had bitten me before, when my ISP back in the day Deutsche Telekom decided to slow down many downloads from AWS to the degree, that they would beginn but never end...

bmeeks

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

If you mean things like enabled rules downloads in the IDS/IPS package, then if those are not updated/downloaded the package is useless as it will have no rules to inspect traffic with.

But they can be updated later, after the upgrade. This behavior had bitten me before, when my ISP back in the day Deutsche Telekom decided to slow down many downloads from AWS to the degree, that they would beginn but never end...

So updated when, exactly? Wait for the user to do it manually? Or just never? How will the upgrade code know what to do when? What will it base a decision on?

Without updating/downloading new rules after installation, the IDS/IPS is non-functional. Do you think that is what the security admin expects after updating it?

I'm not sure you have thoroughly thought about all the adverse consequences of your assertion.

Gblenn

@bmeeks Although Suricata is not running because I prioritize Wireguard, I made a test doing a force update now. First without abuse.ch selected. It ran through as expected in less than a minute according to the System log which shows [Suricata] The Rules update has finished..

However, there is an issue with the popup window which never auto-closes.

Did force update with abuse.ch selected and the system log shows that it starts to download and update and within seconds it gets to Feodo (next being ABUSE) and then nothing...
Left it like that for 8 minutes and then deselected abuse.ch and saved in Global Settings.
System log now shows
[Suricata] Hide Deprecated Rules is enabled. Removing obsoleted rules [Suricata] The Rules update has finished.
Selecting abuse.ch again, and it shows up as updated within the same time frame as the other rule sets...?!

bmeeks

@gblenn:
My current guess is something is weird about the returned md5 file contents when curl attempts the download. It's like it fails to recognize the end of file and thus does not know the download is completed.

The PHP code uses the built-in PHP curl function to download the rules files. The exact same PHP code in the Suricata package is used for every downloaded rules file. The fact only certain ones fail points to an issue with how that particular site is doing something (that curl is not liking or not interpreting properly). But I have been unable to pin it down.

The only thing I have determined is that it's the download of the md5 checksum file that "hangs", but only for certain URLs.

Gblenn

@bmeeks Makes sense, so are there ways to "catch" this if it happens? A time out and then continue in this case since Suricata does not "depend" on it. At least not to the point that it doesn't work as long as other rule sets are installed or the previous one is still in place.

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks Makes sense, so are there ways to "catch" this if it happens? A time out and then continue in this case since Suricata does not "depend" on it. At least not to the point that it doesn't work as long as other rule sets are installed or the previous one is still in place.

There is a timeout in place in the PHP code. That's where my 10-minute value came from. Specifically these lines in the code:

curl_setopt($ch, CURLOPT_TIMEOUT, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TCP_KEEPALIVE, 1);

I am digging into it. It is something with the new PHP 8.1 or perhaps an updated curl library. This issue does not exist in the current pfSense RELEASE branch and the Suricata PHP code here is pretty much identical to that code. The only change is how the code reads some parameters from config.xml with the new PHP 8.1 rules, but none of that is part of the code area that is acting up.

Gblenn

@bmeeks I can't remember what version of Suricata I had in 22.05, but since things were was working there, would it be possible to downgrade somehow?

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks I can't remember what version of Suricata I had in 22.05, but since things were was working there, would it be possible to downgrade somehow?

No, you can't downgrade Suricata because the PHP code is radically changed to work with the new PHP 8.1 that is part of 23.01 and 2.7.0 CE. Older Suricata PHP code will not work at all in the upcoming pfSense release due the new 8.1 PHP environment.

I have found the problem with the "hang" during installation when installing certain rules files. It is related to an update in some shared libraries and has to do with HTTP2 support in cURL. I've tested my fix in a virtual machine, and the stalling is fixed. Will be submitting that pull request later this evening for the Netgate team to review and merge.

As for the other problem with Wireguard, I need help from the Netgate wirequard guru. He said he will investigate, but may not get to it until next week. So that bug may linger for a bit longer.

stephenw10

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

I think, an Upgrade should never depend on some third party lists hosted somewhere, lists etc should be updated after the whole upgrade procedure.

As always, the safest way to perform a pfSense upgrade is to uninstall packages, upgrade, then reinstall packages. Settings are retained.

Not convenient but...

Gblenn

@bmeeks Ah yes php was changed, well I just changed to the 22.05 version and let the 23.01 sit and wait for a while.

BTW, on the Wireguard issue, the passlist itself is complete as far as I can see.
/usr/local/etc/suricata/suricata_28603_igb1/passlist

That's how I knew the two WG IP's were next to be processed, looking at the error reported. After that I have 127.0.0.1/32 and then my LAN and VLAN subnets.
The only thing in that list which makes the WG IP's different, is that they start with 10 and have a /31 mask, where it's /32 and /24 for the others.

Bob.Dig

@stephenw10 said in 23.01RC - Suricata stops working after Wireguard installed:

Not convenient but...

True.
But Netgate could enforce a different behavior on the package maintainers (or maybe not). As a prosumer I say it is more of a standard to upgrade just the app and load "definitions" later. But most of the people probably don't have a hostile ISP. And pfSense is special in so may ways.

mrsunfire

I'm having the same issue and suricata service won't start.

3/2/2023 -- 09:45:45 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL

Gblenn

@mrsunfire And do you also have the WG tunnel IP's directly after that error?
If you go into /usr/local/etc/suricata/suricata_some_directory_name/
Check the passlist in that folder, do you see the IP's and can identify your WG IP's?

mrsunfire

@gblenn Yes the WG IP are in that list. If I uncheck the passlist (set to none) the service is starting.

Until I set it back to normal passlist I get this:

3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_29846_pppoe1/passlist.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 1.0.0.1 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 1.1.1.1 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.4.4 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.8.8 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.8.8/32 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 9.9.9.9 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.0.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.1.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.2.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.40.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL
3/2/2023 -- 10:15:25 - <Info> -- (RX#01-pppoe1) Packets 0, bytes 0
3/2/2023 -- 10:15:25 - <Info> -- (RX#01-pppoe1) Pcap Total:67 Recv:67 Drop:0 (0.0%).
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- cleaning up signature grouping structure... complete
3/2/2023 -- 10:15:25 - <Notice> -- Stats for 'pppoe1':  pkts: 0, drop: 0 (nan%), invalid chksum: 0

Right after 10.0.40.0/24 is the WG ip in the passlist file.

Gblenn

@mrsunfire said in 23.01RC - Suricata stops working after Wireguard installed:

Yes the WG IP are in that list. If I uncheck the passlist (set to none) the service is starting.

Ah, good point, why didn't I think of that... But I suppose you really want the passlist, else you get into all sorts of trouble.

Now I'm back on 22.05 so I can't test right now. If you start Suricata without passlist, and then go back and select your passlist and click save, does that stop Suricata? Suppose it does since you'd get that error again, when it works it's way through the list...

mrsunfire

@gblenn Exactly that happens. Did try this as workaround.

bmeeks

The most likely cause of the Wireguard Pass List problem is the pfSense system function returning an invalid final address entry. In the Suricata package code, the call to the pfSense function is the very last step of a larger function that is grabbing all of the VPN-related IP addresses. Here is the code:

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
	foreach (wg_get_tunnel_networks() as $wgn) {
		$vpns_arr[] = $wgn;
	}
}

The code is calling the pfSense function wg_get_tunnel_networks() and then walking the returned array adding each element to the local VPNs array variable. My suspicion is the last returned element from the pfSense function is the problem.

I don't have a Wireguard setup to test with, and I'm not familiar with the new Wireguard functionality in pfSense either. That's why I asked the Netgate Wireguard developer to lend a hand troubleshooting this one. This VPN gathering sub-function in Suricata was added sometime back by a Netgate developer who has since left the company.