23.01RC - Suricata stops working after Wireguard installed

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks Makes sense, so are there ways to "catch" this if it happens? A time out and then continue in this case since Suricata does not "depend" on it. At least not to the point that it doesn't work as long as other rule sets are installed or the previous one is still in place.

There is a timeout in place in the PHP code. That's where my 10-minute value came from. Specifically these lines in the code:

curl_setopt($ch, CURLOPT_TIMEOUT, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_TCP_KEEPALIVE, 1);

I am digging into it. It is something with the new PHP 8.1 or perhaps an updated curl library. This issue does not exist in the current pfSense RELEASE branch and the Suricata PHP code here is pretty much identical to that code. The only change is how the code reads some parameters from config.xml with the new PHP 8.1 rules, but none of that is part of the code area that is acting up.

Gblenn

@bmeeks I can't remember what version of Suricata I had in 22.05, but since things were was working there, would it be possible to downgrade somehow?

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks I can't remember what version of Suricata I had in 22.05, but since things were was working there, would it be possible to downgrade somehow?

No, you can't downgrade Suricata because the PHP code is radically changed to work with the new PHP 8.1 that is part of 23.01 and 2.7.0 CE. Older Suricata PHP code will not work at all in the upcoming pfSense release due the new 8.1 PHP environment.

I have found the problem with the "hang" during installation when installing certain rules files. It is related to an update in some shared libraries and has to do with HTTP2 support in cURL. I've tested my fix in a virtual machine, and the stalling is fixed. Will be submitting that pull request later this evening for the Netgate team to review and merge.

As for the other problem with Wireguard, I need help from the Netgate wirequard guru. He said he will investigate, but may not get to it until next week. So that bug may linger for a bit longer.

stephenw10

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

I think, an Upgrade should never depend on some third party lists hosted somewhere, lists etc should be updated after the whole upgrade procedure.

As always, the safest way to perform a pfSense upgrade is to uninstall packages, upgrade, then reinstall packages. Settings are retained.

Not convenient but...

Gblenn

@bmeeks Ah yes php was changed, well I just changed to the 22.05 version and let the 23.01 sit and wait for a while.

BTW, on the Wireguard issue, the passlist itself is complete as far as I can see.
/usr/local/etc/suricata/suricata_28603_igb1/passlist

That's how I knew the two WG IP's were next to be processed, looking at the error reported. After that I have 127.0.0.1/32 and then my LAN and VLAN subnets.
The only thing in that list which makes the WG IP's different, is that they start with 10 and have a /31 mask, where it's /32 and /24 for the others.

Bob.Dig

@stephenw10 said in 23.01RC - Suricata stops working after Wireguard installed:

Not convenient but...

True.
But Netgate could enforce a different behavior on the package maintainers (or maybe not). As a prosumer I say it is more of a standard to upgrade just the app and load "definitions" later. But most of the people probably don't have a hostile ISP. And pfSense is special in so may ways.

mrsunfire

I'm having the same issue and suricata service won't start.

3/2/2023 -- 09:45:45 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL

Gblenn

@mrsunfire And do you also have the WG tunnel IP's directly after that error?
If you go into /usr/local/etc/suricata/suricata_some_directory_name/
Check the passlist in that folder, do you see the IP's and can identify your WG IP's?

mrsunfire

@gblenn Yes the WG IP are in that list. If I uncheck the passlist (set to none) the service is starting.

Until I set it back to normal passlist I get this:

3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_29846_pppoe1/passlist.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 1.0.0.1 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 1.1.1.1 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.4.4 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.8.8 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 8.8.8.8/32 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 9.9.9.9 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.0.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.1.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.2.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Info> -- alert-pf -> Added IPv4 address 10.0.40.0/24 from assigned Pass List.
3/2/2023 -- 10:15:24 - <Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL
3/2/2023 -- 10:15:25 - <Info> -- (RX#01-pppoe1) Packets 0, bytes 0
3/2/2023 -- 10:15:25 - <Info> -- (RX#01-pppoe1) Pcap Total:67 Recv:67 Drop:0 (0.0%).
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output inserted 0 IP address blocks
3/2/2023 -- 10:15:25 - <Info> -- alert-pf output processed 0 alerts
3/2/2023 -- 10:15:25 - <Info> -- cleaning up signature grouping structure... complete
3/2/2023 -- 10:15:25 - <Notice> -- Stats for 'pppoe1':  pkts: 0, drop: 0 (nan%), invalid chksum: 0

Right after 10.0.40.0/24 is the WG ip in the passlist file.

Gblenn

@mrsunfire said in 23.01RC - Suricata stops working after Wireguard installed:

Yes the WG IP are in that list. If I uncheck the passlist (set to none) the service is starting.

Ah, good point, why didn't I think of that... But I suppose you really want the passlist, else you get into all sorts of trouble.

Now I'm back on 22.05 so I can't test right now. If you start Suricata without passlist, and then go back and select your passlist and click save, does that stop Suricata? Suppose it does since you'd get that error again, when it works it's way through the list...

mrsunfire

@gblenn Exactly that happens. Did try this as workaround.

bmeeks

The most likely cause of the Wireguard Pass List problem is the pfSense system function returning an invalid final address entry. In the Suricata package code, the call to the pfSense function is the very last step of a larger function that is grabbing all of the VPN-related IP addresses. Here is the code:

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
	foreach (wg_get_tunnel_networks() as $wgn) {
		$vpns_arr[] = $wgn;
	}
}

The code is calling the pfSense function wg_get_tunnel_networks() and then walking the returned array adding each element to the local VPNs array variable. My suspicion is the last returned element from the pfSense function is the problem.

I don't have a Wireguard setup to test with, and I'm not familiar with the new Wireguard functionality in pfSense either. That's why I asked the Netgate Wireguard developer to lend a hand troubleshooting this one. This VPN gathering sub-function in Suricata was added sometime back by a Netgate developer who has since left the company.

Gblenn

@bmeeks Yes I saw that you highlighted that in Redmine.
I did however go in to check and that same code block is present in the version I have on 22.05. And there things are are working fine...

Not sure you need a full working Wireguard setup to be able to test it.
I'm guessing all you need is to create a tunnel and add one peer (where the tunnel IP will be listed).
Whether or not the tunnel is actually up shouldn't impact the passlist?

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

I did however go in to check and that same code block is present in the version I have on 22.05. And there things are are working fine...

It will almost certainly be related to the new PHP 8.1 that comes with the latest pfSense snapshots. A lot of the old ways of doing things in PHP code (especially with arrays) got force-deprecated with the bump from PHP 7.x to 8.1. PHP 7.x and earlier would sort of hold your hand and fix things gracefully in the background in some cases. PHP 8.1 no longer does that. It enforces some hard and fast rules about arrays and strings in particular.

That's why I made my earlier comment about being unable to rollback Suricata but stay current in pfSense DEVEL. The PHP code is incompatible, and PHP is what all of the GUI is written with.

bmeeks

@gblenn:
Would you be willing to test the fix below? It requires a small edit to the /usr/local/pkg/suricata/suricata.inc PHP source file. I'm having trouble reproducing the issue in my test system, but it is likely because I do not have Wireguard configured properly yet. But in reviewing the Wireguard package code, I think I see a place where a problem could occur.

Below is the new code. The change is on line 4299 of the file (so almost at the very bottom of that file).

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
	foreach (wg_get_tunnel_networks() as $wgn) {
		$vpns_arr[] = $wgn['network'] . '/' . $wgn['mask'];
	}
}

If you can perform this test, please report back the results here. Thanks!

Edit: or alternatively, you could copy and paste the following PHP code into an empty file and then execute it on the firewall. It will print out the Wireguard tunnel networks.

<?php
require_once('wireguard/includes/wg.inc');

global $g;
$vpns_arr = array();

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
	foreach (wg_get_tunnel_networks() as $wgn) {
		print_r($wgn, false);
		$vpns_arr[] = $wgn;
	}
	print PHP_EOL . "\$vpns_arr[] contents = " . PHP_EOL;
	print_r($vpns_arr, false);
} else {
	print "Failed to find the wg_get_tunnel_networks() function";
}
?>

So, for example, paste the code above into /tmp/test.php and then at a shell prompt on the firewall execute the code with this command:

php -f /tmp/test.php

Posting what the above code outputs will help me understand the problem better. If you do not want to divulge the IP addresses on the public forum, then PM me and I can supply my email address.

Gblenn

@bmeeks Ok, not sure this was a conclusive test, as I ran this on a VM where I was using vtnet instead of igb. Had to mess around with interface settings a bit to get it up and running.

So php -f /tmp/test.php gave this:

Array
(
[network] => 10.6.250.0
[mask] => 31
[tun] => tun_wg1
[descr] => One_VPN
)

$vpns_arr[] contents =
Array
(
[0] => Array
(
[network] => 10.6.250.0
[mask] => 31
[tun] => tun_wg1
[descr] => One_VPN
)

)

That only shows one of the tunnels though... Also, after fixing the interface assignments, I also changed the names of the WG interfaces, but that was not reflected in the output.
The output looks exactly the same on 22.05 btw...

Editing suricata.inc to look like this:

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
	foreach (wg_get_tunnel_networks() as $wgn) {
		$vpns_arr[] = $wgn['network'] . '/' . $wgn['mask'];
	}

Gave the following error (twice, because of two interfaces?):

PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata.inc, Line: 4310, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/suricata/suricata.inc:4310
Stack trace:
#0 /usr/local/pkg/suricata/suricata.inc(599): suricata_get_vpns_list()
#1 /usr/local/pkg/suricata/suricata_generate_yaml.php(46): suricata_build_list(Array, 'default')
#2 /usr/local/pkg/suricata/suricata.inc(3800): include('/usr/local/pkg/...')
#3 /usr/local/pkg/suricata/suricata.inc(933): suricata_generate_yaml(Array)
#4 /tmp/suricata_vtnet128603_startcmd.php(8): sync_suricata_package_config()
#5 {main}

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks Ok, not sure this was a conclusive test, as I ran this on a VM where I was using vtnet instead of igb. Had to mess around with interface settings a bit to get it up and running.

So php -f /tmp/test.php gave this:

Array
(
[network] => 10.6.250.0
[mask] => 31
[tun] => tun_wg1
[descr] => One_VPN
)

$vpns_arr[] contents =
Array
(
[0] => Array
(
[network] => 10.6.250.0
[mask] => 31
[tun] => tun_wg1
[descr] => One_VPN
)

)

That only shows one of the tunnels though... Also, after fixing the interface assignments, I also changed the names of the WG interfaces, but that was not reflected in the output.
The output looks exactly the same on 22.05 btw...

Editing suricata.inc to look like this:

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
foreach (wg_get_tunnel_networks() as $wgn) {
$vpns_arr[] = $wgn['network'] . '/' . $wgn['mask'];
}

Gave the following error (twice, because of two interfaces?):

PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata.inc, Line: 4310, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/suricata/suricata.inc:4310
Stack trace:
#0 /usr/local/pkg/suricata/suricata.inc(599): suricata_get_vpns_list()
#1 /usr/local/pkg/suricata/suricata_generate_yaml.php(46): suricata_build_list(Array, 'default')
#2 /usr/local/pkg/suricata/suricata.inc(3800): include('/usr/local/pkg/...')
#3 /usr/local/pkg/suricata/suricata.inc(933): suricata_generate_yaml(Array)
#4 /tmp/suricata_vtnet128603_startcmd.php(8): sync_suricata_package_config()
#5 {main}

Thanks for testing. I think the actual error is somewhere else, but this code exposes it. The string offset error error you get when you modify the suricata.inc file like I suggested is a PHP 8.1 thing. That error will not usually happen in earlier PHP versions, so that's why the same setup works for in 22.05. Earlier pfSense versions use PHP 7.4.

I will keep digging into this.

bmeeks

After some more digging around, I think this problem may reside in the custom blocking plugin compiled into Suricata for use on pfSense. The error message "<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL" is coming from the Radix Tree code in the Suricata binary. There was a change in the Radix Tree itself, and some changes in the custom blocking plugin between Suricata 6.0.4 which is the current binary in pfSense Plus 22.05 and Suricata 6.0.8 which is the current binary in pfSense Plus 23.01. The problem may be related to the subnet mask.

Patch

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

enforce a different behavior ... to upgrade just the app and load "definitions" later.

I suspect that is basically how updates are done already.

The difference you are suggesting is for the update to clearly separate application update from configuration update. Which should allow relatively graceful termination of the latter if if fails, and a warning "User configuration update did not complete, manual re-cofiguration maybe required".

Gblenn

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

After some more digging around, I think this problem may reside in the custom blocking plugin compiled into Suricata for use on pfSense. The error message "<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL" is coming from the Radix Tree code in the Suricata binary. There was a change in the Radix Tree itself, and some changes in the custom blocking plugin between Suricata 6.0.4 which is the current binary in pfSense Plus 22.05 and Suricata 6.0.8 which is the current binary in pfSense Plus 23.01. The problem may be related to the subnet mask.

I decided to do some more testing and discovered some wierd issues with the passlist. First of all, I get a discrepancy between what Suricata reports to be doing, like adding an IP to the passlist, and what it then shows when "Loading and parsing passlist."

Going to Interfaces > myVPN (tun_wg0) and changing the IP to e.g. 192.n.n.n or 172.n.n.n the log shows the following:
-> adding firewall interface tun_wg0 IPv4 address 172.6.210.0 to automatic interface IP Pass List.
-> adding firewall interface tun_wg1 IPv4 address 192.168.250.0 to automatic interface IP Pass List.
output device (regular) initialized: block.log
-> Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_28603_vtnet1/passlist.
-> Added IPv4 address 8.8.8.8/32 from assigned Pass List.
This is where it stopped earlier
-> Added IPv4 address 10.6.210.0/31 from assigned Pass List.
-> Added IPv4 address 10.6.250.0/31 from assigned Pass List.
-> Added IPv4 address 127.0.0.1/32 from assigned Pass List.

Suricata now works it's way through and actually starts. However it does throw the same PHP error as before.

The actual list found in /usr/local/etc/suricata/suricata_28603_vtnet1 never changes, no matter what changes I make in Interfaces or inside Wireguard.

Manually removing it and restarting Suricata results in a crash, it does not seem to be created or changed? Rebooting makes no difference...

Manually editing the list and setting the same IP's as the interface has, AND using mask /31 in the passlist, will result in Suricata not starting with the same fault reported as before during parsing (this is basically the original setting from a working environment):
<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL

Using mask /32 in the passlist allows it to continue through, regardless of the interface having the mask /31.

Making a change in the Interface and clicking Save Changes gives the following in suricata.log
-> Received notification of IP address change on firewall interface tun_wg1.
-> deleted address 10.6.250.0 from automatic firewall interface IP Pass List.
-> Received notification of IP address change on firewall interface tun_wg1.
-> added address 10.6.250.0 to automatic firewall interface IP Pass List.

And then the error shows up:
PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata.inc, Line: 4310, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/suricata/suricata.inc:4310
Stack trace:
#0 /usr/local/pkg/suricata/suricata.inc(599): suricata_get_vpns_list()
#1 /usr/local/pkg/suricata/suricata_generate_yaml.php(46): suricata_build_list(Array, 'default')
#2 /usr/local/pkg/suricata/suricata.inc(3800): include('/usr/local/pkg/...')
#3 /usr/local/pkg/suricata/suricata.inc(933): suricata_generate_yaml(Array)
#4 /etc/inc/pkg-utils.inc(715) : eval()'d code(1): sync_suricata_package_config()
#5 /etc/inc/pkg-utils.inc(715): eval()
#6 /etc/rc.start_packages(66): sync_package('suricata')
#7 {main}
thrown @ 2023-02-04 14:09:51