23.01RC - Suricata stops working after Wireguard installed

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks Ok, not sure this was a conclusive test, as I ran this on a VM where I was using vtnet instead of igb. Had to mess around with interface settings a bit to get it up and running.

So php -f /tmp/test.php gave this:

Array
(
[network] => 10.6.250.0
[mask] => 31
[tun] => tun_wg1
[descr] => One_VPN
)

$vpns_arr[] contents =
Array
(
[0] => Array
(
[network] => 10.6.250.0
[mask] => 31
[tun] => tun_wg1
[descr] => One_VPN
)

)

That only shows one of the tunnels though... Also, after fixing the interface assignments, I also changed the names of the WG interfaces, but that was not reflected in the output.
The output looks exactly the same on 22.05 btw...

Editing suricata.inc to look like this:

/* WireGuard */
if (function_exists('wg_get_tunnel_networks')) {
foreach (wg_get_tunnel_networks() as $wgn) {
$vpns_arr[] = $wgn['network'] . '/' . $wgn['mask'];
}

Gave the following error (twice, because of two interfaces?):

PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata.inc, Line: 4310, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/suricata/suricata.inc:4310
Stack trace:
#0 /usr/local/pkg/suricata/suricata.inc(599): suricata_get_vpns_list()
#1 /usr/local/pkg/suricata/suricata_generate_yaml.php(46): suricata_build_list(Array, 'default')
#2 /usr/local/pkg/suricata/suricata.inc(3800): include('/usr/local/pkg/...')
#3 /usr/local/pkg/suricata/suricata.inc(933): suricata_generate_yaml(Array)
#4 /tmp/suricata_vtnet128603_startcmd.php(8): sync_suricata_package_config()
#5 {main}

Thanks for testing. I think the actual error is somewhere else, but this code exposes it. The string offset error error you get when you modify the suricata.inc file like I suggested is a PHP 8.1 thing. That error will not usually happen in earlier PHP versions, so that's why the same setup works for in 22.05. Earlier pfSense versions use PHP 7.4.

I will keep digging into this.

bmeeks

After some more digging around, I think this problem may reside in the custom blocking plugin compiled into Suricata for use on pfSense. The error message "<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL" is coming from the Radix Tree code in the Suricata binary. There was a change in the Radix Tree itself, and some changes in the custom blocking plugin between Suricata 6.0.4 which is the current binary in pfSense Plus 22.05 and Suricata 6.0.8 which is the current binary in pfSense Plus 23.01. The problem may be related to the subnet mask.

Patch

@bob-dig said in 23.01RC - Suricata stops working after Wireguard installed:

enforce a different behavior ... to upgrade just the app and load "definitions" later.

I suspect that is basically how updates are done already.

The difference you are suggesting is for the update to clearly separate application update from configuration update. Which should allow relatively graceful termination of the latter if if fails, and a warning "User configuration update did not complete, manual re-cofiguration maybe required".

Gblenn

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

After some more digging around, I think this problem may reside in the custom blocking plugin compiled into Suricata for use on pfSense. The error message "<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL" is coming from the Radix Tree code in the Suricata binary. There was a change in the Radix Tree itself, and some changes in the custom blocking plugin between Suricata 6.0.4 which is the current binary in pfSense Plus 22.05 and Suricata 6.0.8 which is the current binary in pfSense Plus 23.01. The problem may be related to the subnet mask.

I decided to do some more testing and discovered some wierd issues with the passlist. First of all, I get a discrepancy between what Suricata reports to be doing, like adding an IP to the passlist, and what it then shows when "Loading and parsing passlist."

Going to Interfaces > myVPN (tun_wg0) and changing the IP to e.g. 192.n.n.n or 172.n.n.n the log shows the following:
-> adding firewall interface tun_wg0 IPv4 address 172.6.210.0 to automatic interface IP Pass List.
-> adding firewall interface tun_wg1 IPv4 address 192.168.250.0 to automatic interface IP Pass List.
output device (regular) initialized: block.log
-> Loading and parsing Pass List from: /usr/local/etc/suricata/suricata_28603_vtnet1/passlist.
-> Added IPv4 address 8.8.8.8/32 from assigned Pass List.
This is where it stopped earlier
-> Added IPv4 address 10.6.210.0/31 from assigned Pass List.
-> Added IPv4 address 10.6.250.0/31 from assigned Pass List.
-> Added IPv4 address 127.0.0.1/32 from assigned Pass List.

Suricata now works it's way through and actually starts. However it does throw the same PHP error as before.

The actual list found in /usr/local/etc/suricata/suricata_28603_vtnet1 never changes, no matter what changes I make in Interfaces or inside Wireguard.

Manually removing it and restarting Suricata results in a crash, it does not seem to be created or changed? Rebooting makes no difference...

Manually editing the list and setting the same IP's as the interface has, AND using mask /31 in the passlist, will result in Suricata not starting with the same fault reported as before during parsing (this is basically the original setting from a working environment):
<Error> -- [ERRCODE: SC_ERR_FATAL(171)] - prefix or user NULL

Using mask /32 in the passlist allows it to continue through, regardless of the interface having the mask /31.

Making a change in the Interface and clicking Save Changes gives the following in suricata.log
-> Received notification of IP address change on firewall interface tun_wg1.
-> deleted address 10.6.250.0 from automatic firewall interface IP Pass List.
-> Received notification of IP address change on firewall interface tun_wg1.
-> added address 10.6.250.0 to automatic firewall interface IP Pass List.

And then the error shows up:
PHP ERROR: Type: 1, File: /usr/local/pkg/suricata/suricata.inc, Line: 4310, Message: Uncaught TypeError: Cannot access offset of type string on string in /usr/local/pkg/suricata/suricata.inc:4310
Stack trace:
#0 /usr/local/pkg/suricata/suricata.inc(599): suricata_get_vpns_list()
#1 /usr/local/pkg/suricata/suricata_generate_yaml.php(46): suricata_build_list(Array, 'default')
#2 /usr/local/pkg/suricata/suricata.inc(3800): include('/usr/local/pkg/...')
#3 /usr/local/pkg/suricata/suricata.inc(933): suricata_generate_yaml(Array)
#4 /etc/inc/pkg-utils.inc(715) : eval()'d code(1): sync_suricata_package_config()
#5 /etc/inc/pkg-utils.inc(715): eval()
#6 /etc/rc.start_packages(66): sync_package('suricata')
#7 {main}
thrown @ 2023-02-04 14:09:51

bmeeks

@gblenn:

First off, thank you for the detailed troubleshooting steps. You have confirmed what I suspected and was about to test myself today. I'll give the details on what I suspect is wrong farther down below.

You can remove that edit I suggested to the suricata.inc file. That turned out to be a red herring. While that area of code needs a little work, it is not the cause of Suricata failing to start when your Wireguard tunnels are present.

The problem is the choice of a /31 subnet mask. I know that is now legal per a revision to the RFCs, but not all software is fully compatible yet. I think the Radix Tree subroutine in Suricata has an issue with the /31 mask. I will be looking into that to see if I can come up with a fix.

The Suricata binary does a lot of work with IP addresses in rules, so it needs a fast method for storing lots of IP addresses and subnets and then searching back through them to locate a particular entry. So, the developers included a Radix Tree API. The Radix Tree lets you insert IP addresses or subnet blocks into the tree, and then later search for them when trying to match up an IP or subnet against a previously stored value. In the case of the Pass List function in my custom blocking module, the Pass List IP entries are stored in the Radix Tree. Then, during operation when an alert fires, and the custom plugin needs to know whether to block that IP or not, it looks for it (or a subnet containing that IP) in the Radix Tree. If found, that means the IP is covered by a Pass List entry. If not found, then it is blocked.

There are two Pass Lists present on an interface. One is automatic and hidden. That hidden one contains all of the firewall's interface IP addresses (expressed with a /32 or /128 mask). The custom blocking plugin creates a subscription to receive notifications from the FreeBSD kernel routing table when it changes. That's how it maintains the hidden pass list of firewall interface IPs. The hidden pass list was created to cope with interfaces which have dynamic addresses (such as DHCP on the WAN, for example). Prior to the automatic list being in place, if DHCP changed the WAN IP it was possible for Suricata to block it. The other Pass List is the one you see and can manipulate in the GUI.

As a workaround and additional test, can you try using a /30 mask on your tunnel networks?

Gblenn

@bmeeks I might try /30 or /32 as you suggest, but as of now I'm on 22.05 on my actual pfsense...
In terms of lists that I can see, we have the one I was editing. (SSH inside the folder /usr/local/etc/suricata/suricata_28603_vtnet1). This is the same info that I can see if click View List under Interface settings in Suricata.

If I add a tunnel on my 22.05 version, that new IP shows up in that list. But in my testing, none of the changes I made were reflected in that list, I had to make the changes manually...

mrsunfire

@bmeeks The /30 did the trick! Changing the tunnel network to /30 subnet prior to upgrade and suricata + wireguard is running fine with 23.01-RC.

@cmcdonald

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

If I add a tunnel on my 22.05 version, that new IP shows up in that list. But in my testing, none of the changes I made were reflected in that list, I had to make the changes manually...

I'm a bit confused by this statement. What do you mean you had to make changes manually?

The text file you find in the interface's subdirectory is what the Pass List code within the custom blocking plugin compiled into the binary reads its input from. That text file is recreated by the PHP code each time you start (or restart) Suricata on the interface. The data written to that file is determined by settings stored in the config.xml configuration file for the firewall.

The original problem is the custom blocking plugin is experiencing an issue attempting to insert the tunnel address with a /31 subnet mask into the Radix Tree. That is generating the SC_ERR_FATAL - prefix or user NULL error message. And since Suricata logs that as a fatal error, it does not start.

bmeeks

@mrsunfire said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks The /30 did the trick! Changing the tunnel network to /30 subnet prior to upgrade and suricata + wireguard is running fine with 23.01-RC.

@cmcdonald

Thanks for the confirmation. This is a Radix Tree insertion problem within the Suricata binary and my custom blocking plugin. If a /30 works for you now, just use that and I will work on a fix for the Radix Tree problem. Unfortunately it will require a change in the binary portion of the Suricata package, so not as fast and easy as changing something in the PHP GUI code.

Gblenn

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

If I add a tunnel on my 22.05 version, that new IP shows up in that list. But in my testing, none of the changes I made were reflected in that list, I had to make the changes manually...

I'm a bit confused by this statement. What do you mean you had to make changes manually?

I mean that the file did not change when I made changes in the GUI. However, this must have been some issue with my test environment. Just now I changed over to the 23.01 I had running when I started this thread. And now if I make a change in the GUI, it is in fact reflected in the file as well. Might have something to do with the fact that my test environment was running vtnet instead of igb (I use IOMMU). I need to check this further, at some point, but there was an igb1 folder which also contained a passlist (which I never looked at).

Gblenn

@mrsunfire said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks The /30 did the trick! Changing the tunnel network to /30 subnet prior to upgrade and suricata + wireguard is running fine with 23.01-RC.

@cmcdonald

Thanks for the confirmation. This is a Radix Tree insertion problem within the Suricata binary and my custom blocking plugin. If a /30 works for you now, just use that and I will work on a fix for the Radix Tree problem. Unfortunately it will require a change in the binary portion of the Suricata package, so not as fast and easy as changing something in the PHP GUI code.

How do you change the tunnel network to a /30 subnet? Do you mean that you set this for the Peer(s) in the Wireguard UI?
I switched over to 23.01 now and changing things there have no effect whatsoever. Suricata is monitoring the interface not the peers, and the passlist file doesn't change until you make a change in the interface.
However, changing the Interface VPN (tun_wg0) to a /30 subnet is not allowed (This IPv4 address is the network address and cannot be used). Changing it to /32 is allowed but breaks the tunnel...
Hmmm, I see now that this is because I'm using n.n.n.0 for the gateway (n.n.n.1 at the other site). I guess if I changed that to .1 & .2 I would be able to use /30 at this end... big risk messing with the remote server though...

mrsunfire

@gblenn Well just change the tunnel interface to /30. Remember that you can't use a 0 at the end of an IP address within this subnet.

Gblenn

@mrsunfire said in 23.01RC - Suricata stops working after Wireguard installed:

@gblenn Well just change the tunnel interface to /30. Remember that you can't use a 0 at the end of an IP address within this subnet.

Yes that's what I realized so I need to make changes at the other site as well, which represents a bit of a risk...

mrsunfire

@gblenn I feel that. Had the same. Turns out it was very handy. First change remote alias (if you have one for the tunnel). Then change remote wg interface. Now on your site and you should be good to go. That's what I did.

Gblenn

@mrsunfire Of course it stopped working, since I forgot to change routing on the remote server (as I changed gateway IP at home).... Tailscale to the rescue, I'm back in business...

Next step is to try it on 23.01...

Gblenn

And that worked, I'm up and running with 23.01.r.20230202.0019 with my full config.

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

And that worked, I'm up and running with 23.01.r.20230202.0019 with my full config.

Glad you are back up and running. I will still look into the /31 issue with the Radix Tree, but it may turn out to be something upstream has to modify. The use of /31 subnet masks is still a bit iffy with software apps today. Not everything is fully onboard with that one yet.

Gblenn

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

And that worked, I'm up and running with 23.01.r.20230202.0019 with my full config.

Glad you are back up and running. I will still look into the /31 issue with the Radix Tree, but it may turn out to be something upstream has to modify. The use of /31 subnet masks is still a bit iffy with software apps today. Not everything is fully onboard with that one yet.

Thanks, good that you keep investigating. I have no idea how common it is to use /31 in a site to site config, but I think I was following one of many guides out there...

bmeeks

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

@bmeeks said in 23.01RC - Suricata stops working after Wireguard installed:

@gblenn said in 23.01RC - Suricata stops working after Wireguard installed:

And that worked, I'm up and running with 23.01.r.20230202.0019 with my full config.

Glad you are back up and running. I will still look into the /31 issue with the Radix Tree, but it may turn out to be something upstream has to modify. The use of /31 subnet masks is still a bit iffy with software apps today. Not everything is fully onboard with that one yet.

Thanks, good that you keep investigating. I have no idea how common it is to use /31 in a site to site config, but I think I was following one of many guides out there...

I cannot seem to reproduce your original failure. Suricata is starting for me every time without error when I put in what I think are your old tunnel networks. I want to reliably reproduce the failure so I can know if my fix actually works or not.

Can you share your Wireguard tunnel and peer addresses? And if you can, the other firewall interface IP settings as well so that I can duplicate everything in my test setup. If you don't want to post that information in the open forum, can you share them via a DM (Forum chat message)?

Gblenn

@bmeeks I can provide more detail later on when I'm back home but I think these are the key elements.

The wireguard Interface for the home site has the IP 10.6.210.0 /31 (the peer site has 10.6.210.1 /31)
The allowed IP's for the wireguard peer(s) use 10.6.210.0 /31 at both ends. But I do think it is the Interface that is the actual trigger here. This setting is only an internal thing for wireguard as far as I understand...