Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes

bmeeks

An update for the Suricata package has been posted for the pfSense Plus 23.01 RELEASE branch. This latest update contains two important fixes. One improves the performance of the Pass List logic, and the other change improves Inline IPS Mode performance with the netmap device.

Pass List Improvements:
For quite some time there have been two different Pass Lists used by Suricata when operating in Legacy Blocking Mode. This mode uses a custom output plugin compiled into the binary and used exclusively on pfSense. Pass Lists are text files containing IP addresses and/or networks that should never be blocked by the plugin. One of the Pass Lists used by the plugin is the one configured by the user. This can be either a default list generated by the system or a custom list provided by the user and created on the PASS LIST tab. The other Pass List is actually hidden from view and is generated automatically by the custom output plugin at runtime. This internal list is populated with all the interface IP addresses (using a /32 mask) present on the firewall. These interface addresses are obtained by calling into the operating system and iterating the configured interfaces. Additionally, once Suricata starts, an independent internal thread is launched which subscribes to a kernel routing socket and listens continuously for updates to any interface IP advertised by the OS kernel. In this manner the internal hidden interface IP pass list is automatically updated as firewall interfaces change. You can see this interface IP address monitoring thread log its actions in the suricata.log for the Suricata interface (viewable under the LOGS VIEW tab by selecting an interface and then the suricata.log entry).

In the past, this internal interface monitoring thread and the user-supplied Pass List were both adding firewall interface IPs to the Radix Tree structure that stores, and then looks up later, IP addresses and networks. This duplicating of entries was a potential trouble spot in the Radix Tree. Additionally, when adding user-supplied IP addresses and networks, no test was done to see if the entry being added duplicated an existing entry (or was covered by an existing entry). This behavior risked causing issues in the Radix Tree when later looking up an IP address to see if it was covered by the Pass List. So, the Pass List logic in the custom output plugin has been altered a bit to address these potential trouble spots. As a result of the changes, you will notice that firewall interface IP addresses (those /32 values) are no longer pulled into the user-supplied pass list. Instead, those are pulled into the internal list by the interface IP monitoring thread. Another visible change shows up in the suricata.log. Now, when parsing a user-supplied Pass List, if an entry read from the list either already exists in the Radix Tree or lies within a previously added subnet in the Radix Tree, that entry is not added since it would represent a duplicate entry. A summary of the Pass List processing is printed in the suricata.log if you want to check out the results. Remember that a Pass List is only used with Legacy Blocking Mode. Inline IPS Mode does not need, and thus does not generate nor use, a Pass List.

Inline IPS Mode Improvements:
Inline IPS Mode uses the netmap kernel device to implement the ability to selectively drop packets. This mode uses netmap to establish a sort of pipe between the physical NIC driver and the kernel's network stack (also called the host stack). Suricata's analysis engine sits in the middle of that pipe. It reads packets from one end of the pipe, analyzes them, and then either forwards them on to the other end of the pipe or drops them if the packet triggered a DROP rule. Dropped packets are considered as "blocked". In IPS mode, two netmap pipes are created. One receives from the NIC and transmits to the host stack, and the other receives from the host stack and transmits to the NIC. The host stack end of a netmap connection is denoted by appending a caret ( '^' ) to the interface name. So a typical interface pair would be em0 and emo^. The two netmap connections created for IPS mode would look like this assuming em0 was the chosen interface:

em0 --> em0^
em0 <-- em0^

Suricata offers three threading run modes, but only two are useful for production use. The single thread mode is really only for development and testing purposes. The two modes used in production operation are autofp and workers. The default mode on pfSense has been autofp since Suricata was first added as a package. However, some time back the ability to choose either of the three available run modes was added to the INTERFACE SETTINGS tab in the Performance and Detection Engine Settings section. See the example below.

Details on the available run modes and their corresponding threading models can be found here: https://suricata.readthedocs.io/en/suricata-6.0.0/performance/runmodes.html. For Inline IPS Mode operation, the workers run mode is likely to offer the highest throughput performance. But feel free to experiment to see which offers better performance with your particular setup. The efficiency gain of workers really shines when your hardware or virtual NIC exposes multiple TX/RX queues. But if your NIC only exposes a single pair of TX/RX queues, then it may be a toss-up which mode offers the best throughput performance. You can examine the output in the dmesg.boot log in /var/log on the firewall to see the configuration seen and logged by netmap. Here is an example from dmesg.boot pulled off one of my pfSense virtual machines using the vmx virtual NIC:

vmx0: netmap queues/slots: TX 4/512, RX 4/512

This indicates the adapter is exposing four TX and four RX queues with 512 slots in each queue. The slots are data buffers on the netmap ring. Queues map to netmap rings, so for this vmx virtual adaptor netmap will create four TX rings and four RX rings (as four TX/RX ring pairs) with each ring containing 512 slots to hold the packet data.

But from the same firewall using a different interface with the em virtual NIC driver, notice only a single queue pair is exposed resulting in netmap creating only a single TX/RX ring pair (with the rings having 1024 slots for data):

em1: netmap queues/slots: TX 1/1024, RX 1/1024

Some high performance hardware NICs may have eight or even more TX/RX queues. Higher queue counts translate into greater potential throughput. Suricata, when using netmap IPS mode with run mode workers, will launch a distinct processing thread for each exposed NIC queue. This maximizes performance. So, if you are using Inline IPS Mode with Suricata, experiment with changing the Run Mode setting on the INTERFACE SETTINGS tab to see if you gain throughput increases. Remember to save after making a change to the Run Mode, and then restart Suricata on the interface so it will recognize the change. Run Mode changes are NOT dynamic! The run mode is set at daemon startup only.

Potential Startup Failure:
If, after updating to the latest 6.0.10 version of the Suricata binary you find that Suricata fails to start, examine the suricata.log for the interface looking for any lines similar to this:

21/2/2023 -- 20:14:32 - <Error> -- [ERRCODE: SC_ERR_MEM_ALLOC(1)] - failed to setup/expand stream session pool. Expand stream.memcap?

This error indicates you need to expand the TCP Stream Memcap value on the FLOW/STREAM tab for the interface. The default is 128 MB, but that may not be enough when using high thread counts with multiple CPU cores. Try 256 MB, then keep increasing in 32 or 64 MB chunks until Suricata starts successfully. If your firewall is really tight on free RAM, you can try increasing the value from 128 MB in 32 MB chunks until you get to the minimum amount required.

NRgia

One issue I found yesterday.

Given the user updates the Suricata package from 6.0.8 to 6.0.10_1.
And the configuration is migrated to the new version for 2 interfaces WAN and LAN.
Then the user is presented with 2 WAN interfaces.

Also the user cannot change one of the interfaces back to LAN.

Hope this helps.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Also the user cannot change one of the interfaces back to LAN.

Is the instance actually running on the WAN, or did the text in the Description field on the INTERFACE SETTINGS tab just change? If the latter, you should be table to type LAN in the textbox and save the update.

I never saw this behavior in my testing, but I seem to recall one user reporting it during the 2.7 DEVEL snapshot testing. I think his fixed itself with a reboot of the firewall.

NRgia

@bmeeks Both interfaces were down before the update request. The WAN configuration was duplicated to a duplicated WAN, and the configuration for the LAN was not displayed in the Settings menu. And yes in the settings menu it shows 2x WANs both as naming and the configuration.

I will try with a reboot after a few hours and let you know, I'm using the connection for the moment.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks Both interfaces were down before the update request. The WAN configuration was duplicated to a duplicated WAN, and the configuration for the LAN was not displayed in the Settings menu. And yes in the settings menu it shows 2x WANs both as naming and the configuration.

I will try with a reboot after a few hours and let you know, I'm using the connection for the moment.

Why were the interfaces down? How did the firewall connect to Netgate servers to pull down the update? Just trying to figure out the scenario for reproducing the problem.

And if it is working okay now, how did you fix it? Or do you mean it's running without Suricata on the interfaces?

P.S. -- you might also try simply removing the Suricata package and then reinstalling it. Now that I think more about it, that may be what the user did that had the problem during the DEVEL snapshot testing.

NRgia

@bmeeks Sorry, I meant I disabled Suricata on both interfaces before starting the update process.

Also i removed Suricata package and reinstalled it.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks Sorry, I meant I disabled Suricata on both interfaces before starting the update process.

Also i removed Suricata package and reinstalled it.

Okay... that makes more sense . I was confused there for a moment. This issue has not affected everyone (at least so far as I know), and I have not seen it during my development snapshot tests. Is pfSense (and FreeBSD itself) reporting the physical NIC interfaces with exactly the same name as before?

bmeeks

@nrgia:
Suricata uses the interface names as the "keys" when storing interface configuration. So if you examine the Suricata section of config.xml you should see sections under the <rules> element for <wan> and <lan> with each section containing all the configuration settings for that interface.

Do you see that in config.xml, or do you see two <wan> sections? You can view the file under DIAGNOSTICS > EDIT FILE, then browse to /conf/ and open config.xml.

I am struggling to come up with a theory that describes what happened. Since it has apparently occurred twice to two different users, but not all users, something unique in a setup would seem to be the trigger.

As an example, here is part of the config.xml file from my test virtual machine showing the <rule> sections for WAN and LAN. Scroll down towards the bottom of the config.xml file and find the <suricata> section like this one:

<suricata>
	<config>

... then continue scrolling down until you find the first <rule> tag like this one:

    <rule>
	<interface>wan</interface>
	<enable>on</enable>
	<uuid>14777</uuid>
	<descr><![CDATA[WAN]]></descr>

and then farther down ...

    <rule>
	<interface>lan</interface>
	<enable>on</enable>
	<uuid>22480</uuid>
	<descr><![CDATA[LAN]]></descr>

I am curious what shows in these sections of your config.xml file. That might help me troubleshoot if you can share it. Just these subparts. Don't post the entire file as some potentially sensitive info is also stored in the file.

greenflash

@nrgia I do have a very similar issue.

On version 6.0.8_7 I have the following interfaces configured:

But after upgrading to 6.0.8_8 or 6.0.10_1 my WAN configuration (including its custom rules) have got deleted and I have duplicate LAN1 configurations, see:

For me there is no obvious way to get my WAN configuration back, besides rolling back to the previous version via boot environments.

EDIT:
I have already unsuccessfully tried the following:

Rebooting the firewall
Removing suricata package and reinstalling version 6.0.10_1
Restoring a backup of the config.xml file

NRgia

@greenflash said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

For me there is no obvious way to get my WAN configuration back, besides rolling back to the previous version via boot environments.

For me it was easier, because before the update, I disabled Suricata on both interfaces, so an entry in the Configuration history was added. I just clicked on revert to that point, and I have both of the interfaces again. Sure if I reinstall the package the same happens, doubled interfaces.

Maybe you can find something in Diagnostics>Backup & RestoreConfig> History

NRgia

@bmeeks pfSense or other menus like Interface assignments reports the correct interface mappings. So only in Suricata I have issues. I will reinstall again, in order to trigger the problem again, and will respond to the rest of the questions shortly.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia:
Suricata uses the interface names as the "keys" when storing interface configuration. So if you examine the Suricata section of config.xml you should see sections under the <rules> element for <wan> and <lan> with each section containing all the configuration settings for that interface.

Do you see that in config.xml, or do you see two <wan> sections? You can view the file under DIAGNOSTICS > EDIT FILE, then browse to /conf/ and open config.xml.

I am struggling to come up with a theory that describes what happened. Since it has apparently occurred twice to two different users, but not all users, something unique in a setup would seem to be the trigger.

As an example, here is part of the config.xml file from my test virtual machine showing the <rule> sections for WAN and LAN. Scroll down towards the bottom of the config.xml file and find the <suricata> section like this one:
<suricata>
	<config>
... then continue scrolling down until you find the first <rule> tag like this one:
    <rule>
	<interface>wan</interface>
	<enable>on</enable>
	<uuid>14777</uuid>
	<descr><![CDATA[WAN]]></descr>
and then farther down ...
    <rule>
	<interface>lan</interface>
	<enable>on</enable>
	<uuid>22480</uuid>
	<descr><![CDATA[LAN]]></descr>
I am curious what shows in these sections of your config.xml file. That might help me troubleshoot if you can share it. Just these subparts. Don't post the entire file as some potentially sensitive info is also stored in the file.

This is what I see in the config.xml :

<rule>
				<interface>wan</interface>
				<enable>off</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

and

<rule>
				<interface>wan</interface>
				<enable>off</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

Also the configuration is 100 % for both interfaces, rules, etc.

If I enable Suricata on both WANs from the GUI only one instance of Suricata will start:

Suricata 2 interfaces.png

Also this is from the config.xml previous to updating:

<rule>
				<interface>wan</interface>
				<enable>on</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

and


<rule>
				<interface>lan</interface>
				<enable>on</enable>
				<uuid>42440</uuid>
				<descr><![CDATA[LAN]]></descr>

I have both configs saved now, if you need more sections of them, I'm glad to help.

bmeeks

@NRgia:
Yeah, that first capture from your config.xml is definitely not correct. The entire interface appears duplicated. Notice the UUID also is the same.

Not sure what is happening there. I will need to do some experimentation and hack up a config on my test VM to see if I can replicate your issue.

Your second screenshot from the previous configuration looks correct, so the revert to previous configuration works.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@NRgia:
Yeah, that first capture from your config.xml is definitely not correct. The entire interface appears duplicated. Notice the UUID also is the same.

Not sure what is happening there. I will need to do some experimentation and hack up a config on my test VM to see if I can replicate your issue.

Your second screenshot from the previous configuration looks correct, so the revert to previous configuration works.

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

No problem with the thread topic: this is as good a place as any to figure it out. Thank you for letting me know.

Your guess matches mine that something happens in the "migrate existing configuration" part of the install steps. Give me a bit to run some scenarios around in my head, and then test them. There were many PHP changes required in the package in order to become compatible with the new PHP 8.1 included in 23.01 pfSense Plus and the upcoming 2.7 CE. One of those changes could have easily broken something my testing failed to uncover.

You have a workaround by reverting to the older config section, so that gets you running. Anything I find will have to go into a new package update.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

No problem with the thread topic: this is as good a place as any to figure it out. Thank you for letting me know.

Your guess matches mine that something happens in the "migrate existing configuration" part of the install steps. Give me a bit to run some scenarios around in my head, and then test them. There were many PHP changes required in the package in order to become compatible with the new PHP 8.1 included in 23.01 pfSense Plus and the upcoming 2.7 CE.

You have a workaround by reverting to the older config section, so that gets you running. Anything I find will have to go into a new package update.

Sure Bill take your time, it's not a blocker for me.

returntrip

I have a similar issue. My WAN interfaces is duplicated, (the LAN interface disappeared) and are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

Would it be sufficient for me to change one of the interfaces back to LAN to fix this or should I be wary of something?

bmeeks

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

I have a similar issue. My WAN interfaces is duplicated, (the LAN interface disappeared) and are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

Would it be sufficient for me to change one of the interfaces back to LAN to fix this or should I be wary of something?

Thanks for the report. It's apparent something is wrong in the config migration code that runs when an updated package is installed. I never saw this in my virtual machine testing, but it could simply be because I never hit the right conditions. I have a theory on what may be happening, but I would like to reproduce it so that I can make sure my theory is correct. Will work on that today. I am pretty sure it's a PHP 8.1 thing. The move to 8.1 brought a lot of required changes in existing PHP code with it.

Anything I find will have to go into the next package update, and if my theory is correct, those of you impacted will need to either selectively restore/edit your config.xml from prior to the Suricata upgrade, or else recreate the missing interface from scratch. I suspect it has been permanently overwritten in your current config.xml.

SteveITS

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

You probably have a zombie process. Killing all the Suricata processes, or rebooting, should clear it.

Why have it on WAN and LAN? On WAN it will scan all packets before hitting the firewall so will waste time processing packets that will drop.

NRgia

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach. I have my config created a few versions back.