Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes

greenflash

@nrgia I do have a very similar issue.

On version 6.0.8_7 I have the following interfaces configured:

But after upgrading to 6.0.8_8 or 6.0.10_1 my WAN configuration (including its custom rules) have got deleted and I have duplicate LAN1 configurations, see:

For me there is no obvious way to get my WAN configuration back, besides rolling back to the previous version via boot environments.

EDIT:
I have already unsuccessfully tried the following:

Rebooting the firewall
Removing suricata package and reinstalling version 6.0.10_1
Restoring a backup of the config.xml file

NRgia

@greenflash said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

For me there is no obvious way to get my WAN configuration back, besides rolling back to the previous version via boot environments.

For me it was easier, because before the update, I disabled Suricata on both interfaces, so an entry in the Configuration history was added. I just clicked on revert to that point, and I have both of the interfaces again. Sure if I reinstall the package the same happens, doubled interfaces.

Maybe you can find something in Diagnostics>Backup & RestoreConfig> History

NRgia

@bmeeks pfSense or other menus like Interface assignments reports the correct interface mappings. So only in Suricata I have issues. I will reinstall again, in order to trigger the problem again, and will respond to the rest of the questions shortly.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia:
Suricata uses the interface names as the "keys" when storing interface configuration. So if you examine the Suricata section of config.xml you should see sections under the <rules> element for <wan> and <lan> with each section containing all the configuration settings for that interface.

Do you see that in config.xml, or do you see two <wan> sections? You can view the file under DIAGNOSTICS > EDIT FILE, then browse to /conf/ and open config.xml.

I am struggling to come up with a theory that describes what happened. Since it has apparently occurred twice to two different users, but not all users, something unique in a setup would seem to be the trigger.

As an example, here is part of the config.xml file from my test virtual machine showing the <rule> sections for WAN and LAN. Scroll down towards the bottom of the config.xml file and find the <suricata> section like this one:
<suricata>
	<config>
... then continue scrolling down until you find the first <rule> tag like this one:
    <rule>
	<interface>wan</interface>
	<enable>on</enable>
	<uuid>14777</uuid>
	<descr><![CDATA[WAN]]></descr>
and then farther down ...
    <rule>
	<interface>lan</interface>
	<enable>on</enable>
	<uuid>22480</uuid>
	<descr><![CDATA[LAN]]></descr>
I am curious what shows in these sections of your config.xml file. That might help me troubleshoot if you can share it. Just these subparts. Don't post the entire file as some potentially sensitive info is also stored in the file.

This is what I see in the config.xml :

<rule>
				<interface>wan</interface>
				<enable>off</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

and

<rule>
				<interface>wan</interface>
				<enable>off</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

Also the configuration is 100 % for both interfaces, rules, etc.

If I enable Suricata on both WANs from the GUI only one instance of Suricata will start:

Suricata 2 interfaces.png

Also this is from the config.xml previous to updating:

<rule>
				<interface>wan</interface>
				<enable>on</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

and


<rule>
				<interface>lan</interface>
				<enable>on</enable>
				<uuid>42440</uuid>
				<descr><![CDATA[LAN]]></descr>

I have both configs saved now, if you need more sections of them, I'm glad to help.

bmeeks

@NRgia:
Yeah, that first capture from your config.xml is definitely not correct. The entire interface appears duplicated. Notice the UUID also is the same.

Not sure what is happening there. I will need to do some experimentation and hack up a config on my test VM to see if I can replicate your issue.

Your second screenshot from the previous configuration looks correct, so the revert to previous configuration works.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@NRgia:
Yeah, that first capture from your config.xml is definitely not correct. The entire interface appears duplicated. Notice the UUID also is the same.

Not sure what is happening there. I will need to do some experimentation and hack up a config on my test VM to see if I can replicate your issue.

Your second screenshot from the previous configuration looks correct, so the revert to previous configuration works.

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

No problem with the thread topic: this is as good a place as any to figure it out. Thank you for letting me know.

Your guess matches mine that something happens in the "migrate existing configuration" part of the install steps. Give me a bit to run some scenarios around in my head, and then test them. There were many PHP changes required in the package in order to become compatible with the new PHP 8.1 included in 23.01 pfSense Plus and the upcoming 2.7 CE. One of those changes could have easily broken something my testing failed to uncover.

You have a workaround by reverting to the older config section, so that gets you running. Anything I find will have to go into a new package update.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

No problem with the thread topic: this is as good a place as any to figure it out. Thank you for letting me know.

Your guess matches mine that something happens in the "migrate existing configuration" part of the install steps. Give me a bit to run some scenarios around in my head, and then test them. There were many PHP changes required in the package in order to become compatible with the new PHP 8.1 included in 23.01 pfSense Plus and the upcoming 2.7 CE.

You have a workaround by reverting to the older config section, so that gets you running. Anything I find will have to go into a new package update.

Sure Bill take your time, it's not a blocker for me.

returntrip

I have a similar issue. My WAN interfaces is duplicated, (the LAN interface disappeared) and are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

Would it be sufficient for me to change one of the interfaces back to LAN to fix this or should I be wary of something?

bmeeks

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

I have a similar issue. My WAN interfaces is duplicated, (the LAN interface disappeared) and are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

Would it be sufficient for me to change one of the interfaces back to LAN to fix this or should I be wary of something?

Thanks for the report. It's apparent something is wrong in the config migration code that runs when an updated package is installed. I never saw this in my virtual machine testing, but it could simply be because I never hit the right conditions. I have a theory on what may be happening, but I would like to reproduce it so that I can make sure my theory is correct. Will work on that today. I am pretty sure it's a PHP 8.1 thing. The move to 8.1 brought a lot of required changes in existing PHP code with it.

Anything I find will have to go into the next package update, and if my theory is correct, those of you impacted will need to either selectively restore/edit your config.xml from prior to the Suricata upgrade, or else recreate the missing interface from scratch. I suspect it has been permanently overwritten in your current config.xml.

SteveITS

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

You probably have a zombie process. Killing all the Suricata processes, or rebooting, should clear it.

Why have it on WAN and LAN? On WAN it will scan all packets before hitting the firewall so will waste time processing packets that will drop.

NRgia

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach. I have my config created a few versions back.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach.

That's sort of what I'm doing, but using just the config.xml file and importing/restoring an older one. I don't actually have any pfSense Plus testing VMs. I'm using 2.7.0 for that now as 23.01 and 2.7.0 are pretty much the same at the moment.

I do so much destroying and recreating the test VMs that having to upgrade each one to Plus each time would be a hassle.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach.

That's sort of what I'm doing, but using just the config.xml file and importing/restoring an older one. I don't actually have any pfSense Plus testing VMs. I'm using 2.7.0 for that now as 23.01 and 2.7.0 are pretty much the same at the moment.

I do so much destroying and recreating the test VMs that having to upgrade each one to Plus each time would be a hassle.

I feel you, we, the users with whiteboxes tend to test less due to this cumbersome upgrade path, instead of simple installer images.

returntrip

@steveits If am reading your reply correctly, the best practice is to run suricata only on the LAN interface then?

SteveITS

@returntrip Yep. No need to scan packets twice, no need to scan packets that will be dropped by the firewall, and on LAN the alert will show the LAN IP not the NATted IP of the router WAN.

I suppose there's an argument to use WAN if one has lots of internal interfaces and limited memory...?

bmeeks

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@steveits If am reading your reply correctly, the best practice is to run suricata only on the LAN interface then?

Suricata sits in front of the firewall no matter which interface it is configured on. Packets inbound on an interface (meaning coming from the NIC on the their way into pfSense) encounter Suricata first. That means when you run Suricata on the WAN it is going to see and analyze all the Internet noise that exists there. And most, if not all, of that noise traffic is not allowed through the firewall, so why waste CPU resources and RAM having Suricata scan that Internet noise the firewall rules are probably going to drop anyway?

Better to run IDS/IPS instances on the internal interfaces such as LAN or DMZ and others. As @SteveITS mentioned, about the only time running Suricata on the WAN would make sense is if you have a large number of internal interfaces, but limited RAM and/or CPU. In that case running a single instance on the WAN might make sense due to the resource constraints.

Besides the NAT issue mentioned, another problem with running it on the WAN is all that Internet noise is going to generate a lot of logging data and eat up local disk space. Again, this is because no firewall rules have been applied at the point Suricata is seeing the interface traffic. Putting the instance on an internal interface lets the firewall do the initial traffic filtering, and then Suricata only looks at traffic the firewall allowed, cutting down on useless alerts and what gets logged.

returntrip

Many thanks @bmeeks your reply is really clear and I now understand the reasoning behind WAN vs LAN only monitoring. I have 4 internal interfaces I want to screen so maybe I could monitor only the WAN interface then but then I would not know what internal IP am alert is generated for. Is there any way to overcome this "drawback" , besides monitoring every single internal interface?

bmeeks

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Many thanks @bmeeks your reply is really clear and I now understand the reasoning behind WAN vs LAN only monitoring. I have 4 internal interfaces I want to screen so maybe I could monitor only the WAN interface then but then I would not know what internal IP am alert is generated for. Is there any way to overcome this "drawback" , besides monitoring every single internal interface?

No, unless you want to run the same rules for all the interfaces and they happen to be VLANs. In that case, you could put Suricata on the physical parent interface and it would see the traffic for all VLANs defined on that parent. But that does require that you want the same rules for all the internal networks (or at least all the VLANs defined on that parent).

But also consider carefully tuning the rules you have enabled on interfaces. Customize the rules for the vulnerabilities on that network. Most folks tend to enable way more rules than they actually require. The less rules Suricata has to work through, the smaller the drain on CPU resources and RAM.

returntrip

@bmeeks very nteresting. Am running pfsense as a router on a stick with only 1 physical interface which provides 1 vlan for wan and 4 vlans for internal..... so I guess it would be much easier for me to monitor the parent interface for everything. I do not think I would need separate rules per IF.