Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks Both interfaces were down before the update request. The WAN configuration was duplicated to a duplicated WAN, and the configuration for the LAN was not displayed in the Settings menu. And yes in the settings menu it shows 2x WANs both as naming and the configuration.

I will try with a reboot after a few hours and let you know, I'm using the connection for the moment.

Why were the interfaces down? How did the firewall connect to Netgate servers to pull down the update? Just trying to figure out the scenario for reproducing the problem.

And if it is working okay now, how did you fix it? Or do you mean it's running without Suricata on the interfaces?

P.S. -- you might also try simply removing the Suricata package and then reinstalling it. Now that I think more about it, that may be what the user did that had the problem during the DEVEL snapshot testing.

NRgia

@bmeeks Sorry, I meant I disabled Suricata on both interfaces before starting the update process.

Also i removed Suricata package and reinstalled it.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks Sorry, I meant I disabled Suricata on both interfaces before starting the update process.

Also i removed Suricata package and reinstalled it.

Okay... that makes more sense . I was confused there for a moment. This issue has not affected everyone (at least so far as I know), and I have not seen it during my development snapshot tests. Is pfSense (and FreeBSD itself) reporting the physical NIC interfaces with exactly the same name as before?

bmeeks

@nrgia:
Suricata uses the interface names as the "keys" when storing interface configuration. So if you examine the Suricata section of config.xml you should see sections under the <rules> element for <wan> and <lan> with each section containing all the configuration settings for that interface.

Do you see that in config.xml, or do you see two <wan> sections? You can view the file under DIAGNOSTICS > EDIT FILE, then browse to /conf/ and open config.xml.

I am struggling to come up with a theory that describes what happened. Since it has apparently occurred twice to two different users, but not all users, something unique in a setup would seem to be the trigger.

As an example, here is part of the config.xml file from my test virtual machine showing the <rule> sections for WAN and LAN. Scroll down towards the bottom of the config.xml file and find the <suricata> section like this one:

<suricata>
	<config>

... then continue scrolling down until you find the first <rule> tag like this one:

    <rule>
	<interface>wan</interface>
	<enable>on</enable>
	<uuid>14777</uuid>
	<descr><![CDATA[WAN]]></descr>

and then farther down ...

    <rule>
	<interface>lan</interface>
	<enable>on</enable>
	<uuid>22480</uuid>
	<descr><![CDATA[LAN]]></descr>

I am curious what shows in these sections of your config.xml file. That might help me troubleshoot if you can share it. Just these subparts. Don't post the entire file as some potentially sensitive info is also stored in the file.

greenflash

@nrgia I do have a very similar issue.

On version 6.0.8_7 I have the following interfaces configured:

But after upgrading to 6.0.8_8 or 6.0.10_1 my WAN configuration (including its custom rules) have got deleted and I have duplicate LAN1 configurations, see:

For me there is no obvious way to get my WAN configuration back, besides rolling back to the previous version via boot environments.

EDIT:
I have already unsuccessfully tried the following:

Rebooting the firewall
Removing suricata package and reinstalling version 6.0.10_1
Restoring a backup of the config.xml file

NRgia

@greenflash said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

For me there is no obvious way to get my WAN configuration back, besides rolling back to the previous version via boot environments.

For me it was easier, because before the update, I disabled Suricata on both interfaces, so an entry in the Configuration history was added. I just clicked on revert to that point, and I have both of the interfaces again. Sure if I reinstall the package the same happens, doubled interfaces.

Maybe you can find something in Diagnostics>Backup & RestoreConfig> History

NRgia

@bmeeks pfSense or other menus like Interface assignments reports the correct interface mappings. So only in Suricata I have issues. I will reinstall again, in order to trigger the problem again, and will respond to the rest of the questions shortly.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia:
Suricata uses the interface names as the "keys" when storing interface configuration. So if you examine the Suricata section of config.xml you should see sections under the <rules> element for <wan> and <lan> with each section containing all the configuration settings for that interface.

Do you see that in config.xml, or do you see two <wan> sections? You can view the file under DIAGNOSTICS > EDIT FILE, then browse to /conf/ and open config.xml.

I am struggling to come up with a theory that describes what happened. Since it has apparently occurred twice to two different users, but not all users, something unique in a setup would seem to be the trigger.

As an example, here is part of the config.xml file from my test virtual machine showing the <rule> sections for WAN and LAN. Scroll down towards the bottom of the config.xml file and find the <suricata> section like this one:
<suricata>
	<config>
... then continue scrolling down until you find the first <rule> tag like this one:
    <rule>
	<interface>wan</interface>
	<enable>on</enable>
	<uuid>14777</uuid>
	<descr><![CDATA[WAN]]></descr>
and then farther down ...
    <rule>
	<interface>lan</interface>
	<enable>on</enable>
	<uuid>22480</uuid>
	<descr><![CDATA[LAN]]></descr>
I am curious what shows in these sections of your config.xml file. That might help me troubleshoot if you can share it. Just these subparts. Don't post the entire file as some potentially sensitive info is also stored in the file.

This is what I see in the config.xml :

<rule>
				<interface>wan</interface>
				<enable>off</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

and

<rule>
				<interface>wan</interface>
				<enable>off</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

Also the configuration is 100 % for both interfaces, rules, etc.

If I enable Suricata on both WANs from the GUI only one instance of Suricata will start:

Suricata 2 interfaces.png

Also this is from the config.xml previous to updating:

<rule>
				<interface>wan</interface>
				<enable>on</enable>
				<uuid>27404</uuid>
				<descr><![CDATA[WAN]]></descr>

and


<rule>
				<interface>lan</interface>
				<enable>on</enable>
				<uuid>42440</uuid>
				<descr><![CDATA[LAN]]></descr>

I have both configs saved now, if you need more sections of them, I'm glad to help.

bmeeks

@NRgia:
Yeah, that first capture from your config.xml is definitely not correct. The entire interface appears duplicated. Notice the UUID also is the same.

Not sure what is happening there. I will need to do some experimentation and hack up a config on my test VM to see if I can replicate your issue.

Your second screenshot from the previous configuration looks correct, so the revert to previous configuration works.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@NRgia:
Yeah, that first capture from your config.xml is definitely not correct. The entire interface appears duplicated. Notice the UUID also is the same.

Not sure what is happening there. I will need to do some experimentation and hack up a config on my test VM to see if I can replicate your issue.

Your second screenshot from the previous configuration looks correct, so the revert to previous configuration works.

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

No problem with the thread topic: this is as good a place as any to figure it out. Thank you for letting me know.

Your guess matches mine that something happens in the "migrate existing configuration" part of the install steps. Give me a bit to run some scenarios around in my head, and then test them. There were many PHP changes required in the package in order to become compatible with the new PHP 8.1 included in 23.01 pfSense Plus and the upcoming 2.7 CE. One of those changes could have easily broken something my testing failed to uncover.

You have a workaround by reverting to the older config section, so that gets you running. Anything I find will have to go into a new package update.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

Correct, if I revert to a previous config the issue is not present. It's doing something in the conversion step I think, during the update/install. Sorry to bring this up, in this thread.

No problem with the thread topic: this is as good a place as any to figure it out. Thank you for letting me know.

Your guess matches mine that something happens in the "migrate existing configuration" part of the install steps. Give me a bit to run some scenarios around in my head, and then test them. There were many PHP changes required in the package in order to become compatible with the new PHP 8.1 included in 23.01 pfSense Plus and the upcoming 2.7 CE.

You have a workaround by reverting to the older config section, so that gets you running. Anything I find will have to go into a new package update.

Sure Bill take your time, it's not a blocker for me.

returntrip

I have a similar issue. My WAN interfaces is duplicated, (the LAN interface disappeared) and are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

Would it be sufficient for me to change one of the interfaces back to LAN to fix this or should I be wary of something?

bmeeks

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

I have a similar issue. My WAN interfaces is duplicated, (the LAN interface disappeared) and are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

Would it be sufficient for me to change one of the interfaces back to LAN to fix this or should I be wary of something?

Thanks for the report. It's apparent something is wrong in the config migration code that runs when an updated package is installed. I never saw this in my virtual machine testing, but it could simply be because I never hit the right conditions. I have a theory on what may be happening, but I would like to reproduce it so that I can make sure my theory is correct. Will work on that today. I am pretty sure it's a PHP 8.1 thing. The move to 8.1 brought a lot of required changes in existing PHP code with it.

Anything I find will have to go into the next package update, and if my theory is correct, those of you impacted will need to either selectively restore/edit your config.xml from prior to the Suricata upgrade, or else recreate the missing interface from scratch. I suspect it has been permanently overwritten in your current config.xml.

SteveITS

@returntrip said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

are showing as stopped in the Suricata overview page. However, I can see the alerts tab being populated for both WAN interfaces.

You probably have a zombie process. Killing all the Suricata processes, or rebooting, should clear it.

Why have it on WAN and LAN? On WAN it will scan all packets before hitting the firewall so will waste time processing packets that will drop.

NRgia

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach. I have my config created a few versions back.

bmeeks

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach.

That's sort of what I'm doing, but using just the config.xml file and importing/restoring an older one. I don't actually have any pfSense Plus testing VMs. I'm using 2.7.0 for that now as 23.01 and 2.7.0 are pretty much the same at the moment.

I do so much destroying and recreating the test VMs that having to upgrade each one to Plus each time would be a hassle.

NRgia

@bmeeks said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@nrgia said in Suricata 6.0.10_1 Update for pfSense Plus 23.01 - Release Notes:

@bmeeks You can try another scenario. Install pfSense CE 2.6.0, install Suricata, and then upgrade until you reach 23.01. You have more config migrations with this approach.

That's sort of what I'm doing, but using just the config.xml file and importing/restoring an older one. I don't actually have any pfSense Plus testing VMs. I'm using 2.7.0 for that now as 23.01 and 2.7.0 are pretty much the same at the moment.

I do so much destroying and recreating the test VMs that having to upgrade each one to Plus each time would be a hassle.

I feel you, we, the users with whiteboxes tend to test less due to this cumbersome upgrade path, instead of simple installer images.

returntrip

@steveits If am reading your reply correctly, the best practice is to run suricata only on the LAN interface then?

SteveITS

@returntrip Yep. No need to scan packets twice, no need to scan packets that will be dropped by the firewall, and on LAN the alert will show the LAN IP not the NATted IP of the router WAN.

I suppose there's an argument to use WAN if one has lots of internal interfaces and limited memory...?