Suricata fails to start after pfSense reboot

bmeeks

What kind of hardware are you running pfSense on? Is it an SG-3100 appliance or similar non-Intel CPU box?

If so, then look in the pfSense system log and I bet you will find a Signal 10 Bus Error. This is a memory-alignment error fault. This problem cropped up back in the 4.0.5 or 4.0.6 update of the Suricata binary and is caused by the machine opcode decisions the clang/llvm compiler makes when optimization is enabled.

Even if you are not running non-Intel hardware, look in the pfSense system log to see if any error messages are present.

kiekar

@bmeeks said in Suricata fails to start after pfSense reboot:

What kind of hardware are you running pfSense on? Is it an SG-3100 appliance or similar non-Intel CPU box?

It's a home built PC running on a intel H310 chipset board with an 8 gen cpu and Dell Intel PRO/1000 ET Quad Port 1Gb Gigabit Ethernet Adapter H092P HM9JY.

I was unable to find any errors showing Signal 10 Bus Error in system log except for errors for notpng

bmeeks

@kiekar said in Suricata fails to start after pfSense reboot:

@bmeeks said in Suricata fails to start after pfSense reboot:

What kind of hardware are you running pfSense on? Is it an SG-3100 appliance or similar non-Intel CPU box?

It's a home built PC running on a intel H310 chipset board with an 8 gen cpu and Dell Intel PRO/1000 ET Quad Port 1Gb Gigabit Ethernet Adapter H092P HM9JY.

I was unable to find any errors showing Signal 10 Bus Error in system log except for errors for notpng

OK, then the next most likely problem is a syntax issue in the suricata.yaml file. To see what it may be, do this:

Open a command prompt (shell prompt) on the firewall.
Execute these commands:

ls /usr/local/etc/suricata
cd /usr/local/etc/suricata/suricata_xxxx
suricata -T -c ./suricata.yaml

In the above command sequence, the first command will list the contents of the Suricata base directory. Look for a sub-directory in there named suricata_xxxxx where the xxxxx will be a GUID string and the physical interface name. Substitute this directory name into the second command.

For example, suppose you see a sub-directory named suricata_1287_em0 within the /usr/local/etc/suricata directory. Then the second command would look like this:

cd /usr/local/suricata/suricata_1287_em0

This will start a Suricata session to test the suricata.yaml configuration file. A lot of output will scroll by. Look for any syntax errors that appear and post those back here. If no errors appear, post that back as well. If the process starts successfully, then you will need to type CTRL-C to terminate the process and regain your shell prompt.

bmeeks

One more question:

How did you upgrade to the 4.1.2 version? Did you do like the Release Notes said and completely remove the package before installing again, or did you just click the re-install icon on the Package Manager Installed Packages tab? If you did the latter, go back and completely remove the package by hitting the delete icon (the little trashcan). After the removal completes, go back to the Package Manager screen and install Suricata again from the Available Packages tab. This is a very critical process, and is why I highlighted it in red in the Release Notes for the 4.1.2 package update notice.

kiekar

Thank you for your help so far.

Below is the output.

Shell Output - ls /usr/local/etc/suricata
classification.config
classification.config.sample
community-rules.tar.gz.md5
emerging.rules.tar.gz.md5
reference.config
reference.config.sample
rules
snortrules-snapshot-3000.tar.gz.md5
suricata.yaml
suricata.yaml.sample
suricata_26546_pppoe1
threshold.config
threshold.config.sample

Shell Output - cd /usr/local/etc/suricata/suricata_26546_pppoe1
Execute Shell Command

Shell Output - suricata -T -c ./suricata.yaml
16/1/2019 -- 15:03:54 - <Info> - Running suricata under test mode
16/1/2019 -- 15:03:54 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: ./suricata.yaml: No such file or directory

bmeeks

@kiekar said in Suricata fails to start after pfSense reboot:

Thank you for your help so far.

Below is the output.

Shell Output - ls /usr/local/etc/suricata
classification.config
classification.config.sample
community-rules.tar.gz.md5
emerging.rules.tar.gz.md5
reference.config
reference.config.sample
rules
snortrules-snapshot-3000.tar.gz.md5
suricata.yaml
suricata.yaml.sample
suricata_26546_pppoe1
threshold.config
threshold.config.sample

Shell Output - cd /usr/local/etc/suricata/suricata_26546_pppoe1
Execute Shell Command

Shell Output - suricata -T -c ./suricata.yaml
16/1/2019 -- 15:03:54 - <Info> - Running suricata under test mode
16/1/2019 -- 15:03:54 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: ./suricata.yaml: No such file or directory

Not having a suricata.yaml file is weird. I see you did this from the firewall GUI command for executing a shell script, though, and I don't think that's going to work properly. I have a feeling the "cd" or "change directory" command was transient, and the "cd" was abandoned when the next command was executed.

If you can't get directly to the firewall console, then try running this command that gives the full path to the suricata.yaml file:

suricata -T -c /usr/local/etc/suricata/suricata_26546_pppoe1/suricata.yaml

kiekar

I got the same errcode

[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:

I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?

bmeeks

@kiekar said in Suricata fails to start after pfSense reboot:

I got the same errcode

[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:

I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?

Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?

Is there plenty of free disk space? I would assume "yes", but this is a strange issue.

Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.

You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!

kiekar

@bmeeks said in Suricata fails to start after pfSense reboot:

@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:
I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?

Is there plenty of free disk space? I would assume "yes", but this is a strange issue.

Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.

You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!

Yes, wan interface is setup correctly.

I'm going to uninstall suricata with my current settings intacked.

0_1547671631135_wan network interface.jpg

bmeeks

@kiekar said in Suricata fails to start after pfSense reboot:

@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:
I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?

Is there plenty of free disk space? I would assume "yes", but this is a strange issue.

Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.

You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.

I'm going to uninstall suricata with my current settings intacked.

Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.

kiekar

@bmeeks said in Suricata fails to start after pfSense reboot:

@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:
I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?

Is there plenty of free disk space? I would assume "yes", but this is a strange issue.

Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.

You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.

I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.

Ok Thanks for all your help. I will try to move the pppoe authenication from the firewall to the vsdl modem

bmeeks

@kiekar said in Suricata fails to start after pfSense reboot:

@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:
I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?

Is there plenty of free disk space? I would assume "yes", but this is a strange issue.

Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.

You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.

I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.
Ok Thanks for all your help. I will try to move the pppoe authenication from the firewall to the vsdl modem

If you want to use Inline IPS Mode, you need to have either a static WAN IP or DHCP on the WAN. PPPoE won't work. A PPPoE connection totally bypasses the Netmap pipe set up by Inline IPS Mode.

kiekar

@bmeeks said in Suricata fails to start after pfSense reboot:

@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml
16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode
16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory
[2.4.4-RELEASE][admin@host.mydomain.com]/root:
I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?

Is there plenty of free disk space? I would assume "yes", but this is a strange issue.

Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.

You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.

I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.
Ok Thanks for all your help. I will try to move the pppoe authenication from the firewall to the vsdl modem
If you want to use Inline IPS Mode, you need to have either a static WAN IP or DHCP on the WAN. PPPoE won't work. A PPPoE connection totally bypasses the Netmap pipe set up by Inline IPS Mode.

Ok Thanks, I'm going to have to do some reading how to setup using static IP's or DHCP. I have a block of IP's from my provider and I have been running my current setup for several years now.

bmeeks

@kiekar
Running the Legacy Blocking Mode is fine, and is really just about the same in terms of security. True there may be a leaked packet or two, but in most situations that is of no consequence. You can use all of the SID MGMT features and "alert" and "drop" actions in Legacy Mode. Just check the box on the INTERFACE SETTINGS tab that enables the "block-on-drops-only" option. That will mimic Inline IPS Mode whereby only rules with DROP as the action will actually block traffic. Rules with ALERT as the action will only generate alerts but no blocks.

And finally, if you are using NAT on pfSense, then I recommend putting Suricata on your LAN interface so that the internal host addresses you see will reflect reality. Otherwise, with NAT, all internal hosts seen on the WAN side will have the WAN external IP and you won't be able to tell which internal host generated the alert. This is assuming you use NAT. If you don't use NAT on the firewall, then you can place Suricata either on the WAN or LAN without the IP masking issue.

kiekar

@bmeeks said in Suricata fails to start after pfSense reboot:

@kiekar
Running the Legacy Blocking Mode is fine, and is really just about the same in terms of security. True there may be a leaked packet or two, but in most situations that is of no consequence. You can use all of the SID MGMT features and "alert" and "drop" actions in Legacy Mode. Just check the box on the INTERFACE SETTINGS tab that enables the "block-on-drops-only" option. That will mimic Inline IPS Mode whereby only rules with DROP as the action will actually block traffic. Rules with ALERT as the action will only generate alerts but no blocks.

And finally, if you are using NAT on pfSense, then I recommend putting Suricata on your LAN interface so that the internal host addresses you see will reflect reality. Otherwise, with NAT, all internal hosts seen on the WAN side will have the WAN external IP and you won't be able to tell which internal host generated the alert. This is assuming you use NAT. If you don't use NAT on the firewall, then you can place Suricata either on the WAN or LAN without the IP masking issue.

I've seen some brief topics on SID MGMT, but I have to say I'm a little lost on the function of it. Could you be so kind an point me in the right direction on some good reading on the SID MGMT. Your help would be much appreciated.

Thanks,

bmeeks

@kiekar
Start with these two Sticky Posts here in the IDS/IPS sub-forum.

How Automatic SID Management and User Rule Overrides Work in Snort and Suricata
About the New Block-on-Drops-Only Option in Suricata

To get some examples of SID MGMT, go to the SID MGMT tab and click the "enable" checkbox. That will populate the page with content. Open any of the example files and you will find comments inside each that explain the options. The internal syntax is copied from PulledPork, so you can Google the PulledPork utility used with Snort to get some examples of what you can do with the enablesid.conf, disablesid.conf and dropsid.conf files.

kiekar

@bmeeks said in Suricata fails to start after pfSense reboot:

@kiekar
Start with these two Sticky Posts here in the IDS/IPS sub-forum.

How Automatic SID Management and User Rule Overrides Work in Snort and Suricata
About the New Block-on-Drops-Only Option in Suricata

To get some examples of SID MGMT, go to the SID MGMT tab and click the "enable" checkbox. That will populate the page with content. Open any of the example files and you will find comments inside each that explain the options. The internal syntax is copied from PulledPork, so you can Google the PulledPork utility used with Snort to get some examples of what you can do with the enablesid.conf, disablesid.conf and dropsid.conf files.

Great, thanks so much. You've been a great help