Suricata fails to start after pfSense reboot
-
Hello,
I'm new to Suricata and have been reading threads on using IPS inline mode. I found after rebooting pfSense, Suricata service won't start unless I remove the pid file from /var/run where by Suricata will start. This happens each time I reboot pfSense. Is this an issue with using IPS inline mode? using legacy mode i had no issues. I'm using a intel pro/1000 4 port network card. I'm on the latest version of pfSense 2.4.4-p2. I also noticed before the reboot that no alerts were showing on alerts tab. One other thing i used the quick setup instructions [https://forum.netgate.com/topic/55095/quick-snort-setup-instructions-for-new-users] for IPS inline mode. Any help would be much appreciated.
Thanks,
code 15/1/2019 -- 18:28:16 - <Notice> -- This is Suricata version 4.1.2 RELEASE 15/1/2019 -- 18:28:16 - <Info> -- CPUs/cores online: 2 15/1/2019 -- 18:28:16 - <Info> -- HTTP memcap: 67108864 15/1/2019 -- 18:28:17 - <Notice> -- using flow hash instead of active packets 15/1/2019 -- 18:28:17 - <Error> -- [ERRCODE: SC_ERR_INITIALIZATION(45)] - pid file '/var/run/suricata_pppoe126546.pid' exists but appears stale. Make sure Suricata is not running and then remove /var/run/suricata_pppoe126546.pid. Aborting!and after pid removal and restart of suricata
15/1/2019 -- 18:56:18 - <Notice> -- This is Suricata version 4.1.2 RELEASE 15/1/2019 -- 18:56:18 - <Info> -- CPUs/cores online: 2 15/1/2019 -- 18:56:18 - <Info> -- HTTP memcap: 67108864 15/1/2019 -- 18:56:18 - <Notice> -- using flow hash instead of active packets 15/1/2019 -- 18:56:19 - <Info> -- Netmap: Setting IPS mode 15/1/2019 -- 18:56:19 - <Info> -- fast output device (regular) initialized: alerts.log 15/1/2019 -- 18:56:19 - <Info> -- http-log output device (regular) initialized: http.log 15/1/2019 -- 18:56:20 - <Info> -- 3 rule files processed. 6796 rules successfully loaded, 0 rules failed 15/1/2019 -- 18:56:20 - <Info> -- Threshold config parsed: 0 rule(s) found 15/1/2019 -- 18:56:20 - <Info> -- 6796 signatures processed. 211 are IP-only rules, 3135 are inspecting packet payload, 4270 inspect application layer, 102 are decoder event only 15/1/2019 -- 18:56:20 - <Warning> -- [ERRCODE: SC_WARN_FLOWBIT(306)] - flowbit 'et.http.PK' is checked but not set. Checked in 2019835 and 1 other sigs 15/1/2019 -- 18:56:24 - <Info> -- Using 2 live device(s). 15/1/2019 -- 18:56:24 - <Notice> -- all 4 packet processing threads, 2 management threads initialized, engine started. -
What kind of hardware are you running pfSense on? Is it an SG-3100 appliance or similar non-Intel CPU box?
If so, then look in the pfSense system log and I bet you will find a Signal 10 Bus Error. This is a memory-alignment error fault. This problem cropped up back in the 4.0.5 or 4.0.6 update of the Suricata binary and is caused by the machine opcode decisions the clang/llvm compiler makes when optimization is enabled.
Even if you are not running non-Intel hardware, look in the pfSense system log to see if any error messages are present.
-
@bmeeks said in Suricata fails to start after pfSense reboot:
What kind of hardware are you running pfSense on? Is it an SG-3100 appliance or similar non-Intel CPU box?
It's a home built PC running on a intel H310 chipset board with an 8 gen cpu and Dell Intel PRO/1000 ET Quad Port 1Gb Gigabit Ethernet Adapter H092P HM9JY.
I was unable to find any errors showing Signal 10 Bus Error in system log except for errors for notpng
-
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
What kind of hardware are you running pfSense on? Is it an SG-3100 appliance or similar non-Intel CPU box?
It's a home built PC running on a intel H310 chipset board with an 8 gen cpu and Dell Intel PRO/1000 ET Quad Port 1Gb Gigabit Ethernet Adapter H092P HM9JY.
I was unable to find any errors showing Signal 10 Bus Error in system log except for errors for notpng
OK, then the next most likely problem is a syntax issue in the suricata.yaml file. To see what it may be, do this:
Open a command prompt (shell prompt) on the firewall.
Execute these commands:ls /usr/local/etc/suricata cd /usr/local/etc/suricata/suricata_xxxx suricata -T -c ./suricata.yamlIn the above command sequence, the first command will list the contents of the Suricata base directory. Look for a sub-directory in there named suricata_xxxxx where the xxxxx will be a GUID string and the physical interface name. Substitute this directory name into the second command.
For example, suppose you see a sub-directory named suricata_1287_em0 within the /usr/local/etc/suricata directory. Then the second command would look like this:
cd /usr/local/suricata/suricata_1287_em0This will start a Suricata session to test the suricata.yaml configuration file. A lot of output will scroll by. Look for any syntax errors that appear and post those back here. If no errors appear, post that back as well. If the process starts successfully, then you will need to type CTRL-C to terminate the process and regain your shell prompt.
-
One more question:
How did you upgrade to the 4.1.2 version? Did you do like the Release Notes said and completely remove the package before installing again, or did you just click the re-install icon on the Package Manager Installed Packages tab? If you did the latter, go back and completely remove the package by hitting the delete icon (the little trashcan). After the removal completes, go back to the Package Manager screen and install Suricata again from the Available Packages tab. This is a very critical process, and is why I highlighted it in red in the Release Notes for the 4.1.2 package update notice.
-
Thank you for your help so far.
Below is the output.
Shell Output - ls /usr/local/etc/suricata classification.config classification.config.sample community-rules.tar.gz.md5 emerging.rules.tar.gz.md5 reference.config reference.config.sample rules snortrules-snapshot-3000.tar.gz.md5 suricata.yaml suricata.yaml.sample suricata_26546_pppoe1 threshold.config threshold.config.sampleShell Output - cd /usr/local/etc/suricata/suricata_26546_pppoe1 Execute Shell CommandShell Output - suricata -T -c ./suricata.yaml 16/1/2019 -- 15:03:54 - <Info> - Running suricata under test mode 16/1/2019 -- 15:03:54 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: ./suricata.yaml: No such file or directory -
@kiekar said in Suricata fails to start after pfSense reboot:
Thank you for your help so far.
Below is the output.
Shell Output - ls /usr/local/etc/suricata classification.config classification.config.sample community-rules.tar.gz.md5 emerging.rules.tar.gz.md5 reference.config reference.config.sample rules snortrules-snapshot-3000.tar.gz.md5 suricata.yaml suricata.yaml.sample suricata_26546_pppoe1 threshold.config threshold.config.sampleShell Output - cd /usr/local/etc/suricata/suricata_26546_pppoe1 Execute Shell CommandShell Output - suricata -T -c ./suricata.yaml 16/1/2019 -- 15:03:54 - <Info> - Running suricata under test mode 16/1/2019 -- 15:03:54 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: ./suricata.yaml: No such file or directoryNot having a suricata.yaml file is weird. I see you did this from the firewall GUI command for executing a shell script, though, and I don't think that's going to work properly. I have a feeling the "cd" or "change directory" command was transient, and the "cd" was abandoned when the next command was executed.
If you can't get directly to the firewall console, then try running this command that gives the full path to the suricata.yaml file:
suricata -T -c /usr/local/etc/suricata/suricata_26546_pppoe1/suricata.yaml -
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
-
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?
Is there plenty of free disk space? I would assume "yes", but this is a strange issue.
Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.
You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
-
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?
Is there plenty of free disk space? I would assume "yes", but this is a strange issue.
Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.
You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.
I'm going to uninstall suricata with my current settings intacked.
-
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?
Is there plenty of free disk space? I would assume "yes", but this is a strange issue.
Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.
You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.
I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.
-
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?
Is there plenty of free disk space? I would assume "yes", but this is a strange issue.
Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.
You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.
I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.
Ok Thanks for all your help. I will try to move the pppoe authenication from the firewall to the vsdl modem
-
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?
Is there plenty of free disk space? I would assume "yes", but this is a strange issue.
Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.
You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.
I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.
Ok Thanks for all your help. I will try to move the pppoe authenication from the firewall to the vsdl modem
If you want to use Inline IPS Mode, you need to have either a static WAN IP or DHCP on the WAN. PPPoE won't work. A PPPoE connection totally bypasses the Netmap pipe set up by Inline IPS Mode.
-
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar said in Suricata fails to start after pfSense reboot:
I got the same errcode
[2.4.4-RELEASE][admin@host.mydomain.com]/root: suricata -T -c /usr/local/etc/suricata_26546_pppoe1/suricata.yaml 16/1/2019 -- 15:29:42 - <Info> - Running suricata under test mode 16/1/2019 -- 15:29:42 - <Error> - [ERRCODE: SC_ERR_FATAL(171)] - failed to open file: /usr/local/etc/suricata_26546_pppoe1/suricata.yaml: No such file or directory [2.4.4-RELEASE][admin@host.mydomain.com]/root:I will uninstall surcicata and re-install. Could i leave the checkbox to keep current settings?
Not having a suricata.yaml file for the interface is very strange. Have you gone to the INTERFACE SETTINGS tab and clicked the Save button? Is that interface (pppoe1) the physical name of your WAN interface where you are running Suricata?
Is there plenty of free disk space? I would assume "yes", but this is a strange issue.
Never seen a case where the suricata.yaml for the interface was missing. That file tells the Suricata binary what to do and how to do it. No surprise it won't start with that file missing.
You can try a full remove and install operation. Just be sure you delete the package and then install it again. DO NOT just click the re-install icon!
Yes, wan interface is setup correctly.
I'm going to uninstall suricata with my current settings intacked.
Oh! Something I just remembered and I did not notice earlier. Inline IPS Mode and a PPPoE connection will NOT work! You will have to switch to Legacy Mode blocking with a PPPoE connection. You will never get any alerts with PPPoE and Inline Mode. It is a limitation of FreeBSD and Netmap.
Ok Thanks for all your help. I will try to move the pppoe authenication from the firewall to the vsdl modem
If you want to use Inline IPS Mode, you need to have either a static WAN IP or DHCP on the WAN. PPPoE won't work. A PPPoE connection totally bypasses the Netmap pipe set up by Inline IPS Mode.
Ok Thanks, I'm going to have to do some reading how to setup using static IP's or DHCP. I have a block of IP's from my provider and I have been running my current setup for several years now.
-
@kiekar
Running the Legacy Blocking Mode is fine, and is really just about the same in terms of security. True there may be a leaked packet or two, but in most situations that is of no consequence. You can use all of the SID MGMT features and "alert" and "drop" actions in Legacy Mode. Just check the box on the INTERFACE SETTINGS tab that enables the "block-on-drops-only" option. That will mimic Inline IPS Mode whereby only rules with DROP as the action will actually block traffic. Rules with ALERT as the action will only generate alerts but no blocks.And finally, if you are using NAT on pfSense, then I recommend putting Suricata on your LAN interface so that the internal host addresses you see will reflect reality. Otherwise, with NAT, all internal hosts seen on the WAN side will have the WAN external IP and you won't be able to tell which internal host generated the alert. This is assuming you use NAT. If you don't use NAT on the firewall, then you can place Suricata either on the WAN or LAN without the IP masking issue.
-
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar
Running the Legacy Blocking Mode is fine, and is really just about the same in terms of security. True there may be a leaked packet or two, but in most situations that is of no consequence. You can use all of the SID MGMT features and "alert" and "drop" actions in Legacy Mode. Just check the box on the INTERFACE SETTINGS tab that enables the "block-on-drops-only" option. That will mimic Inline IPS Mode whereby only rules with DROP as the action will actually block traffic. Rules with ALERT as the action will only generate alerts but no blocks.And finally, if you are using NAT on pfSense, then I recommend putting Suricata on your LAN interface so that the internal host addresses you see will reflect reality. Otherwise, with NAT, all internal hosts seen on the WAN side will have the WAN external IP and you won't be able to tell which internal host generated the alert. This is assuming you use NAT. If you don't use NAT on the firewall, then you can place Suricata either on the WAN or LAN without the IP masking issue.
I've seen some brief topics on SID MGMT, but I have to say I'm a little lost on the function of it. Could you be so kind an point me in the right direction on some good reading on the SID MGMT. Your help would be much appreciated.
Thanks,
-
@kiekar
Start with these two Sticky Posts here in the IDS/IPS sub-forum.How Automatic SID Management and User Rule Overrides Work in Snort and Suricata
About the New Block-on-Drops-Only Option in SuricataTo get some examples of SID MGMT, go to the SID MGMT tab and click the "enable" checkbox. That will populate the page with content. Open any of the example files and you will find comments inside each that explain the options. The internal syntax is copied from PulledPork, so you can Google the PulledPork utility used with Snort to get some examples of what you can do with the enablesid.conf, disablesid.conf and dropsid.conf files.
-
@bmeeks said in Suricata fails to start after pfSense reboot:
@kiekar
Start with these two Sticky Posts here in the IDS/IPS sub-forum.How Automatic SID Management and User Rule Overrides Work in Snort and Suricata
About the New Block-on-Drops-Only Option in SuricataTo get some examples of SID MGMT, go to the SID MGMT tab and click the "enable" checkbox. That will populate the page with content. Open any of the example files and you will find comments inside each that explain the options. The internal syntax is copied from PulledPork, so you can Google the PulledPork utility used with Snort to get some examples of what you can do with the enablesid.conf, disablesid.conf and dropsid.conf files.
Great, thanks so much. You've been a great help
