Snort - not starting anymore
-
Hi again!
I updated pfSense (2.15. nano serial 4GB) to the latest snort some days ago by un-installing and re-stalling snort without obvious problems.
Since yesterday there were no alarms or blocked sites at the snort tabs, although updates were apparently successfully installed. I did a reboot, afterwards snort did not start again:
Nov 8 20:52:29 snort[14552]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_re055504/alert: No such file or directory
Nov 8 20:52:28 SnortStartup[14352]: Snort START for OPT1(55504_re0)…
Nov 8 20:52:28 snort[13507]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_re247470/alert: No such file or directory
Nov 8 20:52:28 SnortStartup[13273]: Snort START for LAN fam(47470_re2)…
Nov 8 20:52:28 snort[12811]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_re159777/alert: No such file or directory
Nov 8 20:52:27 SnortStartup[12644]: Snort START for WAN(59777_re1)…
Nov 8 20:49:35 snort[46689]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_re055504/alert: No such file or directory
Nov 8 20:49:35 SnortStartup[46667]: Snort START for OPT1(55504_re0)…
Nov 8 20:49:35 snort[46061]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_re247470/alert: No such file or directory
Nov 8 20:49:35 SnortStartup[45842]: Snort START for LAN fam(47470_re2)…
Nov 8 20:49:35 snort[44977]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_re159777/alert: No such file or directory
Nov 8 20:49:34 SnortStartup[44846]: Snort START for WAN(59777_re1)…Forced a rules update (paid oink VRT, Community, ET rules), did another reboot, switched to other slice without success, snort is not coming up again.
The /var/log/snort looks like given below
Any suggestions what to do (besides switching to a full install)? ;-)
Many thanx in advance
chemlud
-
Package reinstall for snort. Appears to be up and running again…
Which process deleted the directories in /var/log/snort ? :o
-
Well, as you said, my recommendation is moving to a full install on a conventional hard disk and abandon Nano. It seems to give all the packages trouble.
Since I do not have a Nano install to test on, I am a bit hamstrung trying to make the Snort package behave better on Nano. Does anyone know if I can fool NanoBSD into installing on a VMware virtual machine? I have never tried. Or does someone have a Nano-capable piece of hardware they want to donate to the effort so I can test Snort and Suricata on a Nano install?
Bill
-
my three boxes are all "productive" and the shipping to the US would kill me anyways… what would be the minimum hardware requirements for such a box? could I order somefing for you at amazon? :-)
chemlud
....PS: I don't remember if I ever tried to boot a CF card from a USB-card reader device plugged in to a notebook. Could give it a try...
PPS: 2.1.5 nano vga 4gb on CF-card with card reader: boot process stops at
Trying to mount root from ufs:/dev/ufs/pfsense0
"ROOT MOUNT ERROR"
(...same issue as with USB-stick...)
PPPS: Pressing "3" (booting from USB device) and it runs like a charm, apparently...
-
@chemlud:
my three boxes are all "productive" and the shipping to the US would kill me anyways… what would be the minimum hardware requirements for such a box? could I order somefing for you at amazon? :-)
chemlud
....PS: I don't remember if I ever tried to boot a CF card from a USB-card reader device plugged in to a notebook. Could give it a try...
PPS: 2.1.5 nano vga 4gb on CF-card with card reader: boot process stops at
Trying to mount root from ufs:/dev/ufs/pfsense0
"ROOT MOUNT ERROR"
(...same issue as with USB-stick...)
PPPS: Pressing "3" (booting from USB device) and it runs like a charm, apparently...
Having never used Nano, I honestly don't know what the specs should be. As far as Snort goes, the problems seem to occur during installs and updates and not during run times, so the box would not need to be much in terms of CPU or RAM. What I mean is there is no need for large amounts of RAM or a super fast CPU if the only requirement is install and Snort starting up successfully and updating successfully.
The best thing is if I can get it running in a VM. I will try that first. I never have tried before. Will have to research if it is even possible to fool it like that.
Bill
-
Believe, me you will like nano from the very first moment, the two slices are pretty cool! 8)
I have set /tmp to 450 MB and /var to 250 MB under System/Advances/Misc…
chemlud
If you need real hardware and it's too much for me alone, maybe some nano users want to join, could check this, if necessary...
-
After signature update yesterday evening the same problem: No alarms any more. Reboot: Snort does not start due to same error as posted above. Switching slice to copy made yesterday: Same errors (how can that be?).
Next try: A new CF-card. But for me it look like someone is messing up the updates to inactivate snort. That's not funny.
-
@chemlud:
After signature update yesterday evening the same problem: No alarms any more. Reboot: Snort does not start due to same error as posted above. Switching slice to copy made yesterday: Same errors (how can that be?).
Next try: A new CF-card. But for me it look like someone is messing up the updates to inactivate snort. That's not funny.
I did some Google research yesterday. Here is the deal as I currently understand it:
There are some tricks that will allow a NanoBSD install into a VMware VM. So I could test without needing physical hardware.
However, after learning some more during my research about how Nano actually works, I see what the problem is and there is no easy fix. The /var partition is a RAM disk that is wiped out with every reboot. So each time you boot your box, the /var partition starts out fresh and clean (that is, no files or directories are there). As you have noticed, Snort creates its logging directories in /var upon installation. It then assumes those directories stay there until the user manually uninstalls the Snort package. On a conventional full-install setup with a hard disk, that assumption works. Obviously on Nano with its RAM disk setup for /var, all of Snort's created logging directories are gone after a reboot. That's why you see those errors about non-existent files and directories.
It would require rewriting a lot of the Snort package to fix this, and then the fix would have to store log files in places they really should not be (like on the /usr partition). It is likely sufficient free space would also then become an issue.
So I'm afraid for now that Snort and Suricata are not supported on NanoBSD installs. If you want to hack it yourself and see what you can get working, then please do and share any results back with me. Perhaps you may find a way I have not thought about.
Bill
-
Hmmm, your suggested mechanism would imply that the error occurs on each and every nano + snort. But it worked fine on 3 boxes for about 1 year now, and the other boxes are doing fine with snort, even when rebooted from time to time.
The directories in /var/log are created during starting-up snort.
The problem occurs during rules updates, something is messed up, from time to time. Don't know where or what, I'm only a user, not an IT pro…
-
@chemlud:
Hmmm, your suggested mechanism would imply that the error occurs on each and every nano + snort. But it worked fine on 3 boxes for about 1 year now, and the other boxes are doing fine with snort, even when rebooted from time to time.
The directories in /var/log are created during starting-up snort.
The problem occurs during rules updates, something is messed up, from time to time. Don't know where or what, I'm only a user, not an IT pro…
The rule updates download the rule tar balls to a directory on /tmp. The files are unpacked there and then copied to locations on /usr. There have been users reporting problems running out of space on /tmp during the unzip process. Those users fixed it by increasing the /tmp partition to at least 80 MB and most to 100 MB. Perhaps that is happening?
I will try to get a Nano install working in a VM so I can see firsthand what might be happening. I based my earlier remarks on what I learned from some Google research. As shocking as it sounds, it may in fact be true that I should not believe everything I read on the Internet … ;D
Bill
-
As shocking as it sounds, it may in fact be true that I should not believe everything I read on the Internet … ;D
Wise man! :-D
One difference to the other two boxes, which are doing fine: These two have no subscribed oink code, only the free rules sets. One of the few differences I can imagine...
-
@chemlud:
One difference to the other two boxes, which are doing fine: These two have no subscribed oink code, only the free rules sets. One of the few differences I can imagine…
That should not matter. There is no difference at all in the download process other than the actual value of the code. Both use the same URL and download directory, the Oinkcode is passed to Snort.org as a query string.
What could factor in, though, is the free space available. I have not checked the differences in the tarball file sizes between the free and pay rules, but I doubt it would be much of a size difference.
So just to be clear: you have multiple boxes, all on NanoBSD installs, all with the same Snort version, and two of them behave differently than the others. Is that correct? If so, are the four exactly the same in terms of CF size, free space on /tmp and /var, version of the Snort package, version of pfSense, etc.?
Bill
-
After each reboot after installing the latest Snort package, I need to go to Snort settings and save them again to get Snort to load up. It throws an error:
SnortStartup[9906]: Snort START for SnortLAN(330_em1)... pfsense snort[10117]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_em1330/alert: No such file or directoryRunning 2.2 snapshots with ram drive for /var and /tmp so those get wiped on every reboot.
-
After each reboot after installing the latest Snort package, I need to go to Snort settings and save them again to get Snort to load up. It throws an error:
SnortStartup[9906]: Snort START for SnortLAN(330_em1)... pfsense snort[10117]: FATAL ERROR: OpenAlertFile() => fopen() alert file /var/log/snort/snort_em1330/alert: No such file or directoryRunning 2.2 snapshots with ram drive for /var and /tmp so those get wiped on every reboot.
This is expected behavior when you use the RAM drive. Those directories are created when you make a "save" to the Snort configuration. During a reboot, starting of Snort is handled by a shell script instead in /usr/local/etc/rc.d. That script does not create the logging directories – it just assumes they exist.
I guess it would be possible to increase the complexity of the shell script so it creates the directories if missing, but you also will lose all previously logged data, so the ALERTS tab will always come up empty on the reboot. Same thing for the Barnyard2 unified2 files.
Oh, and Snort also stores the IP REP databases and the new Auto-SID management files in /var/db/snort, so all that will be gone after a reboot. You would need to recreate those from scratch after each reboot.
So my real question is what is the draw of running Snort or Suricata on such a limited logging platform? Are you sending everything off someplace else?
Bill
-
All three are 2.1.5 nano serial, two with the latest snort (one ok, one with the problem), the other has snort version 3.1.2 (is located remotely, only updated when I'm physically present).
The two working fine have 1GB RAM, the one with problems 2GB (all Atom systems).
All 4GB Transcend industrial CF cards 200 or 300.
/tmp is set to /var to 250 MB each on the smaller systems (working), the one with the problems has /tmp set to 450 MB and /var to 250 MB.
Logging: I have a cron job sending the snort log to a local email server. I reboot is not that frequent…
-
@chemlud:
All three are 2.1.5 nano serial, two with the latest snort (one ok, one with the problem), the other has snort version 3.1.2 (is located remotely, only updated when I'm physically present).
The two working fine have 1GB RAM, the one with problems 2GB (all Atom systems).
All 4GB Transcend industrial CF cards 200 or 300.
/tmp is set to /var to 250 MB each on the smaller systems (working), the one with the problems has /tmp set to 450 MB and /var to 250 MB.
How much actual free space is available on /tmp during the update procedure, though? Could you find that out? Maybe login via SSH and do a continuous
df -hwhile kicking off the rules update from a web session. Also scour your system logs to see if you are getting any kind of disk messages (particularly about space).
Bill
-
I'm not running on a "limited system" I just turned on ram drive for my setup way back when. This issue is a new one with the latest Snort package version. The package used to work fine for me before.
Only thing I noticed not working with Snort before was enabling the IP list file thingy from the ET Open ruleset.
-
I'm not running on a "limited system" I just turned on ram drive for my setup way back when. This issue is a new one with the latest Snort package version. The package used to work fine for me before.
Only thing I noticed not working with Snort before was enabling the IP list file thingy from the ET Open ruleset.
OK, I was able to get a Nano install working as a VM. That let me test this out. What I suspected is true – namely that /var partition is wiped out on each reboot.
I also remembered moving some code to a different place for efficiency, but that move probably opened up this hole on Nano installs. Some of what the moved code did was create the logging directories. It was moved to reside with all the other install/save code. Seeing how Nano works, that likely introduced this problem.
Give me some time to come up with a good fix, and I will post an update for the pfSense devs to review and approve.
Note that this will fix the problem of Snort not starting on a reboot on Nano platforms. I still have not been able to reproduce chemlud's problem with the rule updates. I will keep testing, though.
Bill
-
As I'm a little busy these days, too, here the RAM RRD graph for the problematic box for this mornings rules update starting at 9:00-something and taking about half an hour to complete…
Hope that helps! If anything else is needed I might find some time next week!
Many thanx for the time spent so far on this issue
chemlud
-
@chemlud:
As I'm a little busy these days, too, here the RAM RRD graph for the problematic box for this mornings rules update starting at 9:00-something and taking about half an hour to complete…
Hope that helps! If anything else is needed I might find some time next week!
Many thanx for the time spent so far on this issue
chemlud
The rules update takes up to a half-hour to complete? That is very unusual. What is the WAN download speed for the box? A long update time would be 4 or 5 minutes. Mine are usually 30 seconds to maybe 90 seconds if both ET and VRT rules update together. I have a 24 megabits/sec download speed.
Bill
-
It's not the download alone (between 16 and 100 Mbit/s download), afterwards the building of sig maps for all three interfaces et. pp…. The hardware is dualcore but not the very latest, so overall it's about half an hour till then snort instances are stopped and finally restarted.
Here an example from another box:
Nov 11 13:03:05 domain php: snort_check_for_rule_updates.php: [Snort] Snort VRT rules are up to date... Nov 11 13:03:08 domain php: snort_check_for_rule_updates.php: [Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz... Nov 11 13:03:11 domain php: snort_check_for_rule_updates.php: [Snort] Snort GPLv2 Community Rules file update downloaded successfully Nov 11 13:03:12 domain php: snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz... Nov 11 13:03:16 domain php: snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully Nov 11 13:03:52 domain php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN ... Nov 11 13:04:17 domain kernel: pid 46476 (snort), uid 0, was killed: out of swap space Nov 11 13:04:17 domain kernel: re2: promiscuous mode disabled Nov 11 13:08:57 domain php: snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: WAN... Nov 11 13:09:21 domain php: snort_check_for_rule_updates.php: [Snort] Building new sig-msg.map file for WAN... Nov 11 13:13:50 domain php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: LAN ... Nov 11 13:19:27 domain php: snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: LAN... Nov 11 13:20:00 domain php: snort_check_for_rule_updates.php: [Snort] Building new sig-msg.map file for LAN... Nov 11 13:25:35 domain php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: OPT1 ... Nov 11 13:31:04 domain php: snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: OPT1... Nov 11 13:31:28 domain php: snort_check_for_rule_updates.php: [Snort] Building new sig-msg.map file for OPT1... Nov 11 13:36:26 domain SnortStartup[12651]: Snort STOP for interface(50118_pppoe0)... Nov 11 13:36:26 domain snort[3398]: *** Caught Term-Signal Nov 11 13:36:26 domain kernel: pppoe0: promiscuous mode disabled Nov 11 13:36:31 domain SnortStartup[14732]: Snort STOP for LAN(12330_re2)... Nov 11 13:36:33 domain SnortStartup[15615]: Snort STOP for OPT1 (54662_re0)... Nov 11 13:36:34 domain snort[47686]: *** Caught Term-Signal Nov 11 13:36:34 domain kernel: re0: promiscuous mode disabled Nov 11 13:36:47 domain php: snort_check_for_rule_updates.php: [Snort] Snort has restarted with your new set of rules... Nov 11 13:36:47 domain php: snort_check_for_rule_updates.php: [Snort] The Rules update has finished. Nov 11 13:36:47 domain SnortStartup[42078]: Snort START for interface(50118_pppoe0)... Nov 11 13:36:50 domain check_reload_status: Syncing firewall Nov 11 13:38:11 domain kernel: pppoe0: promiscuous mode enabled Nov 11 13:38:14 domain SnortStartup[67697]: Snort START for LAN(12330_re2)... Nov 11 13:40:57 domain kernel: re2: promiscuous mode enabled Nov 11 13:40:59 domain SnortStartup[29775]: Snort START for OPT1 (54662_re0)... Nov 11 13:42:35 domain kernel: re0: promiscuous mode enabled
-
OK, I found the problem. It's not what I initially thought it was. It had nothing to do with moved code. Instead, a version string did not get updated in the last release. That string is really not necessary any more, so I will remove it in the next update. For you guys with RAM disks for /tmp and /var, including those running on Nano installs, here is a quick fix you can do yourself if you want to:
1. Go to Diagnostics…Edit File in the pfSense menu and then browse to and open the file /usr/local/pkg/snort.xml:
2. Scroll down to near the bottom of the file and find this section of text
<custom_php_resync_config_command>if ($GLOBALS['pfSense_snort_version'] == "3.1.3") sync_snort_package_config(); ]]></custom_php_resync_config_command>3. Edit it so it looks like this and then save the change:
<custom_php_resync_config_command>sync_snort_package_config(); ]]></custom_php_resync_config_command>Notice that the entire line containing the "if()" statement was removed.
This should fix the problem of Snort failing to restart on a reboot. I will soon post a fix for the pfSense developers to merge to production, but you can make the edit above yourself if need the fix sooner.
EDIT: here is the posted Pull Request for review by the pfSense developers.
https://github.com/pfsense/pfsense-packages/pull/725
Bill
-
@chemlud:
It's not the download alone (between 16 and 100 Mbit/s download), afterwards the building of sig maps for all three interfaces et. pp…. The hardware is dualcore but not the very latest, so overall it's about half an hour till then snort instances are stopped and finally restarted.
Here an example from another box:
Nov 11 13:03:05 domain php: snort_check_for_rule_updates.php: [Snort] Snort VRT rules are up to date... Nov 11 13:03:08 domain php: snort_check_for_rule_updates.php: [Snort] There is a new set of Snort GPLv2 Community Rules posted. Downloading community-rules.tar.gz... Nov 11 13:03:11 domain php: snort_check_for_rule_updates.php: [Snort] Snort GPLv2 Community Rules file update downloaded successfully Nov 11 13:03:12 domain php: snort_check_for_rule_updates.php: [Snort] There is a new set of Emerging Threats Open rules posted. Downloading emerging.rules.tar.gz... Nov 11 13:03:16 domain php: snort_check_for_rule_updates.php: [Snort] Emerging Threats Open rules file update downloaded successfully Nov 11 13:03:52 domain php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: WAN ... Nov 11 13:04:17 domain kernel: pid 46476 (snort), uid 0, was killed: out of swap space Nov 11 13:04:17 domain kernel: re2: promiscuous mode disabled Nov 11 13:08:57 domain php: snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: WAN... Nov 11 13:09:21 domain php: snort_check_for_rule_updates.php: [Snort] Building new sig-msg.map file for WAN... Nov 11 13:13:50 domain php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: LAN ... Nov 11 13:19:27 domain php: snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: LAN... Nov 11 13:20:00 domain php: snort_check_for_rule_updates.php: [Snort] Building new sig-msg.map file for LAN... Nov 11 13:25:35 domain php: snort_check_for_rule_updates.php: [Snort] Updating rules configuration for: OPT1 ... Nov 11 13:31:04 domain php: snort_check_for_rule_updates.php: [Snort] Enabling any flowbit-required rules for: OPT1... Nov 11 13:31:28 domain php: snort_check_for_rule_updates.php: [Snort] Building new sig-msg.map file for OPT1... Nov 11 13:36:26 domain SnortStartup[12651]: Snort STOP for interface(50118_pppoe0)... Nov 11 13:36:26 domain snort[3398]: *** Caught Term-Signal Nov 11 13:36:26 domain kernel: pppoe0: promiscuous mode disabled Nov 11 13:36:31 domain SnortStartup[14732]: Snort STOP for LAN(12330_re2)... Nov 11 13:36:33 domain SnortStartup[15615]: Snort STOP for OPT1 (54662_re0)... Nov 11 13:36:34 domain snort[47686]: *** Caught Term-Signal Nov 11 13:36:34 domain kernel: re0: promiscuous mode disabled Nov 11 13:36:47 domain php: snort_check_for_rule_updates.php: [Snort] Snort has restarted with your new set of rules... Nov 11 13:36:47 domain php: snort_check_for_rule_updates.php: [Snort] The Rules update has finished. Nov 11 13:36:47 domain SnortStartup[42078]: Snort START for interface(50118_pppoe0)... Nov 11 13:36:50 domain check_reload_status: Syncing firewall Nov 11 13:38:11 domain kernel: pppoe0: promiscuous mode enabled Nov 11 13:38:14 domain SnortStartup[67697]: Snort START for LAN(12330_re2)... Nov 11 13:40:57 domain kernel: re2: promiscuous mode enabled Nov 11 13:40:59 domain SnortStartup[29775]: Snort START for OPT1 (54662_re0)... Nov 11 13:42:35 domain kernel: re0: promiscuous mode enabledThis error is a little strange:
Nov 11 13:04:17 domain kernel: pid 46476 (snort), uid 0, was killed: out of swap spaceHow much RAM did you say was in the box?
Bill
-
1 GB only. It's the first time I see this ;-) …copied just from the most recent log... Would have to do some research if this happened in the past. Overall this box is normally doing fine. All boxes are not for high-throughput, more for "controlled access areas" with very limited number of users.
-
@chemlud:
1 GB only. It's the first time I see this ;-) …copied just from the most recent log... Would have to do some research if this happened in the past. Overall this box is normally doing fine. All boxes are not for high-throughput, more for "controlled access areas" with very limited number of users.
This box might be on the ragged edge in terms of RAM with Snort. Depending on the number of enabled rules, Snort can eat a lot of memory. I would recommend bumping this box up to 2GB if possible.
Bill
-
no, not possible. Maybe I should remove some rules, especially from the WAN interface, iirc what is frequently recommended…
-
@chemlud:
no, not possible. Maybe I should remove some rules, especially from the WAN interface, iirc what is frequently recommended…
Yes, you certainly would not want any duplicate rules (same ones on more than one interface) in a box with limited RAM. Also make sure you have the pattern matcher set to AC-BNFA-NQ. That is the most memory and speed efficient setting.
The rules update process can consume up to 200 MB of memory itself on a temporary basis because it needs to hold a lot of data in some in-memory arrays while building the rules files. So the combination of that memory consumption combined with what the Snort binary process is already using could put your 1GB box over the top. That might explain why it sometimes dies during an update. The rules update process will only restart Snort if it is detected as running during the update process.
Bill
-
I have AC-BNFA-NQ as standard..
"The rules update process will only restart Snort if it is detected as running during the update process."
That's what I expected, therefore I controlled this box some minutes ago, but all three snort-interfaces were up and running, strange indeed…
