Snort update coming soon – please read about an important change!
-
So, having issues with Snort again. Been running rather well up until the 2.1.2 update. Now it's back to not working at all.
At first the issue was the same as before, that being the signature errors I mentioned in a previous thread. So I used truncate, that helped but didn't fix, so I did what I did last time to correct it and wiped the DB clean again. Now the communication begins again, everything looks clean, I reboot for a sanity check and….
Apr 13 01:13:56 barnyard2[84684]: Barnyard2 exiting Apr 13 01:13:56 barnyard2[84684]: database mysql_error: Access denied for user 'root'@'domain.com' (using password: YES) Apr 13 01:13:56 barnyard2[84684]: Writing PID "84684" to file "/var/run/barnyard2_em055759.pid" Apr 13 01:13:56 barnyard2[84684]: PID path stat checked out ok, PID path set to /var/run Apr 13 01:13:56 barnyard2[84684]: Daemon initialized, signaled parent pid: 84444 Apr 13 01:13:56 barnyard2[84444]: Daemon parent exiting Apr 13 01:13:56 barnyard2[84444]: Initializing daemon mode
This happens.
Okay, whiskey-tango-foxtrot. Why you use root, pfSense? Root is not a configured user. Anywhere. Seems like a really simple error, but the error has me table bashing my skull as I can't figure out where the directive is coming from. I won't allow root to connect to my MySQL production server unless I'm physically needing it. So that's why it isn't working now. However, pfSense is absolutely determined that it will never connect under anything else. I have checked manually the files ( both Snort's pbi barnyard config files and pfSense overall config ), and they match what is currently shown in the webgui ( that is to say, with the normal dbuser configured for the Snort db ).
In desperation following a remove / install of Snort still facing the same issues, I cleanwiped the drive and reinstalled pfSense from scratch - that is to say, clean from a CD with no imports from a previous config file, everything reentered by hand - and the error remains the same. pfSense is determined to use root only root and nothing but root, and I'm not gonna let it; especially when everything says it should be using the proper user that for some reason it is ignoring.
-
This is a barnyard error. Are you using i386 or x64?
-
This is a barnyard error. Are you using i386 or x64?
x64. This error appears limited to only Barnyard, as Snort will startup without it but I'm unsure given that the issue initially began with signature errors which was a past problem pfSense version issue with Snort's rules.
-
This is a barnyard error. Are you using i386 or x64?
x64. This error appears limited to only Barnyard, as Snort will startup without it but I'm unsure given that the issue initially began with signature errors which was a past problem pfSense version issue with Snort's rules.
This version of the Snort package made some big changes to Barnyard2 like adding more output plugins. One side effect of that was it needed to reconfigure how the MySQL DB login info was stored. I wrote some code in the installation routines that attempts to migrate the old MySQL settings to the new format. It is possible some old settings could trip it up. Your error is a MySQL login error, and when the login fails Barnyard2 will stop. Go to the Barnyard tab in Snort and manually re-enter the login ID and password for the MySQL DB, then click Save. Try to start Barnyard2 after that on the Snort Interfaces tab.
Bill
-
This is a barnyard error. Are you using i386 or x64?
x64. This error appears limited to only Barnyard, as Snort will startup without it but I'm unsure given that the issue initially began with signature errors which was a past problem pfSense version issue with Snort's rules.
its a barnyard issue with the config i think. I comment out line 2030 in the snort.inc file:
config gen_file: {$snortcfgdir}/gen-msg.map # config hostname: {$snortbarnyard_hostname_info} config interface: {$if_real}
then i added something like this to the advance config section in the gui
config hostname: pfsense_wan
i also noticed that I can only get barnyard2 to run on one interface… if I enabled it to 2 other interfaces, i get this error in the log shortly after it connects:
Apr 13 00:48:07 barnyard2[54841]: =============================================================================== Apr 13 00:48:07 barnyard2[54841]: database: Closing connection to database "snort" Apr 13 00:48:07 barnyard2[54841]: Barnyard2 exiting Apr 13 00:48:07 barnyard2[54841]: FATAL ERROR: database mysql_error: Duplicate entry '26455-2' for key 'PRIMARY' SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('18176','26455','2');] Apr 13 00:45:19 barnyard2[8131]: ===============================================================================
i haven't had a chance to research it yet
-
This is a barnyard error. Are you using i386 or x64?
x64. This error appears limited to only Barnyard, as Snort will startup without it but I'm unsure given that the issue initially began with signature errors which was a past problem pfSense version issue with Snort's rules.
its a barnyard issue with the config i think. I comment out line 2030 in the snort.inc file:
config gen_file: {$snortcfgdir}/gen-msg.map # config hostname: {$snortbarnyard_hostname_info} config interface: {$if_real}
then i added something like this to the advance config section in the gui
config hostname: pfsense_wan
i also noticed that I can only get barnyard2 to run on one interface… if I enabled it to 2 other interfaces, i get this error in the log shortly after it connects:
Apr 13 00:48:07 barnyard2[54841]: =============================================================================== Apr 13 00:48:07 barnyard2[54841]: database: Closing connection to database "snort" Apr 13 00:48:07 barnyard2[54841]: Barnyard2 exiting Apr 13 00:48:07 barnyard2[54841]: FATAL ERROR: database mysql_error: Duplicate entry '26455-2' for key 'PRIMARY' SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('18176','26455','2');] Apr 13 00:45:19 barnyard2[8131]: ===============================================================================
i haven't had a chance to research it yet
The "duplicate entry" error is a MySQL problem in the DB caused by the update to Barnyard2 to version 2.1.3. The sig_reference table has to be cleared. I have to be away for a few hours but will post a link with a fix when I get back.
Bill
-
Hello Bill,
Welcome Back!
A few questions/Comments in regards to the new Snort 2.9.6.0 pkg v3.0.6:
-
When I click the DNS resolv Icon, it loads the diag_dns.php page correctly, but when I click back, it sometimes displays "Confirm Form Resubmission" error? If I was on the LAN interface and view an alert it brings me back to the WAN interface in this scenario. (in Chrome)
-
I also noticed that when I rebooted pfSense and click on a Snort Interface to start it, it created duplicate pids per interface. At that time the IPrep alerts were not reporting but
other alerts were being alerted? On the Snort Interface page would it be beneficial to list the number of PIDS found? -
With the addition of the IPrep processor there are a lot more alerts in the Alerts Tab, maybe for a later release you might incorporate a "filter button" as you had suggested previously.
-
What is the order of the processors? Does IPrep block before or after other processors/Rules?
-
A nice feature would be to report the BlockList name in the Alert window? Maybe this could be on the Block Page?
A single Block List file like this where the comment sections gets displayed beside the alert?
172.33.42.32/16 # Emerging Threats
58.253.234.59/32 # IBlock BadPeer
etc …or multiple blocklist files and display the filename which is the Blocklist Name to display in the Block page?
- Adding the DisplayIPlistStats to the Gui to show memory used and total entries loaded would be a good feature.
Apart from that, I have the IPrep processor running on 4 boxes. I am using my custom shell script to download all of the blocklists (including the Iblock list which is in an IP range format that IPrep cant use directly) and saving the single blocklist file in /var/db/snort/iprep and running the pkill -HUP snort command to restart all interfaces in the background. Cron is running the script as per its own scheduler.
I do notice that the snort2c file is not getting cleared anymore? Not sure if thats a fix in 2.1.2 or with not using pfBlocker anymore? To confirm 100% will need to let it run for several days to see if this continues.
Thanks again for all the Hard work you do to maintain these projects. We need to start a Vote for the best pfSense package Maintainer!! ;)
-
-
This is a barnyard error. Are you using i386 or x64?
x64. This error appears limited to only Barnyard, as Snort will startup without it but I'm unsure given that the issue initially began with signature errors which was a past problem pfSense version issue with Snort's rules.
This version of the Snort package made some big changes to Barnyard2 like adding more output plugins. One side effect of that was it needed to reconfigure how the MySQL DB login info was stored. I wrote some code in the installation routines that attempts to migrate the old MySQL settings to the new format. It is possible some old settings could trip it up. Your error is a MySQL login error, and when the login fails Barnyard2 will stop. Go to the Barnyard tab in Snort and manually re-enter the login ID and password for the MySQL DB, then click Save. Try to start Barnyard2 after that on the Snort Interfaces tab.
Bill
Yeah, I'd believe that it was a caching thing or a transfer at the start but then I did this:
In desperation following a remove / install of Snort still facing the same issues, I cleanwiped the drive and reinstalled pfSense from scratch - that is to say, clean from a CD with no imports from a previous config file, everything reentered by hand - and the error remains the same. pfSense is determined to use root only root and nothing but root, and I'm not gonna let it; especially when everything says it should be using the proper user that for some reason it is ignoring.
Gonna go out on a limb and say that it isn't a transfer issue. It simply isn't reading the configuration I gave it - the files I checked in Barynard's conf and the conf.xml for pfSense, as well as the GUI, are all pointing to the correct login but pfSense is still attempting to login with root and getting refused. Reentering the information by hand has no effect.
-
Yeah, I'd believe that it was a caching thing or a transfer at the start but then I did this:
In desperation following a remove / install of Snort still facing the same issues, I cleanwiped the drive and reinstalled pfSense from scratch - that is to say, clean from a CD with no imports from a previous config file, everything reentered by hand - and the error remains the same. pfSense is determined to use root only root and nothing but root, and I'm not gonna let it; especially when everything says it should be using the proper user that for some reason it is ignoring.
Gonna go out on a limb and say that it isn't a transfer issue. It simply isn't reading the configuration I gave it - the files I checked in Barynard's conf and the conf.xml for pfSense, as well as the GUI, are all pointing to the correct login but pfSense is still attempting to login with root and getting refused. Reentering the information by hand has no effect.
This one has me puzzled. I have Barnyard2 running on my production system and several test VM setups with no issues (both i386 and AMD64 and 2.0.3 and 2.1.2 systems). Do you have the login ID and IP address entered into the MySQL DB for Barnyard2? On the MySQL DB server, can you login to the database with the same credentials you are using on the pfSense box? It really should not be using "root" at all as that is not coded anywhere in the source code for the package.
Bill
-
This is a barnyard error. Are you using i386 or x64?
x64. This error appears limited to only Barnyard, as Snort will startup without it but I'm unsure given that the issue initially began with signature errors which was a past problem pfSense version issue with Snort's rules.
its a barnyard issue with the config i think. I comment out line 2030 in the snort.inc file:
config gen_file: {$snortcfgdir}/gen-msg.map # config hostname: {$snortbarnyard_hostname_info} config interface: {$if_real}
then i added something like this to the advance config section in the gui
config hostname: pfsense_wan
i also noticed that I can only get barnyard2 to run on one interface… if I enabled it to 2 other interfaces, i get this error in the log shortly after it connects:
Apr 13 00:48:07 barnyard2[54841]: =============================================================================== Apr 13 00:48:07 barnyard2[54841]: database: Closing connection to database "snort" Apr 13 00:48:07 barnyard2[54841]: Barnyard2 exiting Apr 13 00:48:07 barnyard2[54841]: FATAL ERROR: database mysql_error: Duplicate entry '26455-2' for key 'PRIMARY' SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('18176','26455','2');] Apr 13 00:45:19 barnyard2[8131]: ===============================================================================
i haven't had a chance to research it yet
The "duplicate entry" error is a problem with Barnyard 2.1.3 and the fix is to clear out the entire sig_reference table. It will be repopulated when Barnyard2 starts up again.
Stop all Barnyard2 processes.
Login to the MySQL server and login to the database in question.
Issue the command: ```
DELETE FROM sig_reference;Restart a single Barnyard2 process first and let it completely start up. Now start any additional processes. Bill
-
I updated one of my pfSense systems to snort pkg v 3.0.6 last night. Logging in this morning, I saw this in Status: Dashboard:
Last config change Fri Apr 11 0:30:23 EEST 2014I definitely did not make any config changes at that time, nor was anybody logged in if system logs are to be believed. But this coincides with the time of Snort rules update. Coincidence? My other systems which are still on Snort pkg 3.0.4 (but same version of pfSense and other packages) don't seem to behave like this. One other thing that is different is that on this system Snort is configured to block offenders, while my other systems have this option turned off.
In the afternoon I checked Status: Dashboard again, and Last config change is now:
Fri Apr 11 12:31:01 EEST 2014Opened
https://redmine.pfsense.org/issues/3600(rejected)Looking at Diagnostics: Configuration History shows the following:
4/11/14 12:31:01 9.8 (system): made unknown change
4/11/14 00:30:23 9.8 (system): made unknown changeThe Snort package periodically writes configuration info to the config.xml file depending on what internal process may be going on. It does not "tag" its updates with a Snort identifier. That is something I can add as an enhancement in a later release.
What is happening in this specific instance is the daily update check for new rules is writing the "status" and "time" to the configuration file so it is persisted. This is new information that is displayed on the UPDATES tab. It shows the last time Snort checked for updated rules packages and what the result was: "success" or "fail". Since I have to make some other fixes anyway, I will add some tags to these updates so they don't say "…made unknown change..". I agree that can be an unnerving message… ;)
Bill
-
Read my reply just before your last post.
There's an issue with the IP reputation files when using ramdisk for /tmp and /var. The file gets nuked on reboot and Snort wont start again until a rules download has been made to redownload the file.
snort[45934]: FATAL ERROR: /usr/pbi/snort-amd64/etc/snort/snort_330_em0/snort.conf(398) => Unable to open address file /var/db/snort/iprep/emerging-compromised-ips.txt, Error: No such file or directory
Oops! Looks like I need to find another way to handle this for ramdisk users. I run regular images so I have no experience with the ramdisk setup. Got any suggestions for how handle this?
Bill
-
I cannot add manually suppress list.
(2.1.2-RELEASE (amd64), Snort2.9.6.0 pkg v3.0.6)
This bug is fixed and the updated code will be posted in a Pull Request in the next day or two.
Bill
-
It 's a change with the 3.0.5 package version, which has the new Snort binary also. It can now reload Snort config on the fly instead of restarting Snort completely when you change the rules settings etc.
Bill would have to correct me but with this new snort package, a new binary ver was used. Which I think uses a different rule set (probably why ET Web Client has an issue).
I wish the core team contacted Bill before approving the changes since he normally starts a new tread with each release since he started to maintain the package. Right now we're using the head-ups thread he started but I haven't seen Bill online for a week.
running on 2.1.2 i386
]/root(2): snort --v ,,_ -*> Snort! <*- o" )~ Version 2.9.6.0 GRE (Build 47) FreeBSD '''' By Martin Roesch & The Snort Team: http://www.snort.org/snort/snort-team Copyright (C) 2014 Cisco and/or its affiliates. All rights reserved. Copyright (C) 1998-2013 Sourcefire, Inc., et al. Using libpcap version 1.5.2 Using PCRE version: 8.34 2013-12-15 Using ZLIB version: 1.2.3
I was out of the country on vacation when the Pull Request was accepted. The timing was unfortunate. Normally the reviews have been taking a little time to complete, so I posted the Pull Request a couple of days before I left expecting it to take about the same length of time as some previous ones for approval. I was expecting to be back home well before the request was approved. This time the Core Developers approved it more quickly. That is good, but unfortunately I did not have time to post my normal RELEASE NOTES. I will catch up and do that when I post the fixes for the identified bugs in the next day or so.
Bill
-
he "duplicate entry" error is a problem with Barnyard 2.1.3 and the fix is to clear out the entire sig_reference table. It will be repopulated when Barnyard2 starts up again.
Stop all Barnyard2 processes.
Login to the MySQL server and login to the database in question.
Issue the command:
Code: [Select]
DELETE FROM sig_reference;
Restart a single Barnyard2 process first and let it completely start up.
Now start any additional processes.Bill
This work for few moments and go down…
-
I keep seeing warnings like this by barnayard2 in the system log:
Apr 15 07:46:22 barnyard2[30475]: WARNING database [Database()]: End of failed transaction block Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [4] Failed Query Body [INSERT INTO data (sid,cid,data_payload) VALUES (1,1181,'4873010000010000000000010331783102637A0000FF0001000029FFFF000000000000');] Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [3] Failed Query Body [INSERT INTO iphdr (sid, cid, ip_src, ip_dst, ip_ver, ip_hlen, ip_tos, ip_len, ip_id, ip_flags, ip_off,ip_ttl, ip_proto, ip_csum) VALUES (1,1181,1347571305,1356855242,4,5,40,63,0,0,0,55,17,23585);] Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [2] Failed Query Body [INSERT INTO udphdr (sid, cid, udp_sport, udp_dport, udp_len, udp_csum) VALUES (1, 1181, 34104, 53, 43, 10122);] Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [1] Failed Query Body [INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 1181, 477, '2014-04-15 07:46:16');] Apr 15 07:46:22 barnyard2[30475]: WARNING database: [Database()] Failed transaction with current query transaction Apr 15 07:46:22 barnyard2[30475]: [Database()]: Insertion of Query [INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 1181, 477, '2014-04-15 07:46:16');] failed
I checked the database and this particular event was successfully inserted into the database.
I've also spotted a few instances where classification_id and user_id is NULL.(This is actually filled by Snorby, not Barnyard2) -
bmeeks: ki can you please add to only drop packet, and not to block or/and add 1 min block time?
-
bmeeks: ki can you please add to only drop packet, and not to block or/and add 1 min block time?
Having Snort to drop packages is currently not possible as Snort doesn't work in a true inline mode.
-
bmeeks: ki can you please add to only drop packet, and not to block or/and add 1 min block time?
As fragged says, Snort cannot currently operate in inline mode on pfSense. And even if it did, the packet throughput would be adversely impacted compared to the current libpcap arrangement. This is because network traffic would have to be context-switched from kernel space to user space, inspected, and then either dropped or sent back down from user space to kernel space to continue through the network stack. This context switching is not very efficient. One of the pfSense Core Developers told me that with true inline mode using Snort or Suricata the top throughput before lag became a serious issue was in the 50-100 megabit/sec range (depending on the hardware used). For some folks that would be OK, but for many others that level of throughput would not be sufficient.
I am planning to try and offer inline mode as an option, but with the big caveat that throughput will be limited in that mode. Unfortunately true inline mode also requires some changes to the pfSense kernel, so that means it can't happen with the current releases.
Bill
-
he "duplicate entry" error is a problem with Barnyard 2.1.3 and the fix is to clear out the entire sig_reference table. It will be repopulated when Barnyard2 starts up again.
Stop all Barnyard2 processes.
Login to the MySQL server and login to the database in question.
Issue the command:
Code: [Select]
DELETE FROM sig_reference;
Restart a single Barnyard2 process first and let it completely start up.
Now start any additional processes.Bill
This work for few moments and go down…
There are lots of complaints on the Google lists about these kinds of Barnyard2 errors. I'm not a Barnyard2 expert, but apparently some significant database changes were made with the last Barnyard2 update (which Snort and Suricata are now using on pfSense). These changes result in these kinds of errors. I will research some more to see if I can come up with a solution. In the meantime, do some Google searches on those errors and you should come across several solutions posted by the Barnyard2 developers. The solutions involve running some complicated SQL scripts directly in the database.
I posted some links a couple of months ago to an excellent set of scripts that fixed the problem for me. I will see if I can find my old post and update this thread with it. The root cause here is having old Barnyard2 tables and data in the MySQL database but using the new Barnyard2 binary.
I think this also is a problem when multiple Barnyard2 processes are sharing the same DB. I don't know what the Barnyard2 developers did to cause this, but I agree it is irritating.
Bill