Snort update coming soon – please read about an important change!
-
he "duplicate entry" error is a problem with Barnyard 2.1.3 and the fix is to clear out the entire sig_reference table. It will be repopulated when Barnyard2 starts up again.
Stop all Barnyard2 processes.
Login to the MySQL server and login to the database in question.
Issue the command:
Code: [Select]
DELETE FROM sig_reference;
Restart a single Barnyard2 process first and let it completely start up.
Now start any additional processes.Bill
This work for few moments and go down…
-
I keep seeing warnings like this by barnayard2 in the system log:
Apr 15 07:46:22 barnyard2[30475]: WARNING database [Database()]: End of failed transaction block Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [4] Failed Query Body [INSERT INTO data (sid,cid,data_payload) VALUES (1,1181,'4873010000010000000000010331783102637A0000FF0001000029FFFF000000000000');] Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [3] Failed Query Body [INSERT INTO iphdr (sid, cid, ip_src, ip_dst, ip_ver, ip_hlen, ip_tos, ip_len, ip_id, ip_flags, ip_off,ip_ttl, ip_proto, ip_csum) VALUES (1,1181,1347571305,1356855242,4,5,40,63,0,0,0,55,17,23585);] Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [2] Failed Query Body [INSERT INTO udphdr (sid, cid, udp_sport, udp_dport, udp_len, udp_csum) VALUES (1, 1181, 34104, 53, 43, 10122);] Apr 15 07:46:22 barnyard2[30475]: WARNING database: Failed Query Position [1] Failed Query Body [INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 1181, 477, '2014-04-15 07:46:16');] Apr 15 07:46:22 barnyard2[30475]: WARNING database: [Database()] Failed transaction with current query transaction Apr 15 07:46:22 barnyard2[30475]: [Database()]: Insertion of Query [INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 1181, 477, '2014-04-15 07:46:16');] failedI checked the database and this particular event was successfully inserted into the database.
I've also spotted a few instances where classification_id and user_id is NULL.(This is actually filled by Snorby, not Barnyard2) -
bmeeks: ki can you please add to only drop packet, and not to block or/and add 1 min block time?
-
bmeeks: ki can you please add to only drop packet, and not to block or/and add 1 min block time?
Having Snort to drop packages is currently not possible as Snort doesn't work in a true inline mode.
-
bmeeks: ki can you please add to only drop packet, and not to block or/and add 1 min block time?
As fragged says, Snort cannot currently operate in inline mode on pfSense. And even if it did, the packet throughput would be adversely impacted compared to the current libpcap arrangement. This is because network traffic would have to be context-switched from kernel space to user space, inspected, and then either dropped or sent back down from user space to kernel space to continue through the network stack. This context switching is not very efficient. One of the pfSense Core Developers told me that with true inline mode using Snort or Suricata the top throughput before lag became a serious issue was in the 50-100 megabit/sec range (depending on the hardware used). For some folks that would be OK, but for many others that level of throughput would not be sufficient.
I am planning to try and offer inline mode as an option, but with the big caveat that throughput will be limited in that mode. Unfortunately true inline mode also requires some changes to the pfSense kernel, so that means it can't happen with the current releases.
Bill
-
he "duplicate entry" error is a problem with Barnyard 2.1.3 and the fix is to clear out the entire sig_reference table. It will be repopulated when Barnyard2 starts up again.
Stop all Barnyard2 processes.
Login to the MySQL server and login to the database in question.
Issue the command:
Code: [Select]
DELETE FROM sig_reference;
Restart a single Barnyard2 process first and let it completely start up.
Now start any additional processes.Bill
This work for few moments and go down…
There are lots of complaints on the Google lists about these kinds of Barnyard2 errors. I'm not a Barnyard2 expert, but apparently some significant database changes were made with the last Barnyard2 update (which Snort and Suricata are now using on pfSense). These changes result in these kinds of errors. I will research some more to see if I can come up with a solution. In the meantime, do some Google searches on those errors and you should come across several solutions posted by the Barnyard2 developers. The solutions involve running some complicated SQL scripts directly in the database.
I posted some links a couple of months ago to an excellent set of scripts that fixed the problem for me. I will see if I can find my old post and update this thread with it. The root cause here is having old Barnyard2 tables and data in the MySQL database but using the new Barnyard2 binary.
I think this also is a problem when multiple Barnyard2 processes are sharing the same DB. I don't know what the Barnyard2 developers did to cause this, but I agree it is irritating.
Bill
-
So, having issues with Snort again. Been running rather well up until the 2.1.2 update. Now it's back to not working at all.
At first the issue was the same as before, that being the signature errors I mentioned in a previous thread. So I used truncate, that helped but didn't fix, so I did what I did last time to correct it and wiped the DB clean again. Now the communication begins again, everything looks clean, I reboot for a sanity check and….
Apr 13 01:13:56 barnyard2[84684]: Barnyard2 exiting Apr 13 01:13:56 barnyard2[84684]: database mysql_error: Access denied for user 'root'@'domain.com' (using password: YES) Apr 13 01:13:56 barnyard2[84684]: Writing PID "84684" to file "/var/run/barnyard2_em055759.pid" Apr 13 01:13:56 barnyard2[84684]: PID path stat checked out ok, PID path set to /var/run Apr 13 01:13:56 barnyard2[84684]: Daemon initialized, signaled parent pid: 84444 Apr 13 01:13:56 barnyard2[84444]: Daemon parent exiting Apr 13 01:13:56 barnyard2[84444]: Initializing daemon modeThis happens.
Okay, whiskey-tango-foxtrot. Why you use root, pfSense? Root is not a configured user. Anywhere. Seems like a really simple error, but the error has me table bashing my skull as I can't figure out where the directive is coming from. I won't allow root to connect to my MySQL production server unless I'm physically needing it. So that's why it isn't working now. However, pfSense is absolutely determined that it will never connect under anything else. I have checked manually the files ( both Snort's pbi barnyard config files and pfSense overall config ), and they match what is currently shown in the webgui ( that is to say, with the normal dbuser configured for the Snort db ).
In desperation following a remove / install of Snort still facing the same issues, I cleanwiped the drive and reinstalled pfSense from scratch - that is to say, clean from a CD with no imports from a previous config file, everything reentered by hand - and the error remains the same. pfSense is determined to use root only root and nothing but root, and I'm not gonna let it; especially when everything says it should be using the proper user that for some reason it is ignoring.
I had this exact issue (along with continuing "Duplicate Entry" issues), but to fix it, I removed the "Sensor Name" in the Barnyard2 portion of the web gui. This corrected barnyard2 logging in as root. I was able to repeat the issue and correct it by adding a name to the "Sensor Name" - broke, and then removing it.. fixed.
-
he "duplicate entry" error is a problem with Barnyard 2.1.3 and the fix is to clear out the entire sig_reference table. It will be repopulated when Barnyard2 starts up again.
Stop all Barnyard2 processes.
Login to the MySQL server and login to the database in question.
Issue the command:
Code: [Select]
DELETE FROM sig_reference;
Restart a single Barnyard2 process first and let it completely start up.
Now start any additional processes.Bill
This work for few moments and go down…
There are lots of complaints on the Google lists about these kinds of Barnyard2 errors. I'm not a Barnyard2 expert, but apparently some significant database changes were made with the last Barnyard2 update (which Snort and Suricata are now using on pfSense). These changes result in these kinds of errors. I will research some more to see if I can come up with a solution. In the meantime, do some Google searches on those errors and you should come across several solutions posted by the Barnyard2 developers. The solutions involve running some complicated SQL scripts directly in the database.
I posted some links a couple of months ago to an excellent set of scripts that fixed the problem for me. I will see if I can find my old post and update this thread with it. The root cause here is having old Barnyard2 tables and data in the MySQL database but using the new Barnyard2 binary.
I think this also is a problem when multiple Barnyard2 processes are sharing the same DB. I don't know what the Barnyard2 developers did to cause this, but I agree it is irritating.
Bill
Is there any fix for the duplicate entry mysql issue with Barnyard2? I have started with a fresh Snorby install and DB, and after 5 minutes barnyard2 stops with the duplicate entry. I have cleared the sig_reference table and still get the same issue.
-
Is there any fix for the duplicate entry mysql issue with Barnyard2? I have started with a fresh Snorby install and DB, and after 5 minutes barnyard2 stops with the duplicate entry. I have cleared the sig_reference table and still get the same issue.
I posted a new thread last evening containing links to two potential solutions. Those worked for me on my Snorby install, but admittedly I only have a home network's traffic being logged so not very many events at all. Here is link to that thread: https://forum.pfsense.org/index.php?topic=75357.0
Also, I'm a bit confused by your two successive posts. The first one at 8:51 AM says you fixed the problem by removing the sensor name, but then your newer one at 8:55 AM implies the problem still exists. What underlying DB engine is your MySQL install using? The Barnyard2 developers say it must be InnoDB.
Bill
-
Is there any fix for the duplicate entry mysql issue with Barnyard2? I have started with a fresh Snorby install and DB, and after 5 minutes barnyard2 stops with the duplicate entry. I have cleared the sig_reference table and still get the same issue.
I posted a new thread last evening containing links to two potential solutions. Those worked for me on my Snorby install, but admittedly I only have a home network's traffic being logged so not very many events at all. Here is link to that thread: https://forum.pfsense.org/index.php?topic=75357.0
Also, I'm a bit confused by your two successive posts. The first one at 8:51 AM says you fixed the problem by removing the sensor name, but then your newer one at 8:55 AM implies the problem still exists. What underlying DB engine is your MySQL install using? The Barnyard2 developers say it must be InnoDB.
Bill
Bill - thanks.. I am trying your fixes now.. I have a production firewall/IDS in place that I can generate a ton of events from so I can put it to a good test in a few..
Im using InnoDB with the mysql install - it's actually a fresh Security Onion install that I am logging to.
The issue I fixed with removing the sensor name was not the duplicate entry problem. It was the problem of Barnyard2 wanting to log into mysql with the "root" user instead of whatever user you put in as the mysql username in the barnyard2 tab. If I put in a sensor name (versus leaving it blank) then barnyard always tries to login as 'root'. If I leave the sensor name blank, it logs in properly. It does this with multiple sensors.
-
Is there any fix for the duplicate entry mysql issue with Barnyard2? I have started with a fresh Snorby install and DB, and after 5 minutes barnyard2 stops with the duplicate entry. I have cleared the sig_reference table and still get the same issue.
I posted a new thread last evening containing links to two potential solutions. Those worked for me on my Snorby install, but admittedly I only have a home network's traffic being logged so not very many events at all. Here is link to that thread: https://forum.pfsense.org/index.php?topic=75357.0
Also, I'm a bit confused by your two successive posts. The first one at 8:51 AM says you fixed the problem by removing the sensor name, but then your newer one at 8:55 AM implies the problem still exists. What underlying DB engine is your MySQL install using? The Barnyard2 developers say it must be InnoDB.
Bill
Bill - thanks.. I am trying your fixes now.. I have a production firewall/IDS in place that I can generate a ton of events from so I can put it to a good test in a few..
Im using InnoDB with the mysql install - it's actually a fresh Security Onion install that I am logging to.
The issue I fixed with removing the sensor name was not the duplicate entry problem. It was the problem of Barnyard2 wanting to log into mysql with the "root" user instead of whatever user you put in as the mysql username in the barnyard2 tab. If I put in a sensor name (versus leaving it blank) then barnyard always tries to login as 'root'. If I leave the sensor name blank, it logs in properly. It does this with multiple sensors.
Thanks for the feedback on the sensor name anomaly. I will look into that. Could be I have a mistake in the code someplace. I know I never coded "root" by design, but I do have fat fingers when typing… :(
Bill
-
Is there any fix for the duplicate entry mysql issue with Barnyard2? I have started with a fresh Snorby install and DB, and after 5 minutes barnyard2 stops with the duplicate entry. I have cleared the sig_reference table and still get the same issue.
I posted a new thread last evening containing links to two potential solutions. Those worked for me on my Snorby install, but admittedly I only have a home network's traffic being logged so not very many events at all. Here is link to that thread: https://forum.pfsense.org/index.php?topic=75357.0
Also, I'm a bit confused by your two successive posts. The first one at 8:51 AM says you fixed the problem by removing the sensor name, but then your newer one at 8:55 AM implies the problem still exists. What underlying DB engine is your MySQL install using? The Barnyard2 developers say it must be InnoDB.
Bill
Bill - thanks.. I am trying your fixes now.. I have a production firewall/IDS in place that I can generate a ton of events from so I can put it to a good test in a few..
Im using InnoDB with the mysql install - it's actually a fresh Security Onion install that I am logging to.
The issue I fixed with removing the sensor name was not the duplicate entry problem. It was the problem of Barnyard2 wanting to log into mysql with the "root" user instead of whatever user you put in as the mysql username in the barnyard2 tab. If I put in a sensor name (versus leaving it blank) then barnyard always tries to login as 'root'. If I leave the sensor name blank, it logs in properly. It does this with multiple sensors.
Thanks for the feedback on the sensor name anomaly. I will look into that. Could be I have a mistake in the code someplace. I know I never coded "root" by design, but I do have fat fingers when typing… :(
Bill
Anytime, and anything I can do to help…
The first link you posted fixed the duplicate entry issue - BUT - although I now see events in Snorby, the dashboard is not showing any events, but when I look at events or sensors is shows all the events (I have a few thousand events now).. any ideas?
-
Hi gscasny,
Did you do a full Security Onion install or is this a Virtual Machine ISO boot?
Would be interested to know the steps you took to get pfSense to SO working?
Are the alerts visible in SGUIL or just Snorby?
Appreciate any info you can share.
Thanks in advance.
-
@BBcan17:
Hi gscasny,
Did you do a full Security Onion install or is this a Virtual Machine ISO boot?
Would be interested to know the steps you took to get pfSense to SO working?
Are the alerts visible in SGUIL or just Snorby?
Appreciate any info you can share.
Thanks in advance.
It's a full install. It's a "Server" install from the "Advanced" install as I run Snort on our edge FW's as sensors. I am logging only to Snorby at the moment, as SGUIL is on my list after OSSEC as we currently use OSSEC at client sites but not via Sec onion…. it bums me a little that SGUIL X windows application only... But I am happy to share any/all info I find out..
-
S.O. is great. I have OSSEC running on all of my Windows and Linux servers (Works really slick) Let me know if you need any help with that.
I run Snort on our edge FW's as sensors.
Are these SO sensors or pfSense snort?
I'm not sure what you mean by "SGUIL X windows application only"?
-
@BBcan17:
S.O. is great. I have OSSEC running on all of my Windows and Linux servers (Works really slick) Let me know if you need any help with that.
I run Snort on our edge FW's as sensors.
Are these SO sensors or pfSense snort?
I'm not sure what you mean by "SGUIL X windows application only"?
The sensors are our build of pfsense.
SGUIL X win only - meaning I have to run the interface in X, instead of via a web interface like Snorby or ELSA (which I don't use..we use a SIEM product for log storage and correlation)..
I really like OSSEC as a host ids.. we have a few installs, but I am looking to standardize the deployment and monitoring of OSSEC along with our IDS sensors and SIEM…
-
ok. I am using Xrdp for Sguil. Another option is just to load a VM from the live ISO cd and map the ports from there?
pfSense
What did you use as the Database name and credentials? I thought that SO uses "root" without a password?
Enable Barnyard
Add Dump payload ?
Sensor name = pfsenseEnable MySQL
hostname = x.x.x.x
Database name =
Database User =
Database password =Advanced Settings ?
S.O.
In Security Onion, what port do you need to open up in UFW? 8000?
I understood that SO only listens on the local host only?
Can you detail what you changed in SO?
This is my /etc/nsm/<sensor>/barnyard2.conf file
config logdir: /nsm/sensor_data/ <sensor>config classification_file: /etc/nsm/<sensor>/classification.config
config reference_file: /etc/nsm/<sensor>/reference.config
config sid_file: /etc/nsm/<sensor>/sid-msg.map
config gen_file: /etc/nsm/<sensor>/gen-msg.map
config hostname: <sensor>config interface: eth1
input unified2
output sguil: sensor_name= <sensor>agent_port=8000
output database: alert, mysql, user=root dbname=snorby host=127.0.0.1
output alert_syslog: LOG_LOCAL6 LOG_ALERTI predominantly use SGUIL as I think its a total Workhorse of a tool. Ver 0.9.0 was recently released a few weeks ago. I have the Server setup as a single sensor installation and one remote sensor. Wouldn't the pfense Barnyard2 info overlap into the existing sensors info?
Wouldn't it be better to setup a separate sensor for pfsense?
As I don't use Snorby, I wonder if I could disable events from SO and use that for pfSense? Getting it all into SGUIL would ultimately be the best setup. Wonder if SO iptables could forward the pfsense incoming port to the local host of SO? I dont think I would want to re-compile the SO Barnyard2 application to listen on its actual interface.</sensor></sensor></sensor></sensor></sensor></sensor></sensor></sensor> -
@BBcan17:
ok. I am using Xrdp for Sguil. Another option is just to load a VM from the live ISO cd and map the ports from there?
pfSense
What did you use as the Database name and credentials? I thought that SO uses "root" without a password?
Enable Barnyard
Add Dump payload ?
Sensor name = pfsenseEnable MySQL
hostname = x.x.x.x
Database name =
Database User =
Database password =Advanced Settings ?
S.O.
In Security Onion, what port do you need to open up in UFW? 8000?
I understood that SO only listens on the local host only?
Can you detail what you changed in SO?
This is my /etc/nsm/<sensor>/barnyard2.conf file
config logdir: /nsm/sensor_data/ <sensor>config classification_file: /etc/nsm/<sensor>/classification.config
config reference_file: /etc/nsm/<sensor>/reference.config
config sid_file: /etc/nsm/<sensor>/sid-msg.map
config gen_file: /etc/nsm/<sensor>/gen-msg.map
config hostname: <sensor>config interface: eth1
input unified2
output sguil: sensor_name= <sensor>agent_port=8000
output database: alert, mysql, user=root dbname=snorby host=127.0.0.1
output alert_syslog: LOG_LOCAL6 LOG_ALERTI predominantly use SGUIL as I think its a total Workhorse of a tool. Ver 0.9.0 was recently released a few weeks ago. I have the Server setup as a single sensor installation and one remote sensor. Wouldn't the pfense Barnyard2 info overlap into the existing sensors info?
Wouldn't it be better to setup a separate sensor for pfsense?
As I don't use Snorby, I wonder if I could disable events from SO and use that for pfSense? Getting it all into SGUIL would ultimately be the best setup. Wonder if SO iptables could forward the pfsense incoming port to the local host of SO? I dont think I would want to re-compile the SO Barnyard2 application to listen on its actual interface.</sensor></sensor></sensor></sensor></sensor></sensor></sensor></sensor>Ok - Ill start from the top and work my way down :)
xrdp is ok if you are local, but we support alot of customers all over the country and alot don't have the fastest inet connections, so tunneling x win over their links isn't feasible. :)
As far as database credentials, SO uses root with no password for local only logins… you can't login a remote sensor with root, nor would you want to, so we create a user per sensor - i.e. location-fw-interface in mysql and only give them privs on the snorby DB.
after that we open port 3306 (mysql) ONLY from the sensor to the security onion box (we delete all other ufw rules except port 444 for snorby), I don't like that SO allows connections from anywhere.
You also have to go into /etc/mysql/my.conf and change the 'bind address' to 0.0.0.0 and then do a 'service restart mysql' - it defaults to 127.0.0.1 which will not allow your remote sensors to connect.
We run our main sensors at the perimeter via our build of PFsense as we use the block function of snort on the WAN and LAN via a cutomized ET Pro ruleset and we use pfblocker with several customized ip block lists.. but since snort runs in userspace, we get reports on the blocked ip's via snort. It's important to us to catch, log an block any attacks at the perimeter, which Pfsense is amazingly efficient at.
We dont run any sensors on SO at the moment..though I would not mind testing Bro in conjunction with snort as it doesn't use signatures from what we have read.
The biggest thing for us is to be able to monitor and alerts exception (high severity) alerts as we monitor alot of networks (hundreds) along with our SIEM to keep any breach attempts to a minimum.
-
Ok - Ill start from the top and work my way down :)
xrdp is ok if you are local, but we support alot of customers all over the country and alot don't have the fastest inet connections, so tunneling x win over their links isn't feasible. :)
As far as database credentials, SO uses root with no password for local only logins… you can't login a remote sensor with root, nor would you want to, so we create a user per sensor - i.e. location-fw-interface in mysql and only give them privs on the snorby DB.
I found this link http://community.spiceworks.com/topic/466735-security-onion-and-pfsense
Does this look correct?
===========================
Open the terminal run the following commands
This will allow your pfsense to connect through Security Onions firewall to mysql
sudo ufw allow proto tcp from xx.xx.xx.xx/32 to any port 3306
mysql
This should open the prompt for mysql, run the following commands hereshow databases;
Make sure the databasesnorbyexists.This part will setup a user for Barnyard2 to use
create user 'sensorname'@'xx.xx.xx.xx' IDENTIFIED BY 'SENSORPASSWORD'
grant all privileges on snorby. to 'sensorname'@'xx.xx.xx.xx' with grant option;*
flush privileges;
exit
You should be out of the mysql prompt now, run these commands in terminalThis will allow any device to connect to mysql as MySQL will be listening for connections on any IP instead of just the loopback address (assuming its allowed through Security Onions firewall)
sudo vi /etc/mysql/my.cnf
You should have the mysql config open, use the down arrows to move down until you see the part that says
bind-address
Move the cursor (The one in the terminal window that indicates where you're typing) to where it says "= 127.0.0.1" and type 'i' (Without the ' )
It should say "– INSERT --" at the the bottom of the terminal window
Delete 127.0.0.1 and replace it with 0.0.0.0
Press 'esc' so that "– INSERT --" is gone, then type ':wq!' and hit return (Without the ' but with the Colon )
This should save and exit viNow run the command
sudo mysql restart
Wait for the MySQL Service to restartSetup Barnyard2 on pfSense in Snort
In web interface for pfSense goto Services > Snort, Under Snort Interfaces Click the
Editbutton and open the Barnyard2 tabChange and input the following settings in the
Log to MySQL Databasefieldoutput database: alert, mysql, dbname=snorby user=sensorname host=xx.xx.xx.yy password=SENSORPASSWORD
Save
goto Services > Snort > Snort Interfaces
Click the start Icon next under the Barnyard 2 Column, wait for it to go green.===========================
We run our main sensors at the perimeter via our build of PFsense as we use the block function of snort on the WAN and LAN via a cutomized ET Pro ruleset and we use pfblocker with several customized ip block lists.. but since snort runs in userspace, we get reports on the blocked ip's via snort. It's important to us to catch, log an block any attacks at the perimeter, which Pfsense is amazingly efficient at.
As SO is using "SALT", it would be good to try to incorporate that into the mix to mange the rules update process. But I guess as pfSense Snort is not using Pulled Pork that might not work? Salt is on the Freebsd ports.
Snort now has the IPrep processor which could replace pfBlocker functionality. The current pfsense integration doesn't reload the blocklists automatically when they are updated so will need to wait for the next release to see if that is fixed.
We don't run any sensors on SO at the moment..though I would not mind testing Bro in conjunction with snort as it doesn't use signatures from what we have read.
The biggest thing for us is to be able to monitor and alerts exception (high severity) alerts as we monitor alot of networks (hundreds) along with our SIEM to keep any breach attempts to a minimum.
Will changing mysql to 0.0.0.0 affect SO at all?
Getting the pfSense snort into Sguil would make this a whole lot better. The BRO integration would be much better.
I need to see if I can get Snorby to only listen to the pfSense boxes and not the local sensor installations.
Does pfSense push the full pcaps of the alert to SO Snorby? or just the alert?
Thanks for your help.
-
Anytime, and anything I can do to help…
The first link you posted fixed the duplicate entry issue - BUT - although I now see events in Snorby, the dashboard is not showing any events, but when I look at events or sensors is shows all the events (I have a few thousand events now).. any ideas?
I have seen other posts on Google relative to Snorby and the disappearing of the Dashboard events. It's something with the background cache jobs if I recall. I had that problem once several months ago, and some protracted Google searches led me to the fix. Sorry, but I don't remember exactly what fixed it or I would readily share.
Bill
