Barnyard2 stops running
-
Hi!
I'm running snort (and barnyard) on two interfaces, my WAN and LAN. After having it run for a few minutes, it seems like the WAN interface stops barnyard with the following error:
Nov 25 07:16:49 pfSense barnyard2[87750]: FATAL ERROR: database mysql_error: Duplicate entry '23697-2' for key 'PRIMARY' SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('22298','23697','2');]
I did a little research and it looks like it is a common problem, and I'm not exactly sure if there's a fix for it. The thread I was reading through said that you should check the "Disable Signature Reference Table" box on one interface but leave it unchecked for the other. On the "LAN" interface, it's checked; on the WAN interface, it's unchecked. And the WAN interface is the one that barnyard2 stops running. Some people also said to make sure you have two instances of snort and barnyard running, which I do.
[2.1.5-RELEASE][root@pfSense.mwiz.org]/var/log(4): ps auxwww | grep -i barnyard root 5989 61.0 4.0 54772 40900 ?? Rs 7:44AM 0:06.19 /usr/local/bin/barnyard2 -r 16083 -f snort_16083_em0.u2 --pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-amd64/etc/snort/snort_16083_em0/barnyard2.conf -d /var/log/snort/snort_em016083 -D -q root 35353 0.0 6.2 91636 63480 ?? SN 12:12AM 1:26.03 /usr/local/bin/barnyard2 -r 30562 -f snort_30562_em1.u2 --pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-amd64/etc/snort/snort_30562_em1/barnyard2.conf -d /var/log/snort/snort_em130562 -D -q [2.1.5-RELEASE][root@pfSense.mwiz.org]/var/log(5): ps auxwww | grep -i snort root 5989 78.0 5.3 69108 54792 ?? Rs 7:44AM 0:15.26 /usr/local/bin/barnyard2 -r 16083 -f snort_16083_em0.u2 --pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-amd64/etc/snort/snort_16083_em0/barnyard2.conf -d /var/log/snort/snort_em016083 -D -q root 34694 0.0 46.2 1257452 473444 ?? SNs 12:12AM 1:54.60 /usr/pbi/snort-amd64/bin/snort -R 16083 -D -q -l /var/log/snort/snort_em016083 --pid-path /var/run --nolock-pidfile -G 16083 -c /usr/pbi/snort-amd64/etc/snort/snort_16083_em0/snort.conf -i em0 root 35159 0.0 17.5 591852 179692 ?? SNs 12:12AM 0:29.78 /usr/pbi/snort-amd64/bin/snort -R 30562 -D -q -l /var/log/snort/snort_em130562 --pid-path /var/run --nolock-pidfile -G 30562 -c /usr/pbi/snort-amd64/etc/snort/snort_30562_em1/snort.conf -i em1 root 35353 0.0 6.2 91636 63480 ?? SN 12:12AM 1:26.03 /usr/local/bin/barnyard2 -r 30562 -f snort_30562_em1.u2 --pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-amd64/etc/snort/snort_30562_em1/barnyard2.conf -d /var/log/snort/snort_em130562 -D -q [2.1.5-RELEASE][root@pfSense.mwiz.org]/var/log(6):
I was wondering if there's something that I can do to fix this problem.
EDIT: I am also using Snort 2.9.6.2 pkg v3.1.5.
-
Barnyard2 has a big problem when more than one instance of barnyard is updating the same MySQL database. This occurred with some new code that went in to one of the latest Barnyard2 upstream updates. You will find many, many posts if you search Google long enough. Many folks have this problem whether they are using pfSense or not.
Here is the best way to fix it in my opinion. I have done this and it stopped all my problems.
Click the "disable signature reference table" block on all of your Snort instances. On your MySQL DB server, install Barnyard2 and PulledPork (either compile from source or get the appropriate packages for your OS distribution). Configure that Barnyard2 instance as the only one to update the Signature Reference Table. Configure PulledPork to pull the same vendor rules you use for Snort on pfSense. Create cron tasks on your MySQL box to run PulledPork, then a script to login to the DB and "delete all the records in the signature reference table", and then run Barnyard2 immediately afterwards.
I did this about 2 months ago and have not had another single problem with Barnyard2 on my firewall.
I got this idea from a post by one of the principals from the Security Onion team and one of the Barnyard2 developers.
I can't explain the problem very well in a short sentence, but it seems that upon startup (and only on startup) each Barnyard2 process will read the sid-msg.map file and attempt to insert all the references from each signature into the signature reference table in the DB. Small changes in the layout of the sid-msg.map file will trip up this process. For example, if a new reference is added to an existing rule as part of a daily rule update, and that reference is inserted between two existing references instead of simply being appended to the end, then Barnyard2 gets confused and creates a duplicate key when creating the primary key used to insert that new reference. This duplicate key problem can also happen when more than one Barnyard2 process attempts to update the same signature reference table.
So the bottom line is that even with only a single BY2 process updating the DB, you can still occasionally get the duplicate key error if a reference gets updated in a rule. That's why it is best to just clear out the entire table and repopulate it each time you fetch new rules (and thus generate a new sig-msg.map file). The signature reference table means nothing directly to Snort or Barnyard2. Some SIEM tools use the data from that table in their displays.
Bill
-
Thank you very much for the response. I've seen a number of issues related to this online, and didn't really know of a good fix for it. What I ended up doing last night was remove barnyard2 from pfsense and stopped all the snort instances. Then, removed all the snorby data with the rake(?) command and reinstall snorby. Ran some MySQL commands that created a procedure to fix the barnyard database/tables (found here on pfsense's forums). Go back into pfsense's WAN snort interface, enable barnyard2 and kept the "disable signature reference table" unchecked so that it updated the table. Started the WAN snort instance and the barnyard2 with that interface, and let it populate snorby for about an hour. Came back, enabled barnyard on LAN interface, checked the "disable sig ref table" so it wouldn't update it, enabled the interface and barnyard, and run with it.
So far, it seems to be working OK. I had a weird thing happen today when I was disabling rules where the LAN interface wouldn't start back up, but I'm not sure if it's related to barnyard or something else.
Either way, it sounds like if I run into this problem again, I should follow the steps below. Thanks for doing this and giving me the suggestion. I'm sure it will happen again and at least now I know what I can do to fix it.
Thank you!
-
Bill,
Do you have a reference on how you setup barnyard2 and pulledpork? I got it setup and running, but Snorby can't read the events. For example, when I look at the events in snorby, I see…
Snort Alert [1:2010935:2]
In my signature table, I see this…
select * from signature where sig_sid=2402001 -> ; +--------+--------------+---------------------------------------------+--------------+---------+---------+---------+--------------+ | sig_id | sig_class_id | sig_name | sig_priority | sig_rev | sig_sid | sig_gid | events_count | +--------+--------------+---------------------------------------------+--------------+---------+---------+---------+--------------+ | 2680 | 14 | ET DROP Dshield Block Listed Source group 1 | 2 | 3547 | 2402001 | 1 | 0 | | 10828 | 14 | ET DROP Dshield Block Listed Source group 1 | 2 | 3547 | 2402001 | 1 | 0 | | 47053 | 30 | Snort Alert [1:2402001:3544] | 2 | 3544 | 2402001 | 1 | 3 | | 47079 | 30 | Snort Alert [1:2402001:3544] | 2 | 3545 | 2402001 | 1 | 11 | | 47122 | 30 | Snort Alert [1:2402001:3544] | 2 | 3546 | 2402001 | 1 | 4 | | 47142 | 30 | Snort Alert [1:2402001:3544] | 2 | 3547 | 2402001 | 1 | 1 | +--------+--------------+---------------------------------------------+--------------+---------+---------+---------+--------------+ 6 rows in set (0.05 sec)
So I know things are getting put in there, but it seems like it's not using the information barnyard2 put in the db.
Thanks in advanced.
-
Bill,
Do you have a reference on how you setup barnyard2 and pulledpork? I got it setup and running, but Snorby can't read the events. For example, when I look at the events in snorby, I see…
Snort Alert [1:2010935:2]
In my signature table, I see this…
select * from signature where sig_sid=2402001 -> ; +--------+--------------+---------------------------------------------+--------------+---------+---------+---------+--------------+ | sig_id | sig_class_id | sig_name | sig_priority | sig_rev | sig_sid | sig_gid | events_count | +--------+--------------+---------------------------------------------+--------------+---------+---------+---------+--------------+ | 2680 | 14 | ET DROP Dshield Block Listed Source group 1 | 2 | 3547 | 2402001 | 1 | 0 | | 10828 | 14 | ET DROP Dshield Block Listed Source group 1 | 2 | 3547 | 2402001 | 1 | 0 | | 47053 | 30 | Snort Alert [1:2402001:3544] | 2 | 3544 | 2402001 | 1 | 3 | | 47079 | 30 | Snort Alert [1:2402001:3544] | 2 | 3545 | 2402001 | 1 | 11 | | 47122 | 30 | Snort Alert [1:2402001:3544] | 2 | 3546 | 2402001 | 1 | 4 | | 47142 | 30 | Snort Alert [1:2402001:3544] | 2 | 3547 | 2402001 | 1 | 1 | +--------+--------------+---------------------------------------------+--------------+---------+---------+---------+--------------+ 6 rows in set (0.05 sec)
So I know things are getting put in there, but it seems like it's not using the information barnyard2 put in the db.
Thanks in advanced.
You need to configure in Snorby the path to the Snort rules file that PulledPork creates. When PulledPork runs and pulls down the rules, it will create a snort.rules file containing all the rules. It will also create a sid-msg.map file for the rules. Make sure that Snorby knows where that snort.rules file is located and that Snorby has access to read the file. Also make sure Barnyard2 on the Snorby box knows where the sid-msg.map file is located and that it can read it.
Bill
-
Thanks for the information again. I ran through the snorby docs and it doesn't really say much on how to get snorby to recognize defined events. I ended up finding an obscure reference to edit the snorby_config.yml file and add in your snort rules to the 'rules' location. I did this, but it still doesn't work.
Ultimately, pfsense's snort is putting the information into the database. It's just something wrong with snorby that it won't read the rules and translate them from numbers into english. Now when I'm running barnyard2 on the non-pfsense box, I'm getting the duplicate entry error again.
So, the bottom line is that it's not pfsense's fault and I think at this point, I'll go ask over at barnyard2 to fix that error, then snorby to fix the event id's error.
Thanks!
-
I've found Snorby to get the rule text right most of the time. Glad you found that rules setting. I forgot to point it out. If you find out something from your other queries that you think needs changing in the Snort and Suricata packages with respect to Barnyard2, post back here or send me a PM.
Bill