Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    If you are having Barnyard2 troubles with the last Snort update – try this

    IDS/IPS
    9
    37
    14.8k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • bmeeksB
      bmeeks
      last edited by

      Some folks have reported database errors coming from Barnyard2 with the latest Snort package.  One of the things that changed recently with the Snort package is Barnyard2 was updated from 1.2 to 1.3.  This was done to keep pace with upstream and because 1.3 of Barnyard2 offers a warm restart ability via SIGHUP.  Also, the newer Barnyard2 version supports the new v2 sid-msg.map file format.  This updated format includes more information about rule signatures.  The Snort package on pfSense now generates v2 sid-msg.map files for use by Barnyard2.

      However, there were apparently some important changes to the way Barnyard2 v1.3 interacts with a MySQL database as compared to the older versions.  This can become a problem when your MySQL database contains records put there by a previous Barnyard2 version.  Here are two links to post on the web that can help you fix this problem if you have it.

      This is the one I recommend you try first –
      http://sourceforge.net/p/snort/mailman/message/31925851/

      Here is another longer thread with similar information –
      https://groups.google.com/forum/#!topic/barnyard2-users/IIoyClc7XTc

      In the first link I posted, you can copy and paste the script there into a PuTTY session with your DB server and run it all quite easily.  I was having issues just like users have reported here on the Forum, and running the scripts in the first link above fixed them.

      Bill

      1 Reply Last reply Reply Quote 0
      • G
        gscasny
        last edited by

        Tried both of these on a 2 clean Security Onion installs, same problem.. duplicate entry:

        barnyard2[14279]: FATAL ERROR: database mysql_error: Duplicate entry '857-2' for key 'PRIMARY' SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('392','857','2');]

        I can delete this record, but the error come back wit a different alert, and I even cleared the sif_reference table… stays up for about 5 minutes and then pukes :(

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by

          @gscasny:

          Tried both of these on a 2 clean Security Onion installs, same problem.. duplicate entry:

          barnyard2[14279]: FATAL ERROR: database mysql_error: Duplicate entry '857-2' for key 'PRIMARY' SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('392','857','2');]

          I can delete this record, but the error come back wit a different alert, and I even cleared the sif_reference table… stays up for about 5 minutes and then pukes :(

          Do you have more than one Barnyard2 instance feeding the DB?  If so, do they all have different SIDs and CIDs?  I would like to get to the bottom of this problem in order to see if I need to change something in the way Barnyard2 is configured within Snort.

          Did this only start with the latest v3.0.5 version of the Snort package GUI?  That would be the version that was posted early last week.  The Barnyard2 binary version was actually updated to 1.3 in the previous Snort package update.  What changed this time was the addition of additional configuration and the sid-msg.map file format was migrated to the new v2 format.

          Sorry you are having this problem, but if you will bear with me, hopefully we can straighten it out.

          Bill

          1 Reply Last reply Reply Quote 0
          • G
            gscasny
            last edited by

            No need to apologize - I really appreciate the help and anyway I can contribute to helping get this resolved I will :)

            I have 2 sensors on a pfSense box (One LAN on WAN) with different Snort rulesets. Each runs it's own instance of Barnyard with 2 different config files:

            /usr/pbi/snort-i386/etc/snort/snort_6190_em0/barnyard2.conf  (WAN)
            /usr/pbi/snort-i386/etc/snort/snort_42710_em1/barnyard2.conf (LAN)

            When I start Barnyard2, it creates 2 processes from what I see in the logs:

            barnyard2[25584]: Writing PID "25584" to file "/var/run/barnyard2_em06190.pid"
            barnyard2[30270]: Writing PID "30270" to file "/var/run/barnyard2_em142710.pid"

            and there are 2 processes running (I realize the PID's don't match above, this was taken after a restart):
            root    58898 58.7  1.6 21932 16304  ??  Rs  11:15AM  0:08.47 /usr/local/bin/barnyard2 -r 42710 -f snort_42710_em1.u2 –pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-i386/etc/snort/snort_42710_em1/barnyard2.conf -d /var/log/snort/snort_em142710 -D -q
            root    25584  0.0  3.0 41388 30384  ??  Ss    9:56AM  1:14.18 /usr/local/bin/barnyard2 -r 6190 -f snort_6190_em0.u2 --pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-i386/etc/snort/snort_6190_em0/barnyard2.conf -d /var/log/snort/snort_em06190 -D -q

            1 Reply Last reply Reply Quote 0
            • bmeeksB
              bmeeks
              last edited by

              @gscasny:

              No need to apologize - I really appreciate the help and anyway I can contribute to helping get this resolved I will :)

              I have 2 sensors on a pfSense box (One LAN on WAN) with different Snort rulesets. Each runs it's own instance of Barnyard with 2 different config files:

              /usr/pbi/snort-i386/etc/snort/snort_6190_em0/barnyard2.conf  (WAN)
              /usr/pbi/snort-i386/etc/snort/snort_42710_em1/barnyard2.conf (LAN)

              When I start Barnyard2, it creates 2 processes from what I see in the logs:

              barnyard2[25584]: Writing PID "25584" to file "/var/run/barnyard2_em06190.pid"
              barnyard2[30270]: Writing PID "30270" to file "/var/run/barnyard2_em142710.pid"

              and there are 2 processes running (I realize the PID's don't match above, this was taken after a restart):
              root    58898 58.7  1.6 21932 16304  ??  Rs  11:15AM  0:08.47 /usr/local/bin/barnyard2 -r 42710 -f snort_42710_em1.u2 –pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-i386/etc/snort/snort_42710_em1/barnyard2.conf -d /var/log/snort/snort_em142710 -D -q
              root    25584  0.0  3.0 41388 30384  ??  Ss    9:56AM  1:14.18 /usr/local/bin/barnyard2 -r 6190 -f snort_6190_em0.u2 --pid-path /var/run --nolock-pidfile -c /usr/pbi/snort-i386/etc/snort/snort_6190_em0/barnyard2.conf -d /var/log/snort/snort_em06190 -D -q

              OK, all that is good.  Two configs and two processes is normal.  Each will work from separate sub-directories in /var/log/snort in terms of the unified2 logs they process.

              I have three instances of Barnyard2 running on my system, but mine is a home network and really the only interface with any events is the WAN.  And even those are really just noise due to the NIC being in promiscuous mode.  I have not seen this error, but it very well could simply be because my event count is low and is basically coming from only one of the three interfaces.  The last event I had no my LAN was March 12th, and my DMZ has not recorded an event since I migrated to a new firewall late last year.

              I will dig around on Google to see if I can find some clues.  Feel free to do the same, and if you find something interesting either post back here or send me a PM.

              Bill

              1 Reply Last reply Reply Quote 0
              • bmeeksB
                bmeeks
                last edited by

                I find quite a few examples of this "duplicate entry" error with the newest Barnyard2 binary when searching on Google.  I found one post that I think sums up exactly what is wrong, and it points to a concurrency issue within the Barnyard2 code itself.  Boiling it down to the bare essentials, when inserting an event into the DB, Barnyard2 also inserts any references included with the signature into the sig_references table.  These references are part of the sid-msg.map file generated by Snort for each interface.  If multiple interfaces run some of the same rules, then parts of the sid-msg.map files will be similar across Snort interfaces.  A race condition could develop where two Barnyard2 processes are attempting to insert the same signature at essentially the same time.  Only one will win due to transaction locking on the DB.  The other will fail due to a duplicate primary key error, and thus you get the error being seen in the Snort package.  That theory was posted by a user on Google, and I think it is most likely the correct one.

                The ultimate fix is to modify the Barnyard2 source code to work around this problem.  However, in the interim there is a parameter that can be included on the configuration line for the DB output plugin that disables updates to the signature references table.  I will add this parameter to the Snort GUI page for Barnyard2 so that users experiencing this duplicate entry error can work around it.

                I think this explains why I have not seen the error, and why perhaps some others have not.  It is dependent on exactly which rules you run on each interface if you run Snort and Barnyard2 on more than one interface.

                Here is a link to one of the posts I found describing the problem:  http://sourceforge.net/p/snort/mailman/snort-users/thread/CAFU9AX-B_dGQDmM1af0W-umFLGPoEENfaC9bNEkpGdEwwgXeZw@mail.gmail.com/

                Bill

                1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks
                  last edited by

                  I have posted a new Pull Request on GitHub for the Snort package.  It is being reviewed by the pfSense Core Team developers.  The update will appear as Snort package version 2.9.6.0 v3.0.7 when they approve and merge the changes.  A workaround fix for the Barnyard2 duplicate entry error is included in the update.

                  Bill

                  1 Reply Last reply Reply Quote 0
                  • G
                    gscasny
                    last edited by

                    Thanks so much Bill!

                    I have access to several snort systems that I can run attacks against to generate a ton of events to test against. Just say the word and I am happy to help debug, etc.

                    Greg

                    1 Reply Last reply Reply Quote 0
                    • G
                      gscasny
                      last edited by

                      I think your explanation is right on the mark with what I see. I do have some overlap rules on my 2 sensors (LAN and WAN), and only 1 will run continually, usually the first one I start.  Thanks again for your work on this.

                      1 Reply Last reply Reply Quote 0
                      • B
                        BhavinP
                        last edited by

                        Bill,

                        I would like to contribute the following to this error:

                        Error also happens in the event table –

                        Apr 23 20:03:56 labkord-lxfw01 barnyard2[47108]: FATAL ERROR: database mysql_error: Duplicate entry '1-11' for key 'PRIMARY'    SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 11, 289, '2014-04-23 20:03:55');]
                        Apr 23 20:03:56 labkord-lxfw01 barnyard2[69591]: FATAL ERROR: database mysql_error: Duplicate entry '1-11' for key 'PRIMARY'    SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 11, 289, '2014-04-23 20:03:55');]
                        Apr 23 20:05:15 labkord-lxfw01 barnyard2[92662]: FATAL ERROR: database mysql_error: Duplicate entry '1-13' for key 'PRIMARY'    SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 13, 301, '2014-04-23 20:05:15');]

                        As we all know the sig_reference table too:

                        Apr 23 09:28:00 labkord-lxfw01 barnyard2[20704]: FATAL ERROR: database mysql_error: Duplicate entry '10274-3' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('12401','10274','3');]
                        Apr 23 09:35:30 labkord-lxfw01 barnyard2[22589]: FATAL ERROR: database mysql_error: Duplicate entry '10274-3' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('12401','10274','3');]
                        Apr 23 10:16:35 labkord-lxfw01 barnyard2[62763]: FATAL ERROR: database mysql_error: Duplicate entry '10274-3' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('12401','10274','3');]
                        Apr 23 10:28:06 labkord-lxfw01 barnyard2[40004]: FATAL ERROR: database mysql_error: Duplicate entry '10274-2' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('6892','10274','2');]

                        1 Reply Last reply Reply Quote 0
                        • bmeeksB
                          bmeeks
                          last edited by

                          @BhavinP:

                          Bill,

                          I would like to contribute the following to this error:

                          Error also happens in the event table –

                          Apr 23 20:03:56 labkord-lxfw01 barnyard2[47108]: FATAL ERROR: database mysql_error: Duplicate entry '1-11' for key 'PRIMARY'    SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 11, 289, '2014-04-23 20:03:55');]
                          Apr 23 20:03:56 labkord-lxfw01 barnyard2[69591]: FATAL ERROR: database mysql_error: Duplicate entry '1-11' for key 'PRIMARY'    SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 11, 289, '2014-04-23 20:03:55');]
                          Apr 23 20:05:15 labkord-lxfw01 barnyard2[92662]: FATAL ERROR: database mysql_error: Duplicate entry '1-13' for key 'PRIMARY'    SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (1, 13, 301, '2014-04-23 20:05:15');]

                          As we all know the sig_reference table too:

                          Apr 23 09:28:00 labkord-lxfw01 barnyard2[20704]: FATAL ERROR: database mysql_error: Duplicate entry '10274-3' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('12401','10274','3');]
                          Apr 23 09:35:30 labkord-lxfw01 barnyard2[22589]: FATAL ERROR: database mysql_error: Duplicate entry '10274-3' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('12401','10274','3');]
                          Apr 23 10:16:35 labkord-lxfw01 barnyard2[62763]: FATAL ERROR: database mysql_error: Duplicate entry '10274-3' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('12401','10274','3');]
                          Apr 23 10:28:06 labkord-lxfw01 barnyard2[40004]: FATAL ERROR: database mysql_error: Duplicate entry '10274-2' for key 'PRIMARY'        SQL=[INSERT INTO sig_reference (ref_id,sig_id,ref_seq) VALUES ('6892','10274','2');]

                          Thanks for the additional information.  I'm pretty sure this is a Barnyard2 binary problem as outlined above.  The Barnyard2 maintainers are going to have to look at their code IMHO.  In the meantime, to help users on pfSense work around the problem, I've submitted some patched code as a Pull Request on Github.  It is awaiting approval and merging by the pfSense Core Team guys.  It has been posted for a while, and hopefully they can soon get some time to review and approve.

                          Bill

                          1 Reply Last reply Reply Quote 0
                          • F
                            fearnothing
                            last edited by

                            Would be interested to know if there's any update on this. I've followed all the instructions here and yet I'm still having the problem. Thought it was working after I updated snort and pfsense but the problem returned after an hour.

                            1 Reply Last reply Reply Quote 0
                            • C
                              Cino
                              last edited by

                              bill's patch has been added to snort already… barnyard2 needs to be updated but that is out of scope from the pfsense project.. The creators of barnyard2 would have to fix their code... When I run into the problem, I purge a few tables; start barnyard2 and everything is good again.

                              1 Reply Last reply Reply Quote 0
                              • bmeeksB
                                bmeeks
                                last edited by

                                @fearnothing:

                                Would be interested to know if there's any update on this. I've followed all the instructions here and yet I'm still having the problem. Thought it was working after I updated snort and pfsense but the problem returned after an hour.

                                Go to the BARNYARD tab and down in the MySQL DB section on all your Snort interfaces (except one) and check the box to disable updates to the signature reference table.  Leave this box unchecked on only a single Barnyard2 interface (typically the one that registers the most alerts).

                                The problem is with Barnyard2 itself and concurrent updates to the same DB table from multiple processes.  The Barnyard2 guys, in my opinion, broke their code in the version 2.1.3 update a while back.  The Snort and Suricata packages on pfSense use this updated version since the FreeBSD ports tree was also updated.

                                Bill

                                1 Reply Last reply Reply Quote 0
                                • F
                                  fearnothing
                                  last edited by

                                  The option is already checked on both interfaces. Is it significant that I'm only getting the duplicate entry error for the 'event' table, and not 'sig_reference?'

                                  1 Reply Last reply Reply Quote 0
                                  • bmeeksB
                                    bmeeks
                                    last edited by

                                    @fearnothing:

                                    The option is already checked on both interfaces. Is it significant that I'm only getting the duplicate entry error for the 'event' table, and not 'sig_reference?'

                                    Do you by chance have the same rules running on more than one interface, and are the Snort instances each alerting on the same traffic?  In other words, duplicate rules on WAN and LAN and each Snort instance alerts on the same traffic.

                                    The Barnyard2 guys made a few "adjustments" to the MySQL DB code in their last update, and those adjustments seem to have caused a race condition with data inserts and some tables.

                                    I was able to fix this by running the scripts linked earlier in this thread to clean up the tables, and then running only one BY2 interface with the checkbox not checked.

                                    Bill

                                    1 Reply Last reply Reply Quote 0
                                    • F
                                      fearnothing
                                      last edited by

                                      Nope, still happening.

                                      Jun 30 11:00:17 barnyard2[33829]: database: Closing connection to database "snorby"
                                      Jun 30 11:00:17 barnyard2[33829]: Barnyard2 exiting
                                      Jun 30 11:00:17 barnyard2[33829]: FATAL ERROR: database mysql_error: Duplicate entry '5-20' for key 'PRIMARY' SQL=[INSERT INTO event (sid,cid,signature,timestamp) VALUES (5, 20, 18861, '2014-06-30 11:00:15');]
                                      Jun 30 11:00:16 snort[57013]: [1:2003315:3] ET P2P Edonkey Search Reply [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 77.80.253.67:27001 -> 192.168.1.102:27960
                                      Jun 30 11:00:16 snort[57013]: [1:2003315:3] ET P2P Edonkey Search Reply [Classification: Potential Corporate Privacy Violation] [Priority: 1] {UDP} 77.80.253.67:27001 -> 192.168.1.102:27960

                                      This rule is ONLY present on LAN. Ironically it's a false positive detecting a Quake Live connection as Edonkey :P

                                      1 Reply Last reply Reply Quote 0
                                      • F
                                        fearnothing
                                        last edited by

                                        The plot thickens… my snorby instance is still receiving alerts from the LAN interface despite the barnyard2 process for LAN being marked as stopped in the GUI.

                                        1 Reply Last reply Reply Quote 0
                                        • G
                                          gscasny
                                          last edited by

                                          Im seeing this rear its head again (im using Suricata), even thought I have the box checked for both sensors that are reporting…

                                          1 Reply Last reply Reply Quote 0
                                          • G
                                            gscasny
                                            last edited by

                                            @gscasny:

                                            Im seeing this rear its head again (im using Suricata), even thought I have the box checked for both sensors that are reporting…

                                            Oops.. It's the event table this time as stated above.. just started happening with the last update..

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post