Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Suricata v4.1.6_1 - Package update Release Notes

    Scheduled Pinned Locked Moved IDS/IPS
    40 Posts 7 Posters 3.9k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jm1384 @bmeeks
      last edited by

      @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

      @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

      @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

      @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

      Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

      Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

      Sorry, I've edited my last post, see above, i think, it's a DB file problem.
      Geoip rules working good with Pfblokerng DB file from december DB.

      Okay, I've got this sorted out and will submit a fix for it soon.

      The root cause is the new database gzip archive has an extra sub-directory path in it where the actual database is stored. My PHP code was not allowing for that and thus wound up copying a corrupted database over to the shared area where Suricata was looking.

      I also shot myself in the foot when investigating the database because I initially was using my own custom rule with a known IP address from Japan for testing. However, later during my testing, I switched over to using your custom rule in my test setup, but I did not notice until MUCH later that your rule does not contain the JP country code for Japan. Thus my tests using the JP IP address were all still failing, even with a "good" database in place. That false result sent me down the path of suspecting the binary ... 😖. When you said copying over the pfBlockerNG database fixed it for you, I went back and carefully checked my testing methodology and discovered the issue with the IP address I was using not actually being covered in your GeoIP rule! Feel really stupid now...

      That's a good news !
      Thank you very much bmeeks ! 👍

      Oh my god ! 😕
      you're going around in circles just for my missing country code in my rule.
      Sorry for this mistake. 😖

      I'm waiting the update, thank's for all bmeeks !
      Best regards.

      bmeeksB 1 Reply Last reply Reply Quote 0
      • bmeeksB
        bmeeks @jm1384
        last edited by

        @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

        @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

        @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

        @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

        @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

        Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

        Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

        Sorry, I've edited my last post, see above, i think, it's a DB file problem.
        Geoip rules working good with Pfblokerng DB file from december DB.

        Okay, I've got this sorted out and will submit a fix for it soon.

        The root cause is the new database gzip archive has an extra sub-directory path in it where the actual database is stored. My PHP code was not allowing for that and thus wound up copying a corrupted database over to the shared area where Suricata was looking.

        I also shot myself in the foot when investigating the database because I initially was using my own custom rule with a known IP address from Japan for testing. However, later during my testing, I switched over to using your custom rule in my test setup, but I did not notice until MUCH later that your rule does not contain the JP country code for Japan. Thus my tests using the JP IP address were all still failing, even with a "good" database in place. That false result sent me down the path of suspecting the binary ... 😖. When you said copying over the pfBlockerNG database fixed it for you, I went back and carefully checked my testing methodology and discovered the issue with the IP address I was using not actually being covered in your GeoIP rule! Feel really stupid now...

        That's a good news !
        Thank you very much bmeeks ! 👍

        Oh my god ! 😕
        you're going around in circles just for my missing country code in my rule.
        Sorry for this mistake. 😖

        I'm waiting the update, thank's for all bmeeks !
        Best regards.

        It was my fault for not double-checking the IP. It just did not cross my mind. I chose the IP initially anyway, and then did not verify that your rule covered the IP I had arbitrarily chosen. My fault all the way ... ☺ .

        There is a happy ending, though. It gives me a chance to make the GeoIP database download script more robust. Also have it checking the posted MD5 hash on the MaxMind site against what is already present on the firewall so that it only downloads a new database when there is a mismatch. I'm going to change the check for a new database version back to once per day since it will only be downloading and checking the 32 byte MD5 file. If the MD5 on the firewall differs from the posted MD5 on the MaxMind site, then it will download the entire database (approximately 4 MB).

        1 Reply Last reply Reply Quote 0
        • bmeeksB
          bmeeks
          last edited by

          The problems with GeoIP database downloads and loss of GeoIP functionality in the Suricata 4.1.6 package have been identified and corrected. Look for an update to 4.1.6_2 to show up in the near future. Here is a link to the pull request containing the fix: https://github.com/pfsense/FreeBSD-ports/pull/749.

          J 1 Reply Last reply Reply Quote 1
          • J
            jm1384 @bmeeks
            last edited by

            @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

            The problems with GeoIP database downloads and loss of GeoIP functionality in the Suricata 4.1.6 package have been identified and corrected. Look for an update to 4.1.6_2 to show up in the near future. Here is a link to the pull request containing the fix: https://github.com/pfsense/FreeBSD-ports/pull/749.

            Thank's bmeeks for the pull request !

            1 Reply Last reply Reply Quote 0
            • J
              jm1384
              last edited by jm1384

              Updated 4.1.6_1 to 4.1.6_2 today (Removed and reinstalled package) and database was corrupted after extraction : 😕
              System log :

              [Suricata] A new GeoLite2-Country IP database is available.
[Suricata] Downloading new GeoLite2-Country IP database...
[Suricata] New GeoLite2-Country IP database gzip archive successfully downloaded.
[Suricata] Extracting new GeoLite2-Country database from the archive...
[Suricata] Moving new database to /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb...
[Suricata] GeoLite2-Country database update completed.
[Suricata] Cleaning up temp files after GeoLite2-Country database update.

              Suricata log :

              Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.

              Temp fix :
              Create a link from PfblockerNG Maxmind DB to the suricata Geolite directory :

              cd /usr/local/share/suricata/GeoLite2/
rm GeoLite2*

cd /usr/local/share/GeoIP
ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

ls -l GeoLite2-Country.mmdb
-rw-r--r--  2 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb
              N bmeeksB 2 Replies Last reply Reply Quote 0
              • N
                NRgia @jm1384
                last edited by NRgia

                @jm1384 For me it works...no errors:

                suricata GeoLite.png

                Also checked the log here /var/log/suricata/suricata_[interface]/suricata.log, and found no errors.

                1 Reply Last reply Reply Quote 0
                • bmeeksB
                  bmeeks @jm1384
                  last edited by bmeeks

                  @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                  Temp fix :

                  If you are implementing a symlink to that file in another directory, that may be causing issues with Suricata unpacking and copying the database. Remove your symlink completely, clean out the Suricata GeoLite2 DB directory and then run this command from a shell prompt:

                  php /usr/local/pkg/suricata/suricata_geoipupdate.php

                  Then check the system log and the suricata.log file for the interface. You should see a successful download. Restart Suricata on the interface and it should be good. The 4.1.6_2 version of the Suricata package fixes the GeoLite2 database corruption issue. I tested it several times to be sure.

                  J 1 Reply Last reply Reply Quote 0
                  • J
                    jm1384 @bmeeks
                    last edited by

                    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

                    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                    Temp fix :

                    If you are implementing a symlink to that file in another directory, that may be causing issues with Suricata unpacking and copying the database. Remove your symlink completely, clean out the Suricata GeoLite2 DB directory and then run this command from a shell prompt:

                    php /usr/local/pkg/suricata/suricata_geoipupdate.php

                    Then check the system log and the suricata.log file for the interface. You should see a successful download. Restart Suricata on the interface and it should be good. The 4.1.6_2 version of the Suricata package fixes the GeoLite2 database corruption issue. I tested it several times to be sure.

                    Hi bmeeks, thank you !
                    ok, let's go !
                    After removed all db files in suricata DB directory :

                    php /usr/local/pkg/suricata/suricata_geoipupdate.php

                    Suricata log :

                    Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.

                    suricata package removed :

                    >>> Removing pfSense-pkg-suricata... 
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	pfSense-pkg-suricata-4.1.6_2

Number of packages to be removed: 1
[1/1] Deinstalling pfSense-pkg-suricata-4.1.6_2...
Removing suricata components...
Menu items... done.
Services... done.
Loading package instructions...
[1/1] Deleting files for pfSense-pkg-suricata-4.1.6_2: .......... done
Removing suricata components...
Configuration... done.
>>> Removing stale packages... done.
Success

                    directory /suricata/GeoLite2/ :

                    ls /suricata/GeoLite2: No such file or directory
ls /suricata: No such file or directory

                    No directory, no files, no links.

                    After reinstall suricata /usr/local/share/suricata/GeoLite2 :

                    ls -lrT  GeoLite2/
total 2084 
-rw-r--r-- 1 root  wheel  32 Jan 19 20:56:35 2020 GeoLite2-Country.mmdb.tar.gz.md5
-rw-r--r-- 1 root  wheel  2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

                    Just one link "1" for suricata GeoLite2-Country.mmdb after unzipped :
                    DB time and day : Jan 19 20:56:37 2020

                    Now i check the pfBlokerNG DB file :

                    ls -l GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb

                    Just one link "1".
                    DB time and day : Jan 7 00:45:59 2020

                    They are no symlink for this pfblockerNG DB file,
                    this DB file is different to suricata DB in days and times.

                    Suricata log after install :

                    Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.

                    Ok, no problem, let's do the trick.
                    After removing all suricata DB files in /usr/local/share/suricata/GeoLite2/ :

                    rm GeoLite2*
ls

                    I create a physical link from pfblockerNG DB file to suricata DB directory :

                    cd /usr/local/share/GeoIP
ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

                    Now, the pfBlokerNG DB has two links (2):

                    ls -l GeoLite2-Country.mmdb 
-rw-r--r-- 2 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb

                    Suricata log after restart :

                    0 rules failed
engine started.

                    GeoIP rules working fine.

                    1 Reply Last reply Reply Quote 0
                    • RonpfSR
                      RonpfS
                      last edited by

                      @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                      -rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

                      From the size I guessed that the mmdb is still in a tar format :

                      tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
-rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
-rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
-rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb

                      2.4.5-RELEASE-p1 (amd64)
                      Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                      Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                      J 1 Reply Last reply Reply Quote 0
                      • J
                        jm1384 @RonpfS
                        last edited by

                        @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

                        @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                        -rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

                        From the size I guessed that the mmdb is still in a tar format :

                        tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
-rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
-rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
-rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb

                        This is your suricata V4.1.6_2 db file size ?

                        J 1 Reply Last reply Reply Quote 0
                        • RonpfSR
                          RonpfS
                          last edited by RonpfS

                          Yes

                          ls -al /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel  2076656 Jan 18 18:51 /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

                          2.4.5-RELEASE-p1 (amd64)
                          Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                          Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                          J 2 Replies Last reply Reply Quote 0
                          • J
                            jm1384 @RonpfS
                            last edited by

                            @RonpfS
                            ok, i will try to remove Suricata completely, package and backup config and reinstall it.

                            Thank's RonpfS 👍

                            1 Reply Last reply Reply Quote 0
                            • J
                              jm1384 @RonpfS
                              last edited by

                              @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

                              Yes

                              ls -al /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel  2076656 Jan 18 18:51 /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

                              Your suricata db file zise is the same as mine after downloading,
                              do you have the same error log in suricata log for Geoip db ?

                              RonpfSR 1 Reply Last reply Reply Quote 0
                              • RonpfSR
                                RonpfS @jm1384
                                last edited by RonpfS

                                @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                                Your suricata db file zise is the same as mine after downloading,

                                Yes it comes from the same server. So GeoLite2-Country_.mmdb is in fact GeoLite2-Country_20200114.mmdb.tar.gz

                                2.4.5-RELEASE-p1 (amd64)
                                Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                J 1 Reply Last reply Reply Quote 0
                                • J
                                  jm1384 @RonpfS
                                  last edited by

                                  @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

                                  @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                                  Your suricata db file zise is the same as mine after downloading,

                                  Yes it comes from the same server. So GeoLite2-Country.mmdb is in fact GeoLite2-Country.mmdb.tar.gz

                                  you have the same problem as me with this db file ?

                                  1 Reply Last reply Reply Quote 0
                                  • RonpfSR
                                    RonpfS
                                    last edited by

                                    Yes it look the same. So until @bmeeks find what's wrong, disable GeoIP update in Suricata use the pfblockerNG one

                                    mv /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb.tar.gz
ln -s /usr/local/share/GeoIP/GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

                                    2.4.5-RELEASE-p1 (amd64)
                                    Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                    Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                    1 Reply Last reply Reply Quote 0
                                    • J
                                      jm1384 @jm1384
                                      last edited by jm1384

                                      @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

                                      @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

                                      -rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

                                      From the size I guessed that the mmdb is still in a tar format :

                                      tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
-rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
-rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
-rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb

                                      ok 👍
                                      your are right about untar archive, wait and see if @bmeeks can resolve this issue.

                                      1 Reply Last reply Reply Quote 0
                                      • bmeeksB
                                        bmeeks
                                        last edited by

                                        I promise this was working correctly when I tested prior to submitting the pull request. Let me do a fresh install in a test VM to see what's happening.

                                        RonpfSR 1 Reply Last reply Reply Quote 1
                                        • RonpfSR
                                          RonpfS @bmeeks
                                          last edited by RonpfS

                                          @bmeeks I upgraded on Jan 18.
                                          To be on the safe side, I uninstalled and installed 1 hour ago. same results.

                                          total 1994
drwxr-xr-x  2 root  wheel        4 Jan 19 18:25 .
drwxr-xr-x  4 root  wheel        4 Jan 19 18:25 ..
-rw-r--r--  1 root  wheel  2076656 Jan 19 18:25 GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel       32 Jan 19 18:25 GeoLite2-Country.mmdb.tar.gz.md5

                                          2.4.5-RELEASE-p1 (amd64)
                                          Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                          Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                          1 Reply Last reply Reply Quote 0
                                          • bmeeksB
                                            bmeeks
                                            last edited by

                                            I screwed the new code up. Working on correcting it. I don't know what I tested, but it did work. Must be losing my mind ... 😞 .

                                            Will get a correction posted soon.

                                            1 Reply Last reply Reply Quote 1
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.