Suricata v4.1.6_1 - Package update Release Notes

bmeeks

The problems with GeoIP database downloads and loss of GeoIP functionality in the Suricata 4.1.6 package have been identified and corrected. Look for an update to 4.1.6_2 to show up in the near future. Here is a link to the pull request containing the fix: https://github.com/pfsense/FreeBSD-ports/pull/749.

jm1384

@bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

The problems with GeoIP database downloads and loss of GeoIP functionality in the Suricata 4.1.6 package have been identified and corrected. Look for an update to 4.1.6_2 to show up in the near future. Here is a link to the pull request containing the fix: https://github.com/pfsense/FreeBSD-ports/pull/749.

Thank's bmeeks for the pull request !

jm1384

Updated 4.1.6_1 to 4.1.6_2 today (Removed and reinstalled package) and database was corrupted after extraction :
System log :

[Suricata] A new GeoLite2-Country IP database is available.
[Suricata] Downloading new GeoLite2-Country IP database...
[Suricata] New GeoLite2-Country IP database gzip archive successfully downloaded.
[Suricata] Extracting new GeoLite2-Country database from the archive...
[Suricata] Moving new database to /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb...
[Suricata] GeoLite2-Country database update completed.
[Suricata] Cleaning up temp files after GeoLite2-Country database update.

Suricata log :

Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.

Temp fix :
Create a link from PfblockerNG Maxmind DB to the suricata Geolite directory :

cd /usr/local/share/suricata/GeoLite2/
rm GeoLite2*

cd /usr/local/share/GeoIP
ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

ls -l GeoLite2-Country.mmdb
-rw-r--r--  2 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb

NRgia

@jm1384 For me it works...no errors:

suricata GeoLite.png

Also checked the log here /var/log/suricata/suricata_[interface]/suricata.log, and found no errors.

bmeeks

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

Temp fix :

If you are implementing a symlink to that file in another directory, that may be causing issues with Suricata unpacking and copying the database. Remove your symlink completely, clean out the Suricata GeoLite2 DB directory and then run this command from a shell prompt:

php /usr/local/pkg/suricata/suricata_geoipupdate.php

Then check the system log and the suricata.log file for the interface. You should see a successful download. Restart Suricata on the interface and it should be good. The 4.1.6_2 version of the Suricata package fixes the GeoLite2 database corruption issue. I tested it several times to be sure.

jm1384

@bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

Temp fix :

If you are implementing a symlink to that file in another directory, that may be causing issues with Suricata unpacking and copying the database. Remove your symlink completely, clean out the Suricata GeoLite2 DB directory and then run this command from a shell prompt:
php /usr/local/pkg/suricata/suricata_geoipupdate.php
Then check the system log and the suricata.log file for the interface. You should see a successful download. Restart Suricata on the interface and it should be good. The 4.1.6_2 version of the Suricata package fixes the GeoLite2 database corruption issue. I tested it several times to be sure.

Hi bmeeks, thank you !
ok, let's go !
After removed all db files in suricata DB directory :

php /usr/local/pkg/suricata/suricata_geoipupdate.php

Suricata log :

Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.

suricata package removed :

>>> Removing pfSense-pkg-suricata... 
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
	pfSense-pkg-suricata-4.1.6_2

Number of packages to be removed: 1
[1/1] Deinstalling pfSense-pkg-suricata-4.1.6_2...
Removing suricata components...
Menu items... done.
Services... done.
Loading package instructions...
[1/1] Deleting files for pfSense-pkg-suricata-4.1.6_2: .......... done
Removing suricata components...
Configuration... done.
>>> Removing stale packages... done.
Success

directory /suricata/GeoLite2/ :

ls /suricata/GeoLite2: No such file or directory
ls /suricata: No such file or directory

No directory, no files, no links.

After reinstall suricata /usr/local/share/suricata/GeoLite2 :

ls -lrT  GeoLite2/
total 2084 
-rw-r--r-- 1 root  wheel  32 Jan 19 20:56:35 2020 GeoLite2-Country.mmdb.tar.gz.md5
-rw-r--r-- 1 root  wheel  2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

Just one link "1" for suricata GeoLite2-Country.mmdb after unzipped :
DB time and day : Jan 19 20:56:37 2020

Now i check the pfBlokerNG DB file :

ls -l GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb

Just one link "1".
DB time and day : Jan 7 00:45:59 2020

They are no symlink for this pfblockerNG DB file,
this DB file is different to suricata DB in days and times.

Suricata log after install :

Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.

Ok, no problem, let's do the trick.
After removing all suricata DB files in /usr/local/share/suricata/GeoLite2/ :

rm GeoLite2*
ls

I create a physical link from pfblockerNG DB file to suricata DB directory :

cd /usr/local/share/GeoIP
ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

Now, the pfBlokerNG DB has two links (2):

ls -l GeoLite2-Country.mmdb 
-rw-r--r-- 2 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb

Suricata log after restart :

0 rules failed
engine started.

GeoIP rules working fine.

RonpfS

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

-rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

From the size I guessed that the mmdb is still in a tar format :

tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
-rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
-rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
-rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb

jm1384

@RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

-rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

From the size I guessed that the mmdb is still in a tar format :
tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
-rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
-rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
-rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb

This is your suricata V4.1.6_2 db file size ?

RonpfS

Yes

ls -al /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel  2076656 Jan 18 18:51 /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

jm1384

@RonpfS
ok, i will try to remove Suricata completely, package and backup config and reinstall it.

Thank's RonpfS

jm1384

@RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

Yes

ls -al /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel  2076656 Jan 18 18:51 /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

Your suricata db file zise is the same as mine after downloading,
do you have the same error log in suricata log for Geoip db ?

RonpfS

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

Your suricata db file zise is the same as mine after downloading,

Yes it comes from the same server. So GeoLite2-Country_.mmdb is in fact GeoLite2-Country_20200114.mmdb.tar.gz

jm1384

@RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

Your suricata db file zise is the same as mine after downloading,

Yes it comes from the same server. So GeoLite2-Country.mmdb is in fact GeoLite2-Country.mmdb.tar.gz

you have the same problem as me with this db file ?

RonpfS

Yes it look the same. So until @bmeeks find what's wrong, disable GeoIP update in Suricata use the pfblockerNG one

mv /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb.tar.gz
ln -s /usr/local/share/GeoIP/GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb

jm1384

@RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

@jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

-rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

From the size I guessed that the mmdb is still in a tar format :
tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
-rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
-rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
-rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb

ok
your are right about untar archive, wait and see if @bmeeks can resolve this issue.

bmeeks

I promise this was working correctly when I tested prior to submitting the pull request. Let me do a fresh install in a test VM to see what's happening.

RonpfS

@bmeeks I upgraded on Jan 18.
To be on the safe side, I uninstalled and installed 1 hour ago. same results.

total 1994
drwxr-xr-x  2 root  wheel        4 Jan 19 18:25 .
drwxr-xr-x  4 root  wheel        4 Jan 19 18:25 ..
-rw-r--r--  1 root  wheel  2076656 Jan 19 18:25 GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel       32 Jan 19 18:25 GeoLite2-Country.mmdb.tar.gz.md5

bmeeks

I screwed the new code up. Working on correcting it. I don't know what I tested, but it did work. Must be losing my mind ... .

Will get a correction posted soon.

bmeeks

Okay. Sorry about the previous screw-up with the GeoIP database. The new fix is posted here for the pfSense team to review and merge. If you want to make the changes yourself in your file before the fix is posted, you can look at the edits in the linked pull request.

Look for a package update to version 4.1.6_3 in the near future.

I don't even have a good lie to use to try and cover this one up. I will just have to own the mistake up front ...

RonpfS

@bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

If you want to make the changes yourself in your file before the fix is posted,

Just did the test and the DB is extracted ok now

rm /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb*
php /usr/local/pkg/suricata/suricata_geoipupdate.php

ls -al
total 2
drwxr-xr-x  2 root  wheel        4 Jan 19 20:56 .
drwxr-xr-x  4 root  wheel        4 Jan 19 18:25 ..
-rw-r--r--  1 root  wheel  4083997 Jan 19 20:56 GeoLite2-Country.mmdb
-rw-r--r--  1 root  wheel       32 Jan 19 20:56 GeoLite2-Country.mmdb.tar.gz.md5