Suricata v4.1.6_1 - Package update Release Notes



  • Suricata-4.1.6_1

    This GUI package update provides support for the latest version of the Suricata binary (v4.1.6). It also corrects five bugs and adds four new features.

    Note: support for the Suricata 5.0.1 binary is forthcoming, but that version will ONLY run on Intel/AMD64 hardware as of now due to the decision from upstream to make Rust a requirement for compiling the 5.x binary. Currently there is no method for compiling Rust to support installing the Suricata 5.x binary on ARM-based hardware such as the SG-1000, SG-1100 and SG-3100 Netgate appliances. I am working with the pfSense developer team to come up with a method to maintain a 4.x version of Suricata for ARM hardware while the 5.x version is made available for Intel/AMD64 hardware.

    New Features:

    1. Added column sorting to the RULES tab so it behaves the same as the ALERTS tab.

    2. Make filters on ALERTS tab sticky across other actions such as suppressing alerts or disabling a SID. Currently any applied filter resets. See Redmine Issue #9902.

    3. Highlight rules with "noalerts;" option on RULES tab by coloring the text using the Bootstrap class "text-success" and using a different ACTION icon. Note that you cannot change the action for these rules. The noalert; option overrides the action verb in such rules, so there is no point in changing the action. If you want such rules to actually alert, then you would need to alter the rule content to remove the noalert; option.

    4. Added option to set unique logging directory for file-store configuration section in suricata.yaml.

    Bug Fixes:

    1. When creating HOME_NET and EXTERNAL_NET (and other ipvars), make sure there is a comma followed by a space for each IP entry. Failure to do so with intermixed IPv4 and IPv6 addresses results in failure to start with no ERRCODE given. See Suricata Redmine Issue #3222.

    2. Use of explode() function in suricata_interfaces_edit.php generates a warning because the second parameter of the function call is not interpreted as an empty string due to being uninitialized on green-field installs.

    3. When SID MGMT changes rules to DROP or REJECT, it is not skipping rules containing a "flowbits:noalert;" tag. This can result in dropped traffic without any logged alert about the drop or reject action.

    4. The IP Reputation List enable option on the IPREP tab actually defaults to "ON" when it should default to "OFF". See Redmine Issue #9981.

    5. Maxmind GeoLite2 IP DB now requires a license key for GeoIP2 database downloads. Add support for user-supplied license key.



  • Re: Suricate 5.0.1 and AMD64/ARM.... Just curious, but what about Intel hardware e.g. Intel Core i5-5250U?



  • Hi bmeeks,
    Since the last update of Suricata-4.1.6_1 my Geoip drop rules are nothing to do.

    • Geoip drop rules are no actions.

    • no drop logs in eve.json.

    • Remove and reinstall suricata do nothing, same things.

    • No error log in suricata log :

    rules successfully loaded, 0 rules failed
    engine started.
    
    • GeoLite2 IP DB downloaded fine with license key :
    ls -l /usr/local/share/suricata/GeoLite2
    total 4000
    -rw-r--r--  1 root  wheel  4044800 Jan  8 10:37 GeoLite2-Country.mmdb
    
    • Geoip lib :
    suricata --build-info
    GeoIP support: yes, libmaxminddb
    

    Thanks in advance !
    best regards.



  • @occamsrazor said in Suricata v4.1.6_1 - Package update Release Notes:

    Re: Suricate 5.0.1 and AMD64/ARM.... Just curious, but what about Intel hardware e.g. Intel Core i5-5250U?

    In the FreeBSD package world, AMD64 means Intel as well sinces those two chips execute essentially identical instructions; and binary code that runs on one will run on the other.

    ARM hardware uses a completely different binary instruction set (the opcodes are quite different), thus a separate compilation environment is required to create ARM code. Intel/AMD code is identical and so you can compile for either chip using the same compilation environment.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Hi bmeeks,
    Since the last update of Suricata-4.1.6_1 my Geoip drop rules are nothing to do.

    • Geoip drop rules are no actions.

    • no drop logs in eve.json.

    • Remove and reinstall suricata do nothing, same things.

    • No error log in suricata log :

    rules successfully loaded, 0 rules failed
    engine started.
    
    • GeoLite2 IP DB downloaded fine with license key :
    ls -l /usr/local/share/suricata/GeoLite2
    total 4000
    -rw-r--r--  1 root  wheel  4044800 Jan  8 10:37 GeoLite2-Country.mmdb
    
    • Geoip lib :
    suricata --build-info
    GeoIP support: yes, libmaxminddb
    

    Thanks in advance !
    best regards.

    Do your custom GeoIP rules by chance contain the noalert; option? I would not think so, but that is the only thing that changed in the package code logic with this release.

    Will you post one or two of your GeoIP rules and let me take a look (and test them myself)?



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    Do your custom GeoIP rules by chance contain the noalert; option? I would not think so, but that is the only thing that changed in the package code logic with this release.

    Will you post one or two of your GeoIP rules and let me take a look (and test them myself)?

    My Geoip does not contain the noalert; option

    This is a simple Geoip rule :

    drop ip any any -> any any (msg:"ASIA GEOIP"; geoip:AA,AF,AM,AZ,BH,BD,BT,IO,IN,BN,KH,KR,CX,CC,GE,JO,HK,ID,IR,IQ,KZ,KW,KG,LA,LB,LK,MO,MV,MN,MM,NP,KP,OM,PK,PH,SA,SY,TJ,TH,TR,TM,UZ,VN,YE; classtype:bad-unknown; sid:9900057; rev:1;)
    

    In the previous version of suricata, the rule match, drop and log.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    Do your custom GeoIP rules by chance contain the noalert; option? I would not think so, but that is the only thing that changed in the package code logic with this release.

    Will you post one or two of your GeoIP rules and let me take a look (and test them myself)?

    My Geoip does not contain the noalert; option

    This is a simple Geoip rule :

    drop ip any any -> any any (msg:"ASIA GEOIP"; geoip:AA,AF,AM,AZ,BH,BD,BT,IO,IN,BN,KH,KR,CX,CC,GE,JO,HK,ID,IR,IQ,KZ,KW,KG,LA,LB,LK,MO,MV,MN,MM,NP,KP,OM,PK,PH,SA,SY,TJ,TH,TR,TM,UZ,VN,YE; classtype:bad-unknown; sid:9900057; rev:1;)
    

    In the previous version of suricata, the rule match, drop and log.

    I am looking into this. It does not work for me either, but I made no GeoIP related changes in the binary at all. The only change was to provide a license key when downloading the MaxMind DB. I am reasonably sure that is unrelated to GeoIP rules not working. I'm beginning to wonder if it is a change from upstream in the binary. A different person did post a patch on Suricata upstream for the GeoIP area in the configure.ac file that was accepted into version 4.1.6. I'm investigating that change to see if if it might have an unanticipated consequence.

    It will take me a few minutes to accomplish, but I will create a version 4.1.5 binary and test it with the current GUI code. That will tell me if the issue is coming from Suricata upstream or not.



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    Do your custom GeoIP rules by chance contain the noalert; option? I would not think so, but that is the only thing that changed in the package code logic with this release.

    Will you post one or two of your GeoIP rules and let me take a look (and test them myself)?

    My Geoip does not contain the noalert; option

    This is a simple Geoip rule :

    drop ip any any -> any any (msg:"ASIA GEOIP"; geoip:AA,AF,AM,AZ,BH,BD,BT,IO,IN,BN,KH,KR,CX,CC,GE,JO,HK,ID,IR,IQ,KZ,KW,KG,LA,LB,LK,MO,MV,MN,MM,NP,KP,OM,PK,PH,SA,SY,TJ,TH,TR,TM,UZ,VN,YE; classtype:bad-unknown; sid:9900057; rev:1;)
    

    In the previous version of suricata, the rule match, drop and log.

    I am looking into this. It does not work for me either, but I made no GeoIP related changes in the binary at all. The only change was to provide a license key when downloading the MaxMind DB. I am reasonably sure that is unrelated to GeoIP rules not working. I'm beginning to wonder if it is a change from upstream in the binary. A different person did post a patch on Suricata upstream for the GeoIP area in the configure.ac file that was accepted into version 4.1.6. I'm investigating that change to see if if it might have an unanticipated consequence.

    It will take me a few minutes to accomplish, but I will create a version 4.1.5 binary and test it with the current GUI code. That will tell me if the issue is coming from Suricata upstream or not.

    Ok, thank you very much for your investigations !



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    Do your custom GeoIP rules by chance contain the noalert; option? I would not think so, but that is the only thing that changed in the package code logic with this release.

    Will you post one or two of your GeoIP rules and let me take a look (and test them myself)?

    My Geoip does not contain the noalert; option

    This is a simple Geoip rule :

    drop ip any any -> any any (msg:"ASIA GEOIP"; geoip:AA,AF,AM,AZ,BH,BD,BT,IO,IN,BN,KH,KR,CX,CC,GE,JO,HK,ID,IR,IQ,KZ,KW,KG,LA,LB,LK,MO,MV,MN,MM,NP,KP,OM,PK,PH,SA,SY,TJ,TH,TR,TM,UZ,VN,YE; classtype:bad-unknown; sid:9900057; rev:1;)
    

    In the previous version of suricata, the rule match, drop and log.

    I am looking into this. It does not work for me either, but I made no GeoIP related changes in the binary at all. The only change was to provide a license key when downloading the MaxMind DB. I am reasonably sure that is unrelated to GeoIP rules not working. I'm beginning to wonder if it is a change from upstream in the binary. A different person did post a patch on Suricata upstream for the GeoIP area in the configure.ac file that was accepted into version 4.1.6. I'm investigating that change to see if if it might have an unanticipated consequence.

    It will take me a few minutes to accomplish, but I will create a version 4.1.5 binary and test it with the current GUI code. That will tell me if the issue is coming from Suricata upstream or not.

    Ok, thank you very much for your investigations !

    It's getting more puzzling. What Suricata version were you running that was working prior to this latest 4.1.6 update? Were you running 4.1.5, or something even older?

    I have tried the 4.1.6 binary, the 4.1.5 binary, and now I'm trying the 4.1.4 binary and none are working for me. I'm beginning to wonder if the actual libmaxminddb library has changed. The next step is to compile a debug version of the Suricata binary and start stepping through the code to see what's changed.

    I did find that the new GeoIP2 download code that uses the license key is unzipping a database with an error in it, but in testing I got around that by copying over a known good database and running the mmdblookup utility. So I know I have a valid database, but even with that, the Suricata binary is not triggering the GeoIP rules.



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    Do your custom GeoIP rules by chance contain the noalert; option? I would not think so, but that is the only thing that changed in the package code logic with this release.

    Will you post one or two of your GeoIP rules and let me take a look (and test them myself)?

    My Geoip does not contain the noalert; option

    This is a simple Geoip rule :

    drop ip any any -> any any (msg:"ASIA GEOIP"; geoip:AA,AF,AM,AZ,BH,BD,BT,IO,IN,BN,KH,KR,CX,CC,GE,JO,HK,ID,IR,IQ,KZ,KW,KG,LA,LB,LK,MO,MV,MN,MM,NP,KP,OM,PK,PH,SA,SY,TJ,TH,TR,TM,UZ,VN,YE; classtype:bad-unknown; sid:9900057; rev:1;)
    

    In the previous version of suricata, the rule match, drop and log.

    I am looking into this. It does not work for me either, but I made no GeoIP related changes in the binary at all. The only change was to provide a license key when downloading the MaxMind DB. I am reasonably sure that is unrelated to GeoIP rules not working. I'm beginning to wonder if it is a change from upstream in the binary. A different person did post a patch on Suricata upstream for the GeoIP area in the configure.ac file that was accepted into version 4.1.6. I'm investigating that change to see if if it might have an unanticipated consequence.

    It will take me a few minutes to accomplish, but I will create a version 4.1.5 binary and test it with the current GUI code. That will tell me if the issue is coming from Suricata upstream or not.

    Ok, thank you very much for your investigations !

    It's getting more puzzling. What Suricata version were you running that was working prior to this latest 4.1.6 update? Were you running 4.1.5, or something even older?

    I have tried the 4.1.6 binary, the 4.1.5 binary, and now I'm trying the 4.1.4 binary and none are working for me. I'm beginning to wonder if the actual libmaxminddb library has changed. The next step is to compile a debug version of the Suricata binary and start stepping through the code to see what's changed.

    I did find that the new GeoIP2 download code that uses the license key is unzipping a database with an error in it, but in testing I got around that by copying over a known good database and running the mmdblookup utility. So I know I have a valid database, but even with that, the Suricata binary is not triggering the GeoIP rules.

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    It's a DB file problem,
    i have removed the suricata DB file downloaded with license key :

    cd /usr/local/share/suricata/GeoLite2/
    rm GeoLite2-Country.mmdb
    

    Then i created a link from pfblokerng DB (DB version is from december) to the suricata Geolite directory :

    ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    

    Then i restarted suricata and Geoip rules working good.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

    Sorry, I've edited my last post, see above, i think, it's a DB file problem.
    Geoip rules working good with Pfblokerng DB file from december DB.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

    I've edited my last post, see above, i think, it's a DB file problem

    Thanks for the update. I will change direction in my investigation and see what's wrong with the new database that is being downloaded. Maybe it is not getting unzipped properly or something. Fixing that will be much easier than chasing down a binary issue.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

    Sorry, I've edited my last post, see above, i think, it's a DB file problem.
    Geoip rules working good with Pfblokerng DB file from december DB.

    Okay, I've got this sorted out and will submit a fix for it soon.

    The root cause is the new database gzip archive has an extra sub-directory path in it where the actual database is stored. My PHP code was not allowing for that and thus wound up copying a corrupted database over to the shared area where Suricata was looking.

    I also shot myself in the foot when investigating the database because I initially was using my own custom rule with a known IP address from Japan for testing. However, later during my testing, I switched over to using your custom rule in my test setup, but I did not notice until MUCH later that your rule does not contain the JP country code for Japan. Thus my tests using the JP IP address were all still failing, even with a "good" database in place. That false result sent me down the path of suspecting the binary ... 😖. When you said copying over the pfBlockerNG database fixed it for you, I went back and carefully checked my testing methodology and discovered the issue with the IP address I was using not actually being covered in your GeoIP rule! Feel really stupid now...



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

    Sorry, I've edited my last post, see above, i think, it's a DB file problem.
    Geoip rules working good with Pfblokerng DB file from december DB.

    Okay, I've got this sorted out and will submit a fix for it soon.

    The root cause is the new database gzip archive has an extra sub-directory path in it where the actual database is stored. My PHP code was not allowing for that and thus wound up copying a corrupted database over to the shared area where Suricata was looking.

    I also shot myself in the foot when investigating the database because I initially was using my own custom rule with a known IP address from Japan for testing. However, later during my testing, I switched over to using your custom rule in my test setup, but I did not notice until MUCH later that your rule does not contain the JP country code for Japan. Thus my tests using the JP IP address were all still failing, even with a "good" database in place. That false result sent me down the path of suspecting the binary ... 😖. When you said copying over the pfBlockerNG database fixed it for you, I went back and carefully checked my testing methodology and discovered the issue with the IP address I was using not actually being covered in your GeoIP rule! Feel really stupid now...

    That's a good news !
    Thank you very much bmeeks ! 👍

    Oh my god ! 😕
    you're going around in circles just for my missing country code in my rule.
    Sorry for this mistake. 😖

    I'm waiting the update, thank's for all bmeeks !
    Best regards.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Prior to 4.1.6 i'm using the 4.1.5 version and all Geoip rules working fine to me.

    Thanks. I'm looking into what the problem could be. Creating a debug build now so I can trace the actual code execution in the binary as it attempts a geoip lookup.

    Sorry, I've edited my last post, see above, i think, it's a DB file problem.
    Geoip rules working good with Pfblokerng DB file from december DB.

    Okay, I've got this sorted out and will submit a fix for it soon.

    The root cause is the new database gzip archive has an extra sub-directory path in it where the actual database is stored. My PHP code was not allowing for that and thus wound up copying a corrupted database over to the shared area where Suricata was looking.

    I also shot myself in the foot when investigating the database because I initially was using my own custom rule with a known IP address from Japan for testing. However, later during my testing, I switched over to using your custom rule in my test setup, but I did not notice until MUCH later that your rule does not contain the JP country code for Japan. Thus my tests using the JP IP address were all still failing, even with a "good" database in place. That false result sent me down the path of suspecting the binary ... 😖. When you said copying over the pfBlockerNG database fixed it for you, I went back and carefully checked my testing methodology and discovered the issue with the IP address I was using not actually being covered in your GeoIP rule! Feel really stupid now...

    That's a good news !
    Thank you very much bmeeks ! 👍

    Oh my god ! 😕
    you're going around in circles just for my missing country code in my rule.
    Sorry for this mistake. 😖

    I'm waiting the update, thank's for all bmeeks !
    Best regards.

    It was my fault for not double-checking the IP. It just did not cross my mind. I chose the IP initially anyway, and then did not verify that your rule covered the IP I had arbitrarily chosen. My fault all the way ... ☺ .

    There is a happy ending, though. It gives me a chance to make the GeoIP database download script more robust. Also have it checking the posted MD5 hash on the MaxMind site against what is already present on the firewall so that it only downloads a new database when there is a mismatch. I'm going to change the check for a new database version back to once per day since it will only be downloading and checking the 32 byte MD5 file. If the MD5 on the firewall differs from the posted MD5 on the MaxMind site, then it will download the entire database (approximately 4 MB).



  • The problems with GeoIP database downloads and loss of GeoIP functionality in the Suricata 4.1.6 package have been identified and corrected. Look for an update to 4.1.6_2 to show up in the near future. Here is a link to the pull request containing the fix: https://github.com/pfsense/FreeBSD-ports/pull/749.



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    The problems with GeoIP database downloads and loss of GeoIP functionality in the Suricata 4.1.6 package have been identified and corrected. Look for an update to 4.1.6_2 to show up in the near future. Here is a link to the pull request containing the fix: https://github.com/pfsense/FreeBSD-ports/pull/749.

    Thank's bmeeks for the pull request !



  • Updated 4.1.6_1 to 4.1.6_2 today (Removed and reinstalled package) and database was corrupted after extraction : 😕
    System log :

    [Suricata] A new GeoLite2-Country IP database is available.
    [Suricata] Downloading new GeoLite2-Country IP database...
    [Suricata] New GeoLite2-Country IP database gzip archive successfully downloaded.
    [Suricata] Extracting new GeoLite2-Country database from the archive...
    [Suricata] Moving new database to /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb...
    [Suricata] GeoLite2-Country database update completed.
    [Suricata] Cleaning up temp files after GeoLite2-Country database update.
    

    Suricata log :

    Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.
    

    Temp fix :
    Create a link from PfblockerNG Maxmind DB to the suricata Geolite directory :

    cd /usr/local/share/suricata/GeoLite2/
    rm GeoLite2*
    
    cd /usr/local/share/GeoIP
    ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    
    ls -l GeoLite2-Country.mmdb
    -rw-r--r--  2 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb
    


  • @jm1384 For me it works...no errors:

    suricata GeoLite.png

    Also checked the log here /var/log/suricata/suricata_[interface]/suricata.log, and found no errors.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Temp fix :

    If you are implementing a symlink to that file in another directory, that may be causing issues with Suricata unpacking and copying the database. Remove your symlink completely, clean out the Suricata GeoLite2 DB directory and then run this command from a shell prompt:

    php /usr/local/pkg/suricata/suricata_geoipupdate.php
    

    Then check the system log and the suricata.log file for the interface. You should see a successful download. Restart Suricata on the interface and it should be good. The 4.1.6_2 version of the Suricata package fixes the GeoLite2 database corruption issue. I tested it several times to be sure.



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Temp fix :

    If you are implementing a symlink to that file in another directory, that may be causing issues with Suricata unpacking and copying the database. Remove your symlink completely, clean out the Suricata GeoLite2 DB directory and then run this command from a shell prompt:

    php /usr/local/pkg/suricata/suricata_geoipupdate.php
    

    Then check the system log and the suricata.log file for the interface. You should see a successful download. Restart Suricata on the interface and it should be good. The 4.1.6_2 version of the Suricata package fixes the GeoLite2 database corruption issue. I tested it several times to be sure.

    Hi bmeeks, thank you !
    ok, let's go !
    After removed all db files in suricata DB directory :

    php /usr/local/pkg/suricata/suricata_geoipupdate.php
    

    Suricata log :

    Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.
    

    suricata package removed :

    >>> Removing pfSense-pkg-suricata... 
    Checking integrity... done (0 conflicting)
    Deinstallation has been requested for the following 1 packages (of 0 packages in the universe):
    
    Installed packages to be REMOVED:
    	pfSense-pkg-suricata-4.1.6_2
    
    Number of packages to be removed: 1
    [1/1] Deinstalling pfSense-pkg-suricata-4.1.6_2...
    Removing suricata components...
    Menu items... done.
    Services... done.
    Loading package instructions...
    [1/1] Deleting files for pfSense-pkg-suricata-4.1.6_2: .......... done
    Removing suricata components...
    Configuration... done.
    >>> Removing stale packages... done.
    Success
    

    directory /suricata/GeoLite2/ :

    ls /suricata/GeoLite2: No such file or directory
    ls /suricata: No such file or directory
    
    

    No directory, no files, no links.

    After reinstall suricata /usr/local/share/suricata/GeoLite2 :

    ls -lrT  GeoLite2/
    total 2084 
    -rw-r--r-- 1 root  wheel  32 Jan 19 20:56:35 2020 GeoLite2-Country.mmdb.tar.gz.md5
    -rw-r--r-- 1 root  wheel  2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb
    

    Just one link "1" for suricata GeoLite2-Country.mmdb after unzipped :
    DB time and day : Jan 19 20:56:37 2020

    Now i check the pfBlokerNG DB file :

    ls -l GeoLite2-Country.mmdb
    -rw-r--r--  1 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb
    

    Just one link "1".
    DB time and day : Jan 7 00:45:59 2020

    They are no symlink for this pfblockerNG DB file,
    this DB file is different to suricata DB in days and times.

    Suricata log after install :

    Failed to open GeoIP2 database: /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb. Error was: The MaxMind DB file contains invalid metadata.  GeoIP rule matching is disabled.
    

    Ok, no problem, let's do the trick.
    After removing all suricata DB files in /usr/local/share/suricata/GeoLite2/ :

    rm GeoLite2*
    ls 
    

    I create a physical link from pfblockerNG DB file to suricata DB directory :

    cd /usr/local/share/GeoIP
    ln GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    

    Now, the pfBlokerNG DB has two links (2):

    ls -l GeoLite2-Country.mmdb 
    -rw-r--r-- 2 root  wheel  4035535 Jan  7 00:45 GeoLite2-Country.mmdb
    

    Suricata log after restart :

    0 rules failed
    engine started.
    

    GeoIP rules working fine.



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    -rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

    From the size I guessed that the mmdb is still in a tar format :

    tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
    -rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
    -rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
    -rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb
    


  • @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    -rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

    From the size I guessed that the mmdb is still in a tar format :

    tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
    -rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
    -rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
    -rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb
    

    This is your suricata V4.1.6_2 db file size ?



  • Yes

    ls -al /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    -rw-r--r--  1 root  wheel  2076656 Jan 18 18:51 /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    


  • @RonpfS
    ok, i will try to remove Suricata completely, package and backup config and reinstall it.

    Thank's RonpfS 👍



  • @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

    Yes

    ls -al /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    -rw-r--r--  1 root  wheel  2076656 Jan 18 18:51 /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    

    Your suricata db file zise is the same as mine after downloading,
    do you have the same error log in suricata log for Geoip db ?



  • @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Your suricata db file zise is the same as mine after downloading,

    Yes it comes from the same server. So GeoLite2-Country_.mmdb is in fact GeoLite2-Country_20200114.mmdb.tar.gz



  • @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    Your suricata db file zise is the same as mine after downloading,

    Yes it comes from the same server. So GeoLite2-Country.mmdb is in fact GeoLite2-Country.mmdb.tar.gz

    you have the same problem as me with this db file ?



  • Yes it look the same. So until @bmeeks find what's wrong, disable GeoIP update in Suricata use the pfblockerNG one

    mv /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb.tar.gz
    ln -s /usr/local/share/GeoIP/GeoLite2-Country.mmdb /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    


  • @RonpfS said in Suricata v4.1.6_1 - Package update Release Notes:

    @jm1384 said in Suricata v4.1.6_1 - Package update Release Notes:

    -rw-r--r-- 1 root wheel 2076656 Jan 19 20:56:37 2020 GeoLite2-Country.mmdb

    From the size I guessed that the mmdb is still in a tar format :

    tar -tvf /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb
    drwxr-xr-x  0 0      0           0 Jan 15 09:22 GeoLite2-Country_20200114/
    -rw-r--r--  0 0      0         398 Jan 15 09:22 GeoLite2-Country_20200114/LICENSE.txt
    -rw-r--r--  0 0      0          55 Jan 15 09:22 GeoLite2-Country_20200114/COPYRIGHT.txt
    -rw-r--r--  0 0      0     4083997 Jan 15 09:22 GeoLite2-Country_20200114/GeoLite2-Country.mmdb
    

    ok 👍
    your are right about untar archive, wait and see if @bmeeks can resolve this issue.



  • I promise this was working correctly when I tested prior to submitting the pull request. Let me do a fresh install in a test VM to see what's happening.



  • @bmeeks I upgraded on Jan 18.
    To be on the safe side, I uninstalled and installed 1 hour ago. same results.

    total 1994
    drwxr-xr-x  2 root  wheel        4 Jan 19 18:25 .
    drwxr-xr-x  4 root  wheel        4 Jan 19 18:25 ..
    -rw-r--r--  1 root  wheel  2076656 Jan 19 18:25 GeoLite2-Country.mmdb
    -rw-r--r--  1 root  wheel       32 Jan 19 18:25 GeoLite2-Country.mmdb.tar.gz.md5
    


  • I screwed the new code up. Working on correcting it. I don't know what I tested, but it did work. Must be losing my mind ... 😞 .

    Will get a correction posted soon.



  • Okay. Sorry about the previous screw-up with the GeoIP database. The new fix is posted here for the pfSense team to review and merge. If you want to make the changes yourself in your file before the fix is posted, you can look at the edits in the linked pull request.

    Look for a package update to version 4.1.6_3 in the near future.

    I don't even have a good lie to use to try and cover this one up. I will just have to own the mistake up front ... ☺



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    If you want to make the changes yourself in your file before the fix is posted,

    Just did the test and the DB is extracted ok now

    rm /usr/local/share/suricata/GeoLite2/GeoLite2-Country.mmdb*
    php /usr/local/pkg/suricata/suricata_geoipupdate.php
    
    ls -al
    total 2
    drwxr-xr-x  2 root  wheel        4 Jan 19 20:56 .
    drwxr-xr-x  4 root  wheel        4 Jan 19 18:25 ..
    -rw-r--r--  1 root  wheel  4083997 Jan 19 20:56 GeoLite2-Country.mmdb
    -rw-r--r--  1 root  wheel       32 Jan 19 20:56 GeoLite2-Country.mmdb.tar.gz.md5
    

    👍



  • @bmeeks said in Suricata v4.1.6_1 - Package update Release Notes:

    If you want to make the changes yourself in your file before the fix is posted, you can look at the edits in the linked pull request.

    same as @RonpfS , the fix working good @bmeeks 👍
    Thank you !


Log in to reply