4. Update lists fail

Update lists fail

  • Q

    I am getting more fails on the lists download in pfBLockerNG 2.1.2_3, anyone experiencing the same?

    [DNSBL_BBcan177 - MS_2 ] Download FAIL [ 05/07/18 00:03:56 ] 
        [ DNSBL_Ads - Disconnect ] Download FAIL [ 05/07/18 00:03:17 ]
        [ DNSBL_Ads - WinTelemetry ] Download FAIL [ 05/07/18 00:03:00 ]
        [ DNSBL_Ads - Quidsup ] Download FAIL [ 05/07/18 00:02:42 ]
        [ DNSBL_Ads - Disconnect_Ad_Filter ] Download FAIL [ 05/07/18 00:02:25 ]
        [ DNSBL_Ads - Disconnect_Trackers ] Download FAIL [ 05/07/18 00:02:08 ]
        [ DNSBL_Ads - PiHole_StevenBlack ] Download FAIL [ 05/07/18 00:01:43 ]

    When I look in the log I see, they are mostly related to SSL certificate problem

    [ WinTelemetry ] Downloading update [ 05/07/18 00:02:45 ] . cURL Error: 60
    SSL certificate problem: unable to get local issuer certificate Retry in 5 seconds…
    . cURL Error: 60
    SSL certificate problem: unable to get local issuer certificate Retry in 5 seconds...
    . cURL Error: 60
    SSL certificate problem: unable to get local issuer certificate Retry in 5 seconds...
    .. unknown http status code

    And that Maxmind is old...

    MaxMind GeoLite2 Date/Time Stamp
    Last-Modified: Mon, 06 Nov 2017 19:15:47 GMT
    Duplicate Represented IP4 Networks: 31851
    Duplicate Represented IP6 Networks: 3052

    0
  • K

    Is the date time correct on the pfsense box?

    0
  • Q

    @Kai_null:

    Is the date time correct on the pfsense box?

    Yup, the dashboard pfS reads "Current date/time Thu May 10 11:23:16 CEST 2018"

    btw what is your MaxMind from, here it reads in the dashboard "MaxMind: Last-Modified: Mon, 06 Nov 2017 19:15:47 GMT"

    0
  • K

    MaxMind: Last-Modified: Tue, 03 Apr 2018 22:41:00 GMT

    I am out of good ideas to help you :(.

    I did see this which I thought was interesting:
    https://serverfault.com/questions/681835/freebsd-pfsense-root-ca-fails

    Suggests to me your problem is not with pfblockerng but rather with openssl.  Perhaps it is unable to update its cert store?

    other ideas which may or may not help you:

    https://www.google.com/search?q=pfblockerng+"unable+to+get+local+issuer+certificate"&oq=pfblockerng+"unable+to+get+local+issuer+certificate"

    https://www.google.com/search?q=pfsense+"unable+to+get+local+issuer+certificate"&oq=pfsense+"unable+to+get+local+issuer+certificate"

    https://www.google.com/search?q=freebsd+"unable+to+get+local+issuer+certificate"&oq=pfsense+"unable+to+get+local+issuer+certificate"

    https://www.google.com/search?q=openssl+"unable+to+get+local+issuer+certificate"&oq=pfsense+"unable+to+get+local+issuer+certificate"

    wish I had answers for you.

    Curious to know what you eventually find.

    ps: I have found that a 2 am reinstall in some cases is a quicker bug fix then diving down the rabbit hole.  I guess it depends on how stock your config is.

    0
  • Moderator

    @Qinn

    All of those feeds are hosted on Github and/or Amazon, so I would assume that a DNSBL Feed or an IP Blocklist is blocking access on download…  Check the pfBlockerNG Alerts Tab...

    Might need to whitelist:

    raw.githubusercontent.com
s3.amazonaws.com

    or wildcard whitelist the whole domain

    .githubusercontent.com
.amazonaws.com

    For the MaxMind issue, from the pfSense box, check to see if you can access the MaxMind site:

    host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.37.47
geolite.maxmind.com has address 104.16.38.47

    Then try to ping the resulting IPs and get a reply.

    The MaxMind download errors are reported to the error.log file…

    Once you have fixed connectivity, you can manually download the MaxMind database with this command:

    php -f /usr/local/www/pfblockerng/pfblockerng.php dc
    0
  • Q

    @Kai_null:

    MaxMind: Last-Modified: Tue, 03 Apr 2018 22:41:00 GMT

    I am out of good ideas to help you :(.

    I did see this which I thought was interesting:
    https://serverfault.com/questions/681835/freebsd-pfsense-root-ca-fails

    Suggests to me your problem is not with pfblockerng but rather with openssl.  Perhaps it is unable to update its cert store?

    other ideas which may or may not help you:

    https://www.google.com/search?q=pfblockerng+"unable+to+get+local+issuer+certificate"&oq=pfblockerng+"unable+to+get+local+issuer+certificate"

    https://www.google.com/search?q=pfsense+"unable+to+get+local+issuer+certificate"&oq=pfsense+"unable+to+get+local+issuer+certificate"

    https://www.google.com/search?q=freebsd+"unable+to+get+local+issuer+certificate"&oq=pfsense+"unable+to+get+local+issuer+certificate"

    https://www.google.com/search?q=openssl+"unable+to+get+local+issuer+certificate"&oq=pfsense+"unable+to+get+local+issuer+certificate"

    wish I had answers for you.

    Curious to know what you eventually find.

    ps: I have found that a 2 am reinstall in some cases is a quicker bug fix then diving down the rabbit hole.  I guess it depends on how stock your config is.

    Thanks for your time and thoughts.

    cheers Qinn

    0
  • Q

    @BBcan177:

    @Qinn

    All of those feeds are hosted on Github and/or Amazon, so I would assume that a DNSBL Feed or an IP Blocklist is blocking access on download…  Check the pfBlockerNG Alerts Tab...

    Might need to whitelist:

    raw.githubusercontent.com
s3.amazonaws.com

    or wildcard whitelist the whole domain

    .githubusercontent.com
.amazonaws.com

    For the MaxMind issue, from the pfSense box, check to see if you can access the MaxMind site:

    host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.37.47
geolite.maxmind.com has address 104.16.38.47

    Then try to ping the resulting IPs and get a reply.

    The MaxMind download errors are reported to the error.log file…

    Once you have fixed connectivity, you can manually download the MaxMind database with this command:

    php -f /usr/local/www/pfblockerng/pfblockerng.php dc

    Thanks, I will try and report back.

    Cheers Qinn

    0
  • Q

    a```
    host -t A geolite.maxmind.com

    returns```
geolite.maxmind.com has address 10.10.10.1

    If I disable DNSBL I get

    host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.38.47
geolite.maxmind.com has address 104.16.37.47

    I added .geolite.maxmind.com to the Custom Domain Whitelist in DNSBL, but nothing changes, still I get the ip of the VIP, instead op 104.16.etc.

    0
  • Moderator

    @Qinn:

    a```
    host -t A geolite.maxmind.com

    returns```
geolite.maxmind.com has address 10.10.10.1

    If I disable DNSBL I get

    host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.38.47
geolite.maxmind.com has address 104.16.37.47

    I added .geolite.maxmind.com to the Custom Domain Whitelist in DNSBL, but nothing changes, still I get the ip of the VIP, instead op 104.16.etc.

    When you manually add a domain or an IP to a whitelist, you need to run a Reload to get it to apply the change or wait until the next cron run to execute… Alternatively, when you whitelist from the Alerts Tab, the whitelist/suppression will take effect immediately.

    0
  • Q

    Thanks yeah I was pulling my hair ;)  After I added it to the whitelist, I did a "Select 'Reload' option" All and hit run got a "exists" in return. Then I realized it and a reload solved it.

    Now

    host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.37.47
geolite.maxmind.com has address 104.16.38.47

    Now I works, btw I can do a manual update of MaxMind, but when will pfblockerNG do it automatically?

    Thanks BBCan177 I hope everything it going well with the next major release, thumbs up!!

    0
  • Moderator

    @Qinn:

    Thanks yeah I was pulling my hair ;)  After I added it to the whitelist, I did a "Select 'Reload' option" All and hit run got a "exists" in return. Then I realized it and a reload solved it.

    Now

    host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.37.47
geolite.maxmind.com has address 104.16.38.47

    Now I works, btw I can do a manual update of MaxMind, but when will pfblockerNG do it automatically?

    Thanks BBCan177 I hope everything it going well with the next major release, thumbs up!!

    MaxMind is updated once per month. You will see the cron task in pfSense for that…

    Thanks! Its been submitted and awaiting review by the devs...

    0
  • Q

    Thanks, did a manual update and worked like a charm.

     php -f /usr/local/www/pfblockerng/pfblockerng.php dc
Country code update Start
 Converting MaxMind Country databases for pfBlockerNG.
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 05/12/18 16:45:27 ]
 Creating pfBlockerNG Continent XML files
 IPv4 Africa                     [ 05/12/18 16:45:48 ]
 IPv6 Africa                     [ 05/12/18 16:45:49 ]
 IPv4 Antarctica                 [ 05/12/18 16:45:50 ]
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 05/12/18 16:45:59 ]
 IPv4 Europe                     [ 05/12/18 16:46:01 ]
 IPv6 Europe                     [ 05/12/18 16:46:30 ]
 IPv4 North America              [ 05/12/18 16:46:40 ]
 IPv6 North America              [ 05/12/18 16:47:01 ]
 IPv4 Oceania                    [ 05/12/18 16:47:04 ]
 IPv6 Oceania                    [ 05/12/18 16:47:06 ]
 IPv4 South America
 IPv6 South America              [ 05/12/18 16:47:08 ]
 IPv4 Proxy and Satellite        [ 05/12/18 16:47:10 ]
 IPv6 Proxy and Satellite
 IPv4 TOP 20
 IPv6 TOP 20
 pfBlockerNG Reputation Tab
Country Code Update Ended [ 05/12/18 16:47:11 ]

[2.4.3-RELEASE][root@pfSense.localdomain]/root:

    Just to be sure I added the MaxMind IP's to my IPV4 whitelist also.

    0
  • Q

    @BBcan177:

    MaxMind is updated once per month. You will see the cron task in pfSense for that…

    I did a```
    crontab -l

    
Cheers Qinn
    0

  • Did you leave MaxMind Updates unticked ?

    There is a Cron package you can install  ;)

    The job run from 4th to 10th of the month

    /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1
    0
  • Q

    @RonpfS:

    Did you leave MaxMind Updates unticked ?

    There is a Cron package you can install  ;)

    The job run from 4th to 10th of the month

    /usr/local/bin/php /usr/local/www/pfblockerng/pfblockerng.php dcc >> /var/log/pfblockerng/extras.log 2>&1

    Thanks for you reply RonpfS. Nope I haven't disabled it, I just thought a```
    crontab -l

    1

  • On my system I get :

    crontab -l

crontab: no crontab for root
```  :o
    0
  • Q

    So what does the MaxMind updates trigger?

    1
    Q
     1 Reply
  • Q

    @qinn Is there a way to check when the next update of MaxMind will run?

    1 1 Reply

  • @qinn You can check /var/log/pfblockerng/extras.log to see when the last update was done.

    1
  • Q

    Thanks, nice overview btw of the last 2 years, seems I have to wait for 11 juni (when updating comes once a month)

    0
  • Q

    Hmm it's been well over a month and as from what it looks in the dashboard MaxMind isn't updated, I looked in

    cat /var/log/pfblockerng/extras.log

    and it reads:

    Download Process Starting [ 06/05/18 07:00:00 ]
 /usr/local/share/GeoIP/GeoIP.dat.gz            200 OK
 /usr/local/share/GeoIP/GeoIPv6.dat.gz          200 OK
 /usr/local/share/GeoIP/GeoLite2-Country-CSV.zip                200 OK
 /var/db/pfblockerng/top-1m.csv.zip             200 OK
Download Process Ended [ 06/05/18 07:00:28 ]

Country code update Start
 Converting MaxMind Country databases for pfBlockerNG.
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 06/05/18 07:01:46 ]
 Creating pfBlockerNG Continent XML files
 IPv4 Africa                     [ 06/05/18 07:02:07 ]
 IPv6 Africa                     [ 06/05/18 07:02:08 ]
 IPv4 Antarctica
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 06/05/18 07:02:17 ]
 IPv4 Europe                     [ 06/05/18 07:02:19 ]
 IPv6 Europe                     [ 06/05/18 07:02:48 ]
 IPv4 North America              [ 06/05/18 07:02:58 ]
 IPv6 North America              [ 06/05/18 07:03:19 ]
 IPv4 Oceania                    [ 06/05/18 07:03:22 ]
 IPv6 Oceania                    [ 06/05/18 07:03:24 ]
 IPv4 South America
 IPv6 South America              [ 06/05/18 07:03:26 ]
 IPv4 Proxy and Satellite        [ 06/05/18 07:03:28 ]
 IPv6 Proxy and Satellite
 IPv4 TOP 20
 IPv6 TOP 20
 pfBlockerNG Reputation Tab
Country Code Update Ended [ 06/05/18 07:03:29 ]

    So it confirms that the last one, was the one I did manually, what goes wrong, why doesn't it update automatically once a month?

    Thnx for any advise

    Cheers Qinn

    0 1 Reply

  • @qinn said in Update lists fail:

    was the one I did manually, what goes wrong

    Nothing is wrong for now. You did a manual update on June 5th. The next automatic update will be done by Cron between the 4th and the 10th of July.

    BTW my last update was on 06/07/18, but the version is from May 1st

    /var/log/pfblockerng/maxmind_ver

    MaxMind GeoLite2 Date/Time Stamp
Last-Modified: Tue, 01 May 2018 16:48:16 GMT
Duplicate Represented IP4 Networks: 36994
    1
  • Q

    Thanks for you reply, always helpful, but I have to say nope. I hoped that to, but 10/15/16 08:20:12 (as you can see below) isn't possible so it's dd-mm-yy, so there is something wrong...

    
Download Process Starting [ 10/15/16 08:18:03 ]
 /usr/local/share/GeoIP/GeoIP.dat.gz            200 OK
 /usr/local/share/GeoIP/GeoIPv6.dat.gz          200 OK
 /usr/local/share/GeoIP/GeoLite2-Country-CSV.zip                200 OK
 /var/db/pfblockerng/top-1m.csv.zip             200 OK
Download Process Ended [ 10/15/16 08:18:40 ]

Country code update Start
 Converting MaxMind Country databases for pfBlockerNG.
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 10/15/16 08:19:49 ]
 Creating pfBlockerNG Continent XML files
 IPv4 Africa                     [ 10/15/16 08:20:03 ]
 IPv6 Africa                     [ 10/15/16 08:20:04 ]
 IPv4 Antarctica
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 10/15/16 08:20:12 ]
 IPv4 Europe                     [ 10/15/16 08:20:14 ]
 IPv6 Europe                     [ 10/15/16 08:20:40 ]
 IPv4 North America              [ 10/15/16 08:20:45 ]
 IPv6 North America              [ 10/15/16 08:21:02 ]
 IPv4 Oceania                    [ 10/15/16 08:21:05 ]
 IPv6 Oceania                    [ 10/15/16 08:21:07 ]
 IPv4 South America
 IPv6 South America              [ 10/15/16 08:21:09 ]
 IPv4 Proxy and Satellite        [ 10/15/16 08:21:10 ]
 IPv6 Proxy and Satellite
 IPv4 TOP 20
 IPv6 TOP 20                     [ 10/15/16 08:21:11 ]
 pfBlockerNG Reputation Tab
Country Code Update Ended


 Download Process Starting [ 05/12/18 16:43:36 ]
 /usr/local/share/GeoIP/GeoIP.dat.gz            200 OK
 /usr/local/share/GeoIP/GeoIPv6.dat.gz          200 OK
 /usr/local/share/GeoIP/GeoLite2-Country-CSV.zip                200 OK
 /var/db/pfblockerng/top-1m.csv.zip             200 OK
Download Process Ended [ 05/12/18 16:44:09 ]

Country code update Start
 Converting MaxMind Country databases for pfBlockerNG.
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 05/12/18 16:45:27 ]
 Creating pfBlockerNG Continent XML files
 IPv4 Africa                     [ 05/12/18 16:45:48 ]
 IPv6 Africa                     [ 05/12/18 16:45:49 ]
 IPv4 Antarctica                 [ 05/12/18 16:45:50 ]
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 05/12/18 16:45:59 ]
 IPv4 Europe                     [ 05/12/18 16:46:01 ]
 IPv6 Europe                     [ 05/12/18 16:46:30 ]
 IPv4 North America              [ 05/12/18 16:46:40 ]
 IPv6 North America              [ 05/12/18 16:47:01 ]
 IPv4 Oceania                    [ 05/12/18 16:47:04 ]
 IPv6 Oceania                    [ 05/12/18 16:47:06 ]
 IPv4 South America
 IPv6 South America              [ 05/12/18 16:47:08 ]
 IPv4 Proxy and Satellite        [ 05/12/18 16:47:10 ]
 IPv6 Proxy and Satellite
 IPv4 TOP 20
 IPv6 TOP 20
 pfBlockerNG Reputation Tab
Country Code Update Ended [ 05/12/18 16:47:11 ]


Download Process Starting [ 06/05/18 07:00:00 ]
 /usr/local/share/GeoIP/GeoIP.dat.gz            200 OK
 /usr/local/share/GeoIP/GeoIPv6.dat.gz          200 OK
 /usr/local/share/GeoIP/GeoLite2-Country-CSV.zip                200 OK
 /var/db/pfblockerng/top-1m.csv.zip             200 OK
Download Process Ended [ 06/05/18 07:00:28 ]

Country code update Start
 Converting MaxMind Country databases for pfBlockerNG.
 Processing ISO IPv4 Continent/Country Data
 Processing ISO IPv6 Continent/Country Data [ 06/05/18 07:01:46 ]
 Creating pfBlockerNG Continent XML files
 IPv4 Africa                     [ 06/05/18 07:02:07 ]
 IPv6 Africa                     [ 06/05/18 07:02:08 ]
 IPv4 Antarctica
 IPv6 Antarctica
 IPv4 Asia
 IPv6 Asia                       [ 06/05/18 07:02:17 ]
 IPv4 Europe                     [ 06/05/18 07:02:19 ]
 IPv6 Europe                     [ 06/05/18 07:02:48 ]
 IPv4 North America              [ 06/05/18 07:02:58 ]
 IPv6 North America              [ 06/05/18 07:03:19 ]
 IPv4 Oceania                    [ 06/05/18 07:03:22 ]
 IPv6 Oceania                    [ 06/05/18 07:03:24 ]
 IPv4 South America
 IPv6 South America              [ 06/05/18 07:03:26 ]
 IPv4 Proxy and Satellite        [ 06/05/18 07:03:28 ]
 IPv6 Proxy and Satellite
 IPv4 TOP 20
 IPv6 TOP 20
 pfBlockerNG Reputation Tab
Country Code Update Ended [ 06/05/18 07:03:29 ]
    0 1 Reply

  • @qinn said in Update lists fail:

    so it’s dd-mm-yy,

    The date is in mm-dd-yy format.

    0
    Q
     1 Reply
  • Q

    @ronpfs please take a look at the logging in extras.log above and below, that can't be as there is no 15 month ...🙃

    Download Process Ended [ 10/15/16 08:18:40 ]
    0 1 Reply

  • @qinn And you care about 2016 events ? 😜

    0
    Q
     1 Reply
  • Q

    @ronpfs ROFL ;)

    0
  • Q

    Somehow this won't go away, last Maxmind is 7 june and still pfBlockerNG did not update it. Is there anyway to restore the "automatic updates" or do I have to remove and reinstall the package?

    0_1532498621196_Maxmind.png

    0
  • Moderator

    Did you check to see if something is blocking access to MaxMind web site? Another possibility is that MaxMind is blocking your IP or rate limiting it for some reason...

    Does MaxMind resolve?

    host -t A geolite.maxmind.com

    Can you download from this URL in a browser window:

    https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz

    Did you check the pfBlockerNG error.log?

    If you run the updater script, do you get any errors?

    php /usr/local/www/pfblockerng/pfblockerng.php dc

    Try the latest pfBlockerNG-devel version and see if that solves your issue....

    1
    Q
     1 Reply
  • Q

    First a big thanks for you help and time, now to your advise, it gave the following results:

    1. Yes, MaxMind did resolve
    code[2.4.3-RELEASE][root@pfSense.localdomain]/root: host -t A geolite.maxmind.com
geolite.maxmind.com has address 104.16.37.47
geolite.maxmind.com has address 104.16.38.47

    1. Yes, I can download the file GeoIP.dat.gz

    2. I added the files

    /var/log/pfblockerng/extras.log and error.log

    0_1532592170771_pfblokcerNG.7z

    I don't see anything out of the ordinary in these two, but I'm not a dev 😉 The 2 last updatesof MaxMind that are mentioned in the logs, where the result of a manually update using

    code php -f /usr/local/www/pfblockerng/pfblockerng.php dc

    My best guess is that if the MaxMind update is a cron job, it's not working anymore, a:

    code crontab -l

    results in:

    code crontab: no crontab for root

    So my last straw would be changing the update settings to the latest development and installing it, but I don't like unsolved cases (sorry that's me 🤓 ) is there anything more I could look into, as I still don't get how MaxMind is updated as it doesn't seem to be a cron job?

    btw From what I've read, when I want to move over to the latest dev of pfblockerNG I have to first deinstall pfblockerNG and then install the dev version and all settings will be restored, is this the right way to move over to the dev version?

    Cheers Qinn

    0
  • K

    PfSense doesn't have user specific crontabs ( for reasons that I have hard time understanding myself) and the only crontab on the system is the /etc/crontab file.

    1
    Q
     1 Reply
  • Q

    @kpa Thanks for pointing that one out to me, awaiting BBcan177 advise on how to proceed, hoping he can do something with the results and logs I provided.

    0
  • Moderator

    @qinn said in Update lists fail:

    btw From what I've read, when I want to move over to the latest dev of pfblockerNG I have to first deinstall pfblockerNG and then install the dev version and all settings will be restored, is this the right way to move over to the dev version?

    Yes I would just go to devel and see if that fixes your issue.... There are a ton of new features in devel....
    https://www.reddit.com/r/PFSENSE/comments/8lnugz/pfblockerng_devel_version_released/

    Also install the "Cron" package for pfSense and you can see the pfSense Cron tasks from there.

    0
    Q
     1 Reply
  • Q

    @bbcan177 said in Update lists fail:

    @qinn said in Update lists fail:

    btw From what I've read, when I want to move over to the latest dev of pfblockerNG I have to first deinstall pfblockerNG and then install the dev version and all settings will be restored, is this the right way to move over to the dev version?

    Yes I would just go to devel and see if that fixes your issue.... There are a ton of new features in devel....
    https://www.reddit.com/r/PFSENSE/comments/8lnugz/pfblockerng_devel_version_released/

    Also install the "Cron" package for pfSense and you can see the pfSense Cron tasks from there.

    Thanks for your advise 👍 👍 hoping for a release soon, I don't care about an ETA, take your time and keep up the good work.

    0
 

Products

Applications

Support

Services

News

Resources

Company

Our Mission

As host of the pfSense open source firewall project, Netgate believes in enhancing network connectivity that maintains both security and privacy. We also believe everyone should be able to afford it.

Subscribe to our Newsletter

Product information, software announcements, and special offers. See our newsletter archive for past announcements.

© Copyright 2002 - 2018 Rubicon Communications, LLC | Privacy Policy