Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfBlockerNG-devel 3.2.0_18

    Scheduled Pinned Locked Moved pfBlockerNG
    23 Posts 7 Posters 3.1k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • J
      jrey @BBcan177
      last edited by

      @BBcan177 said in pfBlockerNG-devel 3.2.0_18:

      there was too many moving parts with all the different branches of the code.

      i get it --

      again just providing the information as heads up for those that might get caught by this.

      1 Reply Last reply Reply Quote 0
      • J jrey referenced this topic on
      • D
        dotsch
        last edited by

        got a lot of errors in pfblocker.log : "ASN Token not defined. Terminating Download." and web gui is not accessable anymore.

        J 1 Reply Last reply Reply Quote 0
        • J
          jrey @dotsch
          last edited by

          @dotsch

          did you setup the ipinfo.io key ?

          ?Screen Shot 2024-10-03 at 9.18.42 AM.png

          when you say the web gui is not acccessable anymore - so the dashboard?

          can you go directly to any other page ?
          https://(your ip/name here)/pfblockerng/pfblockerng_ip.php

          can you ssh in and see if the asn data has been downloaded.

          cat /var/log/pfblockerng/extras.log
          

          or if you can get to other webpages --
          https://(your ip/name here)/pfblockerng/pfblockerng_log.php
          then pick logs, extras.log

          should look like (showing my last three days only) :

          Download Process Starting [ 10/1/24 07:45:00 ]
           /usr/local/share/GeoIP/asn.mmdb		200 OK
           /usr/local/share/GeoIP/asn.csv.gz		200 OK
          ASN Lookup Table has been updated [ 10/01/24 07:45:04 ]
          Download Process Ended [ 10/1/24 07:45:07 ]
          
          
          Download Process Starting [ 10/2/24 07:45:00 ]
           /usr/local/share/GeoIP/asn.mmdb		200 OK
           /usr/local/share/GeoIP/asn.csv.gz		200 OK
          ASN Lookup Table has been updated [ 10/02/24 07:45:05 ]
          Download Process Ended [ 10/2/24 07:45:07 ]
          
          
          Download Process Starting [ 10/3/24 07:45:00 ]
           /usr/local/share/GeoIP/asn.mmdb		200 OK
           /usr/local/share/GeoIP/asn.csv.gz		200 OK
          ASN Lookup Table has been updated [ 10/03/24 07:45:04 ]
          Download Process Ended [ 10/3/24 07:45:06 ]
          
          D 1 Reply Last reply Reply Quote 1
          • E
            emikaadeo
            last edited by emikaadeo

            Hi,
            anyone try to activate new Spamhaus URL feed under PRI1 Group?
            https://www.spamhaus.org/drop/drop_v4.json

            I get this:

            [ Spamhaus_Drop_v4 ]		 Downloading update .. 200 OK
            [PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjason|0]
            
             [ pfB_PRI1_v4 - Spamhaus_Drop_v4 ] Download FAIL [ 10/3/24 15:15:46 ]
              DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.
            
              Restoring previously downloaded file contents... completed ..
            
            J 1 Reply Last reply Reply Quote 0
            • D
              dotsch @jrey
              last edited by dotsch

              @jrey said in pfBlockerNG-devel 3.2.0_18:

              g

              thank you.

              web gui is time out.
              No, had not, not added ipinfo accounted as I expected that there is any error handling...

              Got the following logs...
              Downloading [ IPinfo databases ] [ 10/03/24 15:26:51 ]
              Downloading [ IPinfo databases ] [ 10/03/24 15:26:54 ]
              ..
              Downloading [ IPinfo databases ] [ 10/03/24 15:27:32 ]
              Downloading [ IPinfo databases ] [ 10/03/24 15:27:36 ]
              Downloading [ IPinfo databases ] [ 10/03/24 15:27:40 ]
              ..

              EDIT:
              GUI is avialable, try to recover....
              IP Info Lookups was enabled by default - Ups, expected not so...

              J 1 Reply Last reply Reply Quote 0
              • J
                jrey @dotsch
                last edited by

                @dotsch

                So you need to register with ipinfo get the Token and enter it on the GUI
                (what page was timing out -- the dashboard?)

                also read the second, or my post here:

                https://forum.netgate.com/topic/190009/pfblockerng-devel-v3-2-0_15

                regarding the change regarding the setting for ASN Reporting which must be enabled now.
                The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

                _18 is really the new _15 everything in between was a dream ..

                D 1 Reply Last reply Reply Quote 0
                • D
                  dotsch @jrey
                  last edited by

                  Thank you very much!

                  Got it back and everything is working again.

                  There was some few problems...

                  The ASN Reporing was enabled by default. Got a ton of errors in pfblocker.log, extra logs due the permantent download. System was to bussy, as web GUI timed out with "504 Gateway Time-out" and login was not possible.
                  I expexted that I can add the missing key later.

                  Next problem was, that ipinfo is blocked within snort:
                  1:2025331 ET POLICY Possible External IP Lookup Domain Observed in SNI (ipinfo. io)
                  1:2054168 ET INFO External IP Lookup Domain in DNS Lookup (ipinfo .io)
                  Both was default the snort "connectivity" rules.

                  1 Reply Last reply Reply Quote 1
                  • J
                    jrey @emikaadeo
                    last edited by

                    @emikaadeo

                    I can't comment on this because that list has been json for a while, not specifically related to or because of _18. The change was announced that the txt was being discontinued some time back. The only change is that now the link provided in pfblockerng points to the json not the txt and as you can see doesn't work. there are other threads on this already (so you are not alone), and now that the .txt version is returning a 404 (the discontinuation is complete)

                    preemptively however, when that announcement was made, my solution was not wait and simply download the json file on another system using a scheduled script

                    then process the json on that system to create the txt file format,
                    and then just have pfblocker pull the feed as a txt from from there ..

                    Been working fine for months or since the day the announcement was made.

                    that script looks like --

                    echo -e "-- Spamhaus Drop"
                    uri="https://www.spamhaus.org/drop/drop_v4.json"
                    lfile="drop_v4.json"
                    bkfile="drop_v4.orig"
                    mv "$lfile" "$bkfile"
                    if test -e "$lfile"
                    then zflag="-z '$lfile'"
                    else zflag=
                    fi
                    http_code=$(curl -w "%{http_code}\n" --output "$lfile" $zflag "$uri" --silent)
                    echo -e "$http_code"
                    if [ $http_code == "200" ]; then
                       echo -e "-- Process File"
                       cat "$lfile" | jq .cidr | tr -d '"' | tr -d 'null' > drop_v4.txt
                       echo -e "-- drop_v4.txt created"
                    else
                       echo -e "-- Restore Backup"
                       mv "$bkfile" "$lfile"
                    fi
                    echo -e "--Done"
                    # end of file.
                    

                    then I just pull the txt version of the file from the system that script runs on
                    Screen Shot 2024-10-03 at 10.01.49 AM.png

                    pfblocker can deal with txt files nicely - and if the format changes nothing needs to be changed in pfblocker.

                    so for me moving the source download away from pfblockerng - put it where I have more control and still provides the same data needed without the hassle or sitting around waiting for the world to change.

                    I use similar techniques for building my own GeoIP (not using maxmind at all) and my geoip data is far more granular and updated once daily not every 3 or 4 days or whatever maxmind is using for their free level these days. My GeoIP data is all pulled and processed on the other system. Then pfblocker just gets a txt file reference to the locally hosted system. (the added "bonus" here is that I can point multiple internal systems to the internal data, and I'm not downloading the same thing from source multiple times. just once from external)

                    I'm sure it will get fixed, either by patch or new code at some point.

                    Meanwhile, there is always a way.

                    E 1 Reply Last reply Reply Quote 0
                    • E
                      emikaadeo @jrey
                      last edited by emikaadeo

                      @jrey said in pfBlockerNG-devel 3.2.0_18:

                      The only change is that now the link provided in pfblockerng points to the json not the txt and as you can see doesn't work

                      I see this now https://www.patreon.com/posts/pfblockerng-v3-2-111306340
                      Zrzut ekranu 2024-10-3 o 16.57.43.png
                      Well, just wondering - what's the point then for changing URL and download a .JSON file which pfBlockerNG can't handle ?
                      It can be confusing for users.

                      BBcan177B J 2 Replies Last reply Reply Quote 0
                      • BBcan177B
                        BBcan177 Moderator @emikaadeo
                        last edited by

                        @emikaadeo

                        Didn't catch that before. Will add a fix

                        You can manualy edit

                        /usr/local/pkg/pfblockerng/pfblockerng.inc

                        https://github.com/pfsense/FreeBSD-ports/blob/devel/net%2FpfSense-pkg-pfBlockerNG-devel%2Ffiles%2Fusr%2Flocal%2Fpkg%2Fpfblockerng%2Fpfblockerng.inc

                        After line 266, add this line:

                            'application/x-ndjason',
                        

                        "Experience is something you don't get until just after you need it."

                        Website: http://pfBlockerNG.com
                        Twitter: @BBcan177  #pfBlockerNG
                        Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                        E J 3 Replies Last reply Reply Quote 1
                        • J
                          jrey @emikaadeo
                          last edited by

                          @emikaadeo

                          That post you reference was made long after the announcement of the change by Spamhaus -

                          Because I moved past the whole json issue as described a long ago - I don't think I can answer your question except to say it is -devel code are there are issues - that's why it is devel and not production.

                          and as I'm typing this @BBcan177 has just provided you a "patch" - so there you go .

                          1 Reply Last reply Reply Quote 2
                          • J jrey referenced this topic on
                          • E
                            emikaadeo @BBcan177
                            last edited by

                            @BBcan177 said in pfBlockerNG-devel 3.2.0_18:

                            @emikaadeo

                            Didn't catch that before. Will add a fix

                            You can manualy edit

                            /usr/local/pkg/pfblockerng/pfblockerng.inc

                            https://github.com/pfsense/FreeBSD-ports/blob/devel/net%2FpfSense-pkg-pfBlockerNG-devel%2Ffiles%2Fusr%2Flocal%2Fpkg%2Fpfblockerng%2Fpfblockerng.inc

                            After line 266, add this line:

                                'application/x-ndjason',
                            

                            Hi @BBcan177
                            I edit file manually, then:

                            Firewall -> pfBlockerNG -> Update ->  Select 'Force' option -> Update
                            

                            Logs looks ok now:

                            [ Spamhaus_Drop_v4 ]		 Downloading update .. 200 OK. completed ..
                            
                            1 Reply Last reply Reply Quote 0
                            • J
                              jrey @BBcan177
                              last edited by jrey

                              @BBcan177

                              The default Talos link now provided (which I think is the same as before) is also now wanting to make a stop through a terms and conditions page. Which of course pfb is not "accepting" and therefore you get

                              [ Talos_BL_v4 ]			 Downloading update .. 200 OK. completed ..
                              [ pfB_PRI1_v4 Talos_BL_v4 ] No IPs found! Ensure only IP based Feeds are used! ]
                              

                              Tracked this post down from 7 days ago, it has no responses, but pretty much sums up what is going on.

                              https://forum.netgate.com/topic/190285/changes-to-snort-org-talos-intel-ip-block-list-affecting-pfblockerng?_=1727956822880

                              1 Reply Last reply Reply Quote 0
                              • T
                                tman222
                                last edited by

                                For some reason in my situation ASN information from IPInfo isn't even attempting to download after enabling the ASN setting and entering a token. Could I be missing a setting somewhere?

                                https://forum.netgate.com/topic/190395/pfblockerng-devel-not-downloading-asn-information

                                J 1 Reply Last reply Reply Quote 0
                                • J
                                  jrey @tman222
                                  last edited by

                                  @tman222

                                  so this maybe.

                                  what is in your extras.log ? that is where the download is logged --

                                  @jrey said in pfBlockerNG-devel 3.2.0_18:

                                  regarding the change regarding the setting for ASN Reporting which must be enabled now.
                                  The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

                                  T 1 Reply Last reply Reply Quote 0
                                  • T
                                    tman222 @jrey
                                    last edited by

                                    @jrey said in pfBlockerNG-devel 3.2.0_18:

                                    @tman222

                                    so this maybe.

                                    what is in your extras.log ? that is where the download is logged --

                                    @jrey said in pfBlockerNG-devel 3.2.0_18:

                                    regarding the change regarding the setting for ASN Reporting which must be enabled now.
                                    The old feed would download even if the setting was disabled, the new ones won't even though ASN reporting means something else. Enable 1 hour

                                    Thanks @jrey for the response. I'll continue the conversation regarding this issue in the other thread I posted.

                                    1 Reply Last reply Reply Quote 0
                                    • J jrey referenced this topic on
                                    • J
                                      jrey @BBcan177
                                      last edited by

                                      @BBcan177

                                      after going to _20

                                      isn't jason is a horror movie reference ?

                                      as it appears in the _20 release code as
                                      'application/x-ndjason',

                                      The list fails
                                      Spamhaus_Drop_v4 ] Downloading update .. 200 OK
                                      [PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjson|0]

                                      now s/b json the correct way.

                                      'application/x-ndjson',

                                      if it was working with ndjason (and it was for a while), seems they may have fixed a typo upstream
                                      previously it was:
                                      [ Spamhaus_Drop_v4 ] Downloading update .. 200 OK
                                      [PFB_FILTER - 17] Failed or invalid Mime Type: [application/x-ndjason|0]

                                      1 Reply Last reply Reply Quote 0
                                      • First post
                                        Last post
                                      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.