Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    pfBlockerNG ASN downloads only contain a header

    pfBlockerNG
    10
    70
    11.0k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Bob.DigB
      Bob.Dig LAYER 8 @jrey
      last edited by

      @jrey said in pfBlockerNG ASN downloads only contain a header:

      if contains IP addresses all good on the other hand if it only contains 1 lines (the header) bad.

      They are all only containing something like this:

      ### AS16232: AS16232 ###
      

      Capture.PNG

      And I have problems with other lists as well like FireHOL and maybe many more.

      J 1 Reply Last reply Reply Quote 0
      • J
        jrey @Bob.Dig
        last edited by

        @Bob-Dig

        Exactly - just a header and no IP addresses in the original file.
        Now I am curious as to why you are not logging the parse error (you certainly appear to be getting the empty (header only files) I was.

        What version are you running ? (both pfsense and pfblocker NG)

        Thanks

        Bob.DigB 1 Reply Last reply Reply Quote 0
        • fireodoF
          fireodo @jrey
          last edited by fireodo

          @jrey said in pfBlockerNG ASN downloads only contain a header:

          Could someone who is using an ASN please confirm if they are downloading or if you see the parse error in the pfblockerng.log
          Thanks

          Yes here too, the error logged is:

          parse error: Invalid numeric literal at line 2, column 0
          

          pfsense CE 2.7.0 ; pfBlockerNG 3.2.0_5

          Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
          SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
          pfsense 2.7.2 CE
          Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

          1 Reply Last reply Reply Quote 0
          • Bob.DigB
            Bob.Dig LAYER 8 @jrey
            last edited by

            @jrey said in pfBlockerNG ASN downloads only contain a header:

            (both pfsense and pfblocker NG)

            23.05.1-RELEASE and Devel 3.2.0_5
            Sry, I am not an advanced user of pfSense.

            Interesting other problems showed up when I looked in the logs:

             [ pfB_uceprotect_v4 - dnsbl1_v4 ] Download Fail [ 08/4/23 22:55:41 ]
              DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.
             [ 08/4/23 22:55:41 ]
            [PFB_FILTER - 2] Invalid URL (not allowed2) [ rsync-mirrors.uceprotect.net::RBLDNSD-ALL/dnsbl-1.uceprotect.net ] [ 08/5/23 10:10:22 ]
            
             [ pfB_UCEPROTECTNetwork_v4 - dnsblOne_v4 ] Download Fail [ 08/11/23 23:15:38 ]
             [ 41.208.71.58 ] Firewall IP block found in: [ pfB_Africa_v4 | 41.208.0.0/16 ] for HOST:wget-mirrors.uceprotect.net!
             [ 08/11/23 23:15:40 ]
              Restoring previously downloaded file contents... [ 08/11/23 23:15:40 ]
            
            J 1 Reply Last reply Reply Quote 0
            • fireodoF
              fireodo @jrey
              last edited by

              @jrey said in pfBlockerNG ASN downloads only contain a header:

              Now in my case I have completely tracked this down, and ended up having to make a small code change in pfblockerng.sh

              What exactly code change?

              Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
              SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
              pfsense 2.7.2 CE
              Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

              J 1 Reply Last reply Reply Quote 0
              • J
                jrey @fireodo
                last edited by

                @fireodo

                file /usr/local/pkg/pfblockerng/pfblockerng.sh

                first thing I did was log the actual command being executed
                in all samples the actual AS number has shown as MASKED substitute with just the AS number, if you wish to test)

                [ ASMASKED_v4 ] Downloading update [ 08/12/23 15:08:27 ] .
                Downloading ASN: MASKED./usr/local/bin/curl -H pfSense/pfBlockerNG cURL download agent-(this will be your device ID) -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/pfbtemp10_7137.. completed
                parse error: Invalid numeric literal at line 2, column 0
                . completed ..
                Empty file, Adding '127.1.7.7' to avoid download failure.

                at the end of the day the agent (your device id) is not actually relevant to the success of the result
                the temp file is random (in code) after parsing (good or bad) it is deleted

                from a command line you can use
                /usr/local/bin/curl -H pfSense/pfBlockerNG cURL download agent-bobisyouruncle -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/bob
                in code the tmp file is removed after processing so you can rm /tmp/bob

                if you run this from the command line you should notice it errors as follows
                [23.05.1-RELEASE]/usr/local/pkg/pfblockerng: /usr/local/bin/curl -H pfSense/pfBlockerNG cURL download agent-bobisyouruncle -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/bob
                curl: (6) Could not resolve host: cURL
                curl: (6) Could not resolve host: download
                curl: (6) Could not resolve host: agent-bobisyouruncle

                the agent seems is actually required, because without one the api.bgpview.io also appears to fail
                Since the -H isn't even documented on pfsense curl and I had determined that api wanted one I used
                [ ASMASKED_v4 ] Downloading update .
                Downloading ASN: MASKED./usr/local/bin/curl -A agent-(this will be your device ID) -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/pfbtemp10_77358.. completed
                . completed ..

                again from the command line
                [23.05.1-RELEASE][]/usr/local/pkg/pfblockerng: /usr/local/bin/curl -A agent-bobisyouruncle -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/bob
                [23.05.1-RELEASE][]/usr/local/pkg/pfblockerng:

                No Errors

                So my change in pfblockerng.sh became this
                pfblockerng code change.png

                has been working perfectly again since making the change.
                You will notice that in all changes I just commeted out the original lines and added new replacement lines/comments (or in the case of the ua= just overrode the original value. This so I can go back to original - should a patch ever be created, rather than a completely new version of the file.

                if you do this test from the command line you can delete file tmp/bob

                fireodoF 1 Reply Last reply Reply Quote 1
                • fireodoF
                  fireodo @jrey
                  last edited by

                  @jrey Thanks a lot - I'll take a look ...

                  Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                  SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                  pfsense 2.7.2 CE
                  Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                  J 1 Reply Last reply Reply Quote 0
                  • stephenw10S stephenw10 moved this topic from General pfSense Questions on
                  • J
                    jrey @Bob.Dig
                    last edited by

                    @Bob-Dig said in pfBlockerNG ASN downloads only contain a header:

                    [ pfB_uceprotect_v4 - dnsbl1_v4 ] Download Fail [ 08/4/23 22:55:41 ]
                    DNSBL, Firewall, and IDS (Legacy mode only) are not blocking download.
                    [ 08/4/23 22:55:41 ]
                    [PFB_FILTER - 2] Invalid URL (not allowed2) [ rsync-mirrors.uceprotect.net::RBLDNSD-ALL/dnsbl-1.uceprotect.net ] [ 08/5/23 10:10:22 ]
                    [ pfB_UCEPROTECTNetwork_v4 - dnsblOne_v4 ] Download Fail [ 08/11/23 23:15:38 ]
                    [ 41.208.71.58 ] Firewall IP block found in: [ pfB_Africa_v4 | 41.208.0.0/16 ] for HOST:wget-mirrors.uceprotect.net!
                    [ 08/11/23 23:15:40 ]
                    Restoring previously downloaded file contents... [ 08/11/23 23:15:40 ]

                    i can't comment on these as I don't use these feeds, however the invalid URL sounds suspicious depending on what method that download is actually using. if wherever method that is in code is building a similar curl type command, that might be the cause. (or it could be that the feed has just moved at source and the URL is actually invalid.

                    for the ASN parts if you want to check pfblockerng.sh you can easily compare the code I posted in the thread by going

                    (Menu) Diagnostics -> Edit File
                    Enter the file name shown
                    click Load
                    goto line 761

                    (I wouldn't change it here, when you are done viewing just navigate away from the page and nothing changes)

                    Edit.png

                    1 Reply Last reply Reply Quote 0
                    • J
                      jrey @fireodo
                      last edited by

                      @fireodo

                      I'm assuming the upvote (thanks) implies the workaround I provided, worked for you as well (so your CE 2.7.0 and pfBlocker 3.2.0_5) is / was doing the same thing as you noted.

                      2 things remain,

                      I wonder how many people may not know they even have the problem
                      and
                      as @Bob-Dig points out running 3.2.0_5 devel which seems to have the problem but doesn't seem to log a parse error.
                      (was under the impression that 3.2.0_5 and 3.2.0_5 devel are the same)

                      I took a look at the dashboard part as well, and it seem the notice to the dashboard only looks at the errlog for the word "FAIL"
                      point here is that the download with or without the subsequent "parse error" isn't a download failure as such so it doesn't write to the error log.
                      (it could by looking at the random /tmp/file before parsing, if it contains HTML and not a list of IP address, I think it could write to the error log at that point or after the .orig file is created from the tmp/file and only contains a single header with no IP addresses) In either case it is a failure (completely silent or otherwise) and should be highlighted on the dashboard.

                      I was considering making it do that, and or modifying the dashboard so it also trips on the word "error" if it find found in the pfblockerng.log
                      doing either would work for both of us, but not @Bob-Dig as there was seemingly no parse error logged in that case even though "all the files" as reported only had the header..

                      I'm actually holding and hoping @BBcan177 may chime in.

                      fireodoF 2 Replies Last reply Reply Quote 0
                      • fireodoF
                        fireodo @jrey
                        last edited by

                        @jrey

                        I'm actually holding and hoping @BBcan177 may chime in.

                        Thats what I'm hoping too!
                        It seams also to be a API issue at bgpview ...

                        Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                        SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                        pfsense 2.7.2 CE
                        Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                        J 1 Reply Last reply Reply Quote 0
                        • J
                          jrey @fireodo
                          last edited by

                          @fireodo said in pfBlockerNG ASN downloads only contain a header:

                          It seams also to be a API issue at bgpview .

                          I didn't see that -- it worked as expected from every other platform I test pulled from and has been consistently returning results
                          You can even drop the uri into a browser and it will respond --- so from the client side any agent string works, they don't appear to tie the result success to the content of the agent string.

                          I'm thinking more that curl changed with a recent update and as a result the call with all the "-H" stuff started running into issues (outright failing actually) as demonstrated from the command line test.
                          the api also failed at the pgpview end because it no longer saw an "agent" and returned a response to the tmp/file that said as much. That response file ended up being an "orig" file with only a header. The agent string from my testing is required by the api but the agent string can be anything so "bobisyouruncle" as provided in my test strings works and the api didn't complain. I therefore didn't change the code so it still sends the Netgate device ID for the agent with the -A but all the other stuff that was originally in the -H parameter was obviously removed from the changed -A parameter.

                          Thanks for your feedback...
                          JR

                          BBcan177B 1 Reply Last reply Reply Quote 1
                          • BBcan177B
                            BBcan177 Moderator @jrey
                            last edited by BBcan177

                            @jrey

                            Thanks for digging in. Looks like that was the issue.. Sorry about that!

                            I made a patch which can be downloaded here as there are two lines that needed to be changed:

                            curl -o /usr/local/pkg/pfblockerng/pfblockerng.sh "https://gist.githubusercontent.com/BBcan177/1c1fee14759bc574350a3bc85b63a57e/raw"
                            

                            Will get this into a PR soon.

                            UPDATE:
                            I believe that BGPview is rate limiting any Agent string that contains "pfBlockerNG", I changed it to "pfBNG" for now and will update once I get feedback from them,

                            "Experience is something you don't get until just after you need it."

                            Website: http://pfBlockerNG.com
                            Twitter: @BBcan177  #pfBlockerNG
                            Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                            BBcan177B J 2 Replies Last reply Reply Quote 2
                            • BBcan177B
                              BBcan177 Moderator @BBcan177
                              last edited by

                              @jrey @fireodo If you guys can test the patch would be appreciated Thanks!

                              "Experience is something you don't get until just after you need it."

                              Website: http://pfBlockerNG.com
                              Twitter: @BBcan177  #pfBlockerNG
                              Reddit: https://www.reddit.com/r/pfBlockerNG/new/

                              fireodoF J 2 Replies Last reply Reply Quote 1
                              • J
                                jrey @BBcan177
                                last edited by

                                @BBcan177 said in pfBlockerNG ASN downloads only contain a header:

                                I believe that BGPview is rate limiting any Agent string that contains "pfBlockerNG", I changed it to "pfBNG" for now and will update once I get feedback from them,

                                interesting, I'm likely not hitting a rate limit as I use less than 15 ASN feeds (and clearly they didn't prevent me from only hitting with an agent string like bobisyouruncle.

                                the second case you changed (around line 826) would only be used for IP to ASN right ? (so in the case where ASN Reporting is enabled)

                                also I did a mv to capture the current file, then got your new file -- forgot to set the execute permissions LOL that was fun -- whoops
                                see the permissions on file which
                                seems to be working on a force -- I'll let it run on the schedule for a bit.

                                I'd honestly still like to see that actual call / parameters logged (it is always helpful at some point) and perhaps something that notifies the dashboard if the list ends up with just a "header" and no IP addresses as was the case here.

                                Thanks,
                                JR

                                1 Reply Last reply Reply Quote 0
                                • M
                                  mbentley
                                  last edited by mbentley

                                  I ended up creating a couple of patches that I could apply through the Patches package:

                                  https://gist.github.com/mbentley/a3f93643de57a0f325fbee7bf34afbad

                                  One where the user agent string is fixed from -H to -A and another where it isn't yet fixed (where you're running the original files directly from the package.

                                  Nice thing about the patch there is that it could make it easy to have a unique user agent to try to not get whatever string is being set flagged for rate limiting as even the adding in the netgate device ID to the UA doesn't really change anything.

                                  I've also noticed that the service was REALLY slow before they started blocking. I have to imagine their API was getting hammered and them blocking a ton of bots has helped their APIs be quicker - they're lightning fast right now.

                                  *edit: I noticed that I actually missed one place where the user-agent was set when I was creating the patch file. Updated the patch file at https://gist.github.com/mbentley/a3f93643de57a0f325fbee7bf34afbad#file-pfblockerng-sh-patch

                                  1 Reply Last reply Reply Quote 1
                                  • Bob.DigB
                                    Bob.Dig LAYER 8
                                    last edited by Bob.Dig

                                    As a side question: If I disable

                                    Netgate Device ID - Do NOT send Netgate Device ID with user agent
                                    

                                    in (System-Advanced-Miscellaneous), can that have negative side effects in the way we potentially are seeing it here? Is this the real usecase for that Netgate Device ID? Or is it better to not have it? Will the other side still be able to see "pfSense" if i disable the option?

                                    J fireodoF 2 Replies Last reply Reply Quote 0
                                    • fireodoF
                                      fireodo @BBcan177
                                      last edited by fireodo

                                      @BBcan177 said in pfBlockerNG ASN downloads only contain a header:

                                      If you guys can test the patch would be appreciated Thanks!

                                      Hi,

                                      sorry but I have after the patch the same empty ASN Lists. (BTW. I have only 7 Lists) (I dont get the parsing error anymore)
                                      @jrey The thumbs up was as a compliment for your intensive Work, but (here) it is not successful (I change the user agent and also the curl parameter) 😕
                                      There must be something else too ...
                                      Edit: I look in the log and I saw that it begun on the night from 12 to 13.08. (no changes on pfblockerNG for a long time)

                                      my 2 cents,
                                      fireodo

                                      Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                                      SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                                      pfsense 2.7.2 CE
                                      Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        jrey @BBcan177
                                        last edited by

                                        @BBcan177

                                        riddle me this.

                                        appears are still using a technically failing command. I put back the printf to capture the command being run, because seeing what is happening is important.

                                        once a cycle completed I pulled the command from the log

                                        This is what is being executed from the "patch" - and at the command line it fails.

                                        [23.05.1-RELEASE][mask]: /usr/local/bin/curl -A pfSense/pfBNG cURL download agent-MASKED -sS1 https://api.bgpview.io/MASKED/prefixes > /tmp/pfbtemp10_7790.
                                        curl: (6) Could not resolve host: cURL
                                        curl: (6) Could not resolve host: download
                                        curl: (6) Could not resolve host: agent-MASKED

                                        changing it at the command line so that the -A (agent parameter) only includes the agent-part of the string (No Errors at command line)
                                        [23.05.1-RELEASE][mask]: /usr/local/bin/curl -A agent-MASKED -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/pfbtemp10_7790.
                                        [23.05.1-RELEASE][mask]:

                                        this works equally as well (I removed the redirect of the output away from the tmp file and just let it spew to the screen (still works, every time)
                                        [23.05.1-RELEASE][mask]: /usr/local/bin/curl -A agent-bobisyouruncle -sS1 https://api.bgpview.io/asn/MASKED/prefixes
                                        {"status":"ok","status_message":"Query was successful","......

                                        Now from BPGView API page BGPView.png

                                        they "suggest / try" with only the --include option
                                        this by itself fails from a command line
                                        /usr/local/bin/curl --include https://api.bgpview.io/asn/MASK/prefixes
                                        returns a Forbidden

                                        HTTP/2 403
                                        date: Mon, 14 Aug 2023 11:29:10 GMT
                                        content-type: text/html
                                        cf-cache-status: DYNAMIC
                                        ...

                                        However adding an agent string even "agent-bob" as shown here works just fine.
                                        /usr/local/bin/curl -A agent-bob --include https://api.bgpview.io/asn/MASK/prefixes

                                        this also works (took the spaces out of your agent string replace with -

                                        /usr/local/bin/curl -A pfSense/pfBNG-cURL-download-agent-MASKED -sS1 https://api.bgpview.io/asn/MASKED/prefixes

                                        clearly the version of curl doesn't like spaces in the agent string,
                                        So the documentation for curl "suggests" that the -A string be enclose in quotes if it contains spaces so doing that of course works.
                                        shows this
                                        curl -A "user-agent-name-here" [URL]
                                        and an example like this
                                        curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" https://example.com/

                                        therefore
                                        /usr/local/bin/curl -A "pfSense/pfBNG cURL download agent-MASKED" -sS1 https://api.bgpview.io/asn/MASKED/prefixes > /tmp/pfbtemp10_7790.
                                        also works

                                        and as for BGPView it doesn't appear they care what the agent string is, as long as there is one.

                                        @fireodo
                                        so now it appears for you the patch is not working although the parse error is gone?
                                        You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

                                        also try the above commands from a shell

                                        fireodoF M 3 Replies Last reply Reply Quote 1
                                        • fireodoF
                                          fireodo @jrey
                                          last edited by

                                          @jrey said in pfBlockerNG ASN downloads only contain a header:

                                          so now it appears for you the patch is not working although the parse error is gone?

                                          Thats correct!

                                          You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

                                          I'll do that.

                                          also try the above commands from a shell

                                          That too ...

                                          Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                                          SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                                          pfsense 2.7.2 CE
                                          Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                                          1 Reply Last reply Reply Quote 0
                                          • fireodoF
                                            fireodo @jrey
                                            last edited by fireodo

                                            @jrey said in pfBlockerNG ASN downloads only contain a header:

                                            You might want to try removing all the existing AS* files from directories under /var/db/pfblockerng (deny/original etc) and running again

                                            THAT was the solution! With @BBcan177 s Patch and deleting the old files now the ASN Lists are populated as they should - that should be noticed somewhere when pfblockerNG 3.2.0_6 is available!

                                            Below the line I can say the culprit was the user agent and the curl parameter - maybe the user agent logic should be reconsidered ... if necessary.

                                            Thanks @jrey @BBcan177
                                            fireodo

                                            Kettop Mi4300YL CPU: i5-4300Y @ 1.60GHz RAM: 8GB Ethernet Ports: 4
                                            SSD: SanDisk pSSD-S2 16GB (ZFS) WiFi: WLE200NX
                                            pfsense 2.7.2 CE
                                            Packages: Apcupsd Cron Iftop Iperf LCDproc Nmap pfBlockerNG RRD_Summary Shellcmd Snort Speedtest System_Patches.

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.