Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Phishtank list download fail

    Scheduled Pinned Locked Moved pfBlockerNG
    35 Posts 4 Posters 2.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      revengineer
      last edited by

      I am using the Phishtank block list but have been receiving 403 download errors on an hourly basis even though the list is set to update daily. The error only occurs when the cron job executes or I manually select the "Cron" force update option. If I force "Reload All" then the list will download properly.

      There are two issues here:

      1. Why is the download failing? I have registered and am using the link with api key, which should not be limited.

      2. Why is the download occurring hourly if the update frequency is set to daily?

      Help much appreciated.

      Thank you in advance.

      1 Reply Last reply Reply Quote 0
      • R
        revengineer
        last edited by

        Pinging again on this topic. I am really puzzled because I can download the PhishTank blocklist without issues through any web browser but I get a 403 error using the pfBlockerNG cron job. Interestingly, PhishTank.ip and PhishTank.txt files do get created in the /var/db/pfblockerng/dnsbl directory and seem to have valid content. But I also see a file PhishTank.fail. Help greatly appreciated.

        provelsP 1 Reply Last reply Reply Quote 0
        • GertjanG
          Gertjan
          last edited by Gertjan

          Probably the quota system, or the way their access works, can detect the difference between a user-browser download, and a 'curl' (whatever method the pfBlocker script use) type download.

          Time to contact their support.

          No "help me" PM's please. Use the forum, the community will thank you.
          Edit : and where are the logs ??

          1 Reply Last reply Reply Quote 0
          • provelsP
            provels @revengineer
            last edited by provels

            @revengineer
            I just added it to mine. I only update pfB once a day, so I'll see what happens tonight. Maybe you're blacklisting your own IP by checking too often?

            But checking your link with a browser first time ever gives me this. Maybe their .csv is borked in some way.

            You have exceeded the request rate limit for this method. Please see the response headers for usage stats. For more information about rate limiting on Phishtank, please see our developer site: http://www.phishtank.com/developer_info.php
            

            Peder

            MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
            BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

            1 Reply Last reply Reply Quote 0
            • R
              revengineer
              last edited by

              There is a not on the PhishTank site that automatic downloads require an API key and I got one of those. I can download with a browser without problems, but pfblockerng fails. So it does not seem to be a quota issue. I have this set to download once a day but because it fails it tries updating every time the cron job runs, i.e., every hour.

              I did more googling and it was mentioned somewhere that they are now using cloudflare for distribution and the 403 error could be a result of a captcha that is not visible. I cannot verify this and the captcha does not show up when using the browser.

              @provels let me know if you get this to work. Otherwise, I am inclined to delete this from my list and go with other lists, e.g., shown here.

              RonpfSR provelsP 2 Replies Last reply Reply Quote 0
              • RonpfSR
                RonpfS @revengineer
                last edited by RonpfS

                @revengineer Try using State Flex
                You may have to disable the list for 48hrs then re-enable it to circumvent the blocking.

                2.4.5-RELEASE-p1 (amd64)
                Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                1 Reply Last reply Reply Quote 0
                • provelsP
                  provels @revengineer
                  last edited by

                  @revengineer
                  Did not work. Will change to Flex and see how it goes.

                  [ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded
                  
                   [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
                    Firewall and/or IDS (Legacy mode only) are not blocking download.
                  
                  The Following List has been REMOVED [ PhishTank_v4 ]
                  

                  Peder

                  MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                  BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                  provelsP 1 Reply Last reply Reply Quote 0
                  • provelsP
                    provels @provels
                    last edited by

                    @provels said in Phishtank list download fail:

                    @revengineer
                    Did not work. Will change to Flex and see how it goes.

                    [ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded
                    
                     [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
                      Firewall and/or IDS (Legacy mode only) are not blocking download.
                    
                    The Following List has been REMOVED [ PhishTank_v4 ]
                    

                    Failed with Flex as well, sorry.

                    Peder

                    MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                    BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                    R 1 Reply Last reply Reply Quote 0
                    • RonpfSR
                      RonpfS
                      last edited by RonpfS

                      @provels said in Phishtank list download fail:

                      Maybe post the log with Flex so we can see something.
                      What URL are we talking about ?

                      2.4.5-RELEASE-p1 (amd64)
                      Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                      Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                      R 1 Reply Last reply Reply Quote 0
                      • R
                        revengineer @provels
                        last edited by

                        @provels You are actually getting a different error than I did. Mine was a 403 error. The 509 may be the result of not using an API key, which seems to be required for automated downloads.

                        1 Reply Last reply Reply Quote 0
                        • R
                          revengineer @RonpfS
                          last edited by

                          @RonpfS What log file are you talking about? The pfblockerng.log only shows

                          [ PhishTank ]		 Downloading update .. 403 Forbidden
                          
                           [ DNSBL_Phishing - PhishTank ] Download FAIL
                            Firewall and/or IDS are not blocking download.
                          
                            Restoring previously downloaded file
                          

                          The error.log only shows

                           [ DNSBL_Phishing - PhishTank ] Download Fail
                            Firewall and/or IDS are not blocking download.
                          

                          Neither is very informative. If you know of more detailed logs, please let me know.

                          1 Reply Last reply Reply Quote 0
                          • RonpfSR
                            RonpfS
                            last edited by

                            @revengineer said in Phishtank list download fail:

                            @RonpfS What log file are you talking about?

                            Well something with timestamp help a lot!
                            So is it the API URL ? Why don't you post the URL masking any key...

                            2.4.5-RELEASE-p1 (amd64)
                            Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                            Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                            R 1 Reply Last reply Reply Quote 0
                            • R
                              revengineer @RonpfS
                              last edited by

                              @RonpfS I did not see anything with a time stamp. The URL is

                              http://data.phishtank.com/data/online-valid.csv
                              

                              and if you have an API key, it is

                              http://data.phishtank.com/data/<your app key>/online-valid.csv
                              
                              1 Reply Last reply Reply Quote 0
                              • RonpfSR
                                RonpfS
                                last edited by

                                Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
                                https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2

                                2.4.5-RELEASE-p1 (amd64)
                                Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                R 1 Reply Last reply Reply Quote 0
                                • R
                                  revengineer @RonpfS
                                  last edited by

                                  @RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.

                                  RonpfSR 1 Reply Last reply Reply Quote 0
                                  • RonpfSR
                                    RonpfS @revengineer
                                    last edited by

                                    @revengineer yes try it.

                                    Why don't you register to pull with the API key?

                                    2.4.5-RELEASE-p1 (amd64)
                                    Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                    Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                    R 1 Reply Last reply Reply Quote 0
                                    • R
                                      revengineer @RonpfS
                                      last edited by

                                      @RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.

                                      1 Reply Last reply Reply Quote 0
                                      • RonpfSR
                                        RonpfS
                                        last edited by RonpfS

                                        @revengineer said in Phishtank list download fail:

                                        http://data.phishtank.com/data/online-valid.csv

                                        And the browser goes thru the same pfsense ?
                                        Maybe you are on a block list, wait another 2 days before testing.
                                        Try curl in a shell on the pfsense device to see more log.

                                        2.4.5-RELEASE-p1 (amd64)
                                        Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                        Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                        R 2 Replies Last reply Reply Quote 0
                                        • R
                                          revengineer @RonpfS
                                          last edited by

                                          @RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.

                                          Let me ask you this: Is the PhishTank feed actually working for you?

                                          1 Reply Last reply Reply Quote 0
                                          • RonpfSR
                                            RonpfS
                                            last edited by

                                            @revengineer said in Phishtank list download fail:

                                            Let me ask you this: Is the PhishTank feed actually working for you?

                                            Yes.

                                            2.4.5-RELEASE-p1 (amd64)
                                            Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                            Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                            R 1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.