Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Phishtank list download fail

    Scheduled Pinned Locked Moved pfBlockerNG
    35 Posts 4 Posters 2.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • R
      revengineer @provels
      last edited by

      @provels You are actually getting a different error than I did. Mine was a 403 error. The 509 may be the result of not using an API key, which seems to be required for automated downloads.

      1 Reply Last reply Reply Quote 0
      • R
        revengineer @RonpfS
        last edited by

        @RonpfS What log file are you talking about? The pfblockerng.log only shows

        [ PhishTank ]		 Downloading update .. 403 Forbidden
        
         [ DNSBL_Phishing - PhishTank ] Download FAIL
          Firewall and/or IDS are not blocking download.
        
          Restoring previously downloaded file
        

        The error.log only shows

         [ DNSBL_Phishing - PhishTank ] Download Fail
          Firewall and/or IDS are not blocking download.
        

        Neither is very informative. If you know of more detailed logs, please let me know.

        1 Reply Last reply Reply Quote 0
        • RonpfSR
          RonpfS
          last edited by

          @revengineer said in Phishtank list download fail:

          @RonpfS What log file are you talking about?

          Well something with timestamp help a lot!
          So is it the API URL ? Why don't you post the URL masking any key...

          2.4.5-RELEASE-p1 (amd64)
          Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
          Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

          R 1 Reply Last reply Reply Quote 0
          • R
            revengineer @RonpfS
            last edited by

            @RonpfS I did not see anything with a time stamp. The URL is

            http://data.phishtank.com/data/online-valid.csv
            

            and if you have an API key, it is

            http://data.phishtank.com/data/<your app key>/online-valid.csv
            
            1 Reply Last reply Reply Quote 0
            • RonpfSR
              RonpfS
              last edited by

              Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
              https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2

              2.4.5-RELEASE-p1 (amd64)
              Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
              Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

              R 1 Reply Last reply Reply Quote 0
              • R
                revengineer @RonpfS
                last edited by

                @RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.

                RonpfSR 1 Reply Last reply Reply Quote 0
                • RonpfSR
                  RonpfS @revengineer
                  last edited by

                  @revengineer yes try it.

                  Why don't you register to pull with the API key?

                  2.4.5-RELEASE-p1 (amd64)
                  Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                  Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                  R 1 Reply Last reply Reply Quote 0
                  • R
                    revengineer @RonpfS
                    last edited by

                    @RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.

                    1 Reply Last reply Reply Quote 0
                    • RonpfSR
                      RonpfS
                      last edited by RonpfS

                      @revengineer said in Phishtank list download fail:

                      http://data.phishtank.com/data/online-valid.csv

                      And the browser goes thru the same pfsense ?
                      Maybe you are on a block list, wait another 2 days before testing.
                      Try curl in a shell on the pfsense device to see more log.

                      2.4.5-RELEASE-p1 (amd64)
                      Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                      Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                      R 2 Replies Last reply Reply Quote 0
                      • R
                        revengineer @RonpfS
                        last edited by

                        @RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.

                        Let me ask you this: Is the PhishTank feed actually working for you?

                        1 Reply Last reply Reply Quote 0
                        • RonpfSR
                          RonpfS
                          last edited by

                          @revengineer said in Phishtank list download fail:

                          Let me ask you this: Is the PhishTank feed actually working for you?

                          Yes.

                          2.4.5-RELEASE-p1 (amd64)
                          Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                          Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                          R 1 Reply Last reply Reply Quote 0
                          • R
                            revengineer @RonpfS
                            last edited by

                            @RonpfS And yes, the browser is behind the same pfSense that I am trying to install the feed on.

                            1 Reply Last reply Reply Quote 0
                            • R
                              revengineer @RonpfS
                              last edited by

                              @RonpfS I am stumped. Not sure what else to try.

                              1 Reply Last reply Reply Quote 0
                              • RonpfSR
                                RonpfS
                                last edited by

                                Open a shell and try curl ...
                                Anything in the /tmp folder ?

                                2.4.5-RELEASE-p1 (amd64)
                                Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                R 1 Reply Last reply Reply Quote 0
                                • R
                                  revengineer @RonpfS
                                  last edited by

                                  @RonpfS So tried curl and it returns to the commandline without downloading a file. The verbose output is

                                  *   Trying 104.17.177.85:80...
                                  * TCP_NODELAY set
                                  * Connected to data.phishtank.com (104.17.177.85) port 80 (#0)
                                  > GET /data/online-valid.csv HTTP/1.1
                                  > Host: data.phishtank.com
                                  > User-Agent: curl/7.68.0
                                  > Accept: */*
                                  >
                                  * Mark bundle as not supporting multiuse
                                  < HTTP/1.1 301 Moved Permanently
                                  < Date: Sat, 21 Nov 2020 01:31:10 GMT
                                  < Transfer-Encoding: chunked
                                  < Connection: keep-alive
                                  < Cache-Control: max-age=3600
                                  < Expires: Sat, 21 Nov 2020 02:31:10 GMT
                                  < Location: https://data.phishtank.com/data/online-valid.csv
                                  < cf-request-id: 068a0644d00000cee439007000000001
                                  < Server: cloudflare
                                  < CF-RAY: 5f56a64e1f94cee4-IAD
                                  <
                                  * Connection #0 to host data.phishtank.com left intact
                                  
                                  
                                  provelsP 1 Reply Last reply Reply Quote 0
                                  • provelsP
                                    provels @revengineer
                                    last edited by

                                    @revengineer
                                    Tried HTTPS? I was able to add the https://data.phishtank.com/data/online-valid.csv.bz2
                                    from the pfB Feeds page and for the heck of it registered for an API key and added it to the link. Ran w/o error on Force/Reload and Cron. That's the same list, just a different format, right?
                                    bf140d68-b617-417a-85ae-82d01d6f3927-image.png

                                    Peder

                                    MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                                    BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                                    R 1 Reply Last reply Reply Quote 0
                                    • R
                                      revengineer @provels
                                      last edited by

                                      @provels Good point. I did not write but did indeed try many combinations. For the https case with .bz extension, I get

                                      curl -v https://data.phishtank.com/data/<api_key>/online-valid.csv.bz2
                                      *   Trying 104.16.101.75:443...
                                      * TCP_NODELAY set
                                      * Connected to data.phishtank.com (104.16.101.75) port 443 (#0)
                                      * ALPN, offering h2
                                      * ALPN, offering http/1.1
                                      * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
                                      * successfully set certificate verify locations:
                                      *   CAfile: /usr/local/share/certs/ca-root-nss.crt
                                        CApath: none
                                      * TLSv1.2 (OUT), TLS header, Certificate Status (22):
                                      * TLSv1.2 (OUT), TLS handshake, Client hello (1):
                                      * TLSv1.2 (IN), TLS handshake, Server hello (2):
                                      * TLSv1.2 (IN), TLS handshake, Certificate (11):
                                      * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
                                      * TLSv1.2 (IN), TLS handshake, Server finished (14):
                                      * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
                                      * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
                                      * TLSv1.2 (OUT), TLS handshake, Finished (20):
                                      * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
                                      * TLSv1.2 (IN), TLS handshake, Finished (20):
                                      * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
                                      * ALPN, server accepted to use h2
                                      * Server certificate:
                                      *  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
                                      *  start date: Aug 17 00:00:00 2020 GMT
                                      *  expire date: Aug 17 12:00:00 2021 GMT
                                      *  subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com"
                                      *  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
                                      *  SSL certificate verify ok.
                                      * Using HTTP2, server supports multi-use
                                      * Connection state changed (HTTP/2 confirmed)
                                      * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
                                      * Using Stream ID: 1 (easy handle 0x803abb800)
                                      > GET /data/<api_key>/online-valid.csv.bz2 HTTP/2
                                      > Host: data.phishtank.com
                                      > user-agent: curl/7.68.0
                                      > accept: */*
                                      >
                                      * Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
                                      < HTTP/2 302
                                      < date: Sat, 21 Nov 2020 13:26:33 GMT
                                      < content-type: text/html; charset=UTF-8
                                      < set-cookie: __cfduid=de8c92bb34f744f36bf09f5cfa9c6a7c21605965193; expires=Mon, 21-Dec-20 13:26:33 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure
                                      < x-request-limit-interval: 10800 Seconds
                                      < x-request-limit: 12
                                      < x-request-count: 1
                                      < location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605965203&Signature=efXCsFqG1q8UlLtJihn7Nj6fXJRyjTjXVq96b2gvsnAhyOiM9Kfv4mpuCfY...[ABBREVIATED]
                                      < cf-cache-status: DYNAMIC
                                      < cf-request-id: 068c953aa400002acc66ad6000000001
                                      < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
                                      < strict-transport-security: max-age=15552000
                                      < server: cloudflare
                                      < cf-ray: 5f5abe3ddf3e2acc-IAD
                                      <
                                      * Connection #0 to host data.phishtank.com left intact
                                      
                                      1 Reply Last reply Reply Quote 0
                                      • R
                                        revengineer
                                        last edited by

                                        I have tried downloading the feed from an Ubuntu server as well. The curl command fails in the same manner, wget command works.

                                        @provels I think it may be time for me to contact the PhishTank folks. for comparison, can you post your "curl -v" output?

                                        provelsP 1 Reply Last reply Reply Quote 0
                                        • provelsP
                                          provels @revengineer
                                          last edited by provels

                                          @revengineer said in Phishtank list download fail:

                                          curl -v

                                          curl -v https://data.phishtank.com/data/<api-key>/online-valid.csv.bz2
                                          *   Trying 2606:4700::6810:654b:443...
                                          * TCP_NODELAY set
                                          * Connected to data.phishtank.com (2606:4700::6810:654b) port 443 (#0)
                                          * ALPN, offering h2
                                          * ALPN, offering http/1.1
                                          * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
                                          * successfully set certificate verify locations:
                                          *   CAfile: /usr/local/share/certs/ca-root-nss.crt
                                            CApath: none
                                          * TLSv1.2 (OUT), TLS header, Certificate Status (22):
                                          * TLSv1.2 (OUT), TLS handshake, Client hello (1):
                                          * TLSv1.2 (IN), TLS handshake, Server hello (2):
                                          * TLSv1.2 (IN), TLS handshake, Certificate (11):
                                          * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
                                          * TLSv1.2 (IN), TLS handshake, Server finished (14):
                                          * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
                                          * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
                                          * TLSv1.2 (OUT), TLS handshake, Finished (20):
                                          * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
                                          * TLSv1.2 (IN), TLS handshake, Finished (20):
                                          * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
                                          * ALPN, server accepted to use h2
                                          * Server certificate:
                                          *  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
                                          *  start date: Aug 17 00:00:00 2020 GMT
                                          *  expire date: Aug 17 12:00:00 2021 GMT
                                          *  subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com"
                                          *  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
                                          *  SSL certificate verify ok.
                                          * Using HTTP2, server supports multi-use
                                          * Connection state changed (HTTP/2 confirmed)
                                          * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
                                          * Using Stream ID: 1 (easy handle 0x803aba800)
                                          > GET /data/<api-key>/online-valid.csv.bz2 HTTP/2
                                          > Host: data.phishtank.com
                                          > user-agent: curl/7.68.0
                                          > accept: */*
                                          >
                                          * Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
                                          < HTTP/2 302
                                          < date: Sat, 21 Nov 2020 15:14:17 GMT
                                          < content-type: text/html; charset=UTF-8
                                          < set-cookie: __cfduid=db5c57b87a118488312b7047180d6f9101605971657; expires=Mon, 21-Dec-20 15:14:17 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure
                                          < x-request-limit-interval: 10800 Seconds
                                          < x-request-limit: 12
                                          < x-request-count: 1
                                          < location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605971667&Signature=iUZaI4nsb9LNji0tMhiEsrZW9fryn751OzXVP ... ETC, ETC.
                                          < cf-cache-status: DYNAMIC
                                          < cf-request-id: 068cf7dc0900007f68c79e9000000001
                                          < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
                                          < strict-transport-security: max-age=15552000
                                          < server: cloudflare
                                          < cf-ray: 5f5b5c0cd90f7f68-ORD
                                          <
                                          * Connection #0 to host data.phishtank.com left intact
                                          
                                          

                                          Peder

                                          MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                                          BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                                          R 1 Reply Last reply Reply Quote 0
                                          • R
                                            revengineer @provels
                                            last edited by

                                            @provels This looks similar to my output. And after this you have a file called online-valid.csv.bz2 in the directory you called the command from?

                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.