Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    Phishtank list download fail

    Scheduled Pinned Locked Moved pfBlockerNG
    35 Posts 4 Posters 2.4k Views
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • provelsP
      provels @revengineer
      last edited by

      @revengineer
      Did not work. Will change to Flex and see how it goes.

      [ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded

 [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
  Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ PhishTank_v4 ]

      Peder

      MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
      BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

      provelsP 1 Reply Last reply Reply Quote 0
      • provelsP
        provels @provels
        last edited by

        @provels said in Phishtank list download fail:

        @revengineer
        Did not work. Will change to Flex and see how it goes.

        [ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded

 [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
  Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ PhishTank_v4 ]

        Failed with Flex as well, sorry.

        Peder

        MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
        BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

        R 1 Reply Last reply Reply Quote 0
        • RonpfSR
          RonpfS
          last edited by RonpfS

          @provels said in Phishtank list download fail:

          Maybe post the log with Flex so we can see something.
          What URL are we talking about ?

          2.4.5-RELEASE-p1 (amd64)
          Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
          Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

          R 1 Reply Last reply Reply Quote 0
          • R
            revengineer @provels
            last edited by

            @provels You are actually getting a different error than I did. Mine was a 403 error. The 509 may be the result of not using an API key, which seems to be required for automated downloads.

            1 Reply Last reply Reply Quote 0
            • R
              revengineer @RonpfS
              last edited by

              @RonpfS What log file are you talking about? The pfblockerng.log only shows

              [ PhishTank ]		 Downloading update .. 403 Forbidden

 [ DNSBL_Phishing - PhishTank ] Download FAIL
  Firewall and/or IDS are not blocking download.

  Restoring previously downloaded file

              The error.log only shows

               [ DNSBL_Phishing - PhishTank ] Download Fail
  Firewall and/or IDS are not blocking download.

              Neither is very informative. If you know of more detailed logs, please let me know.

              1 Reply Last reply Reply Quote 0
              • RonpfSR
                RonpfS
                last edited by

                @revengineer said in Phishtank list download fail:

                @RonpfS What log file are you talking about?

                Well something with timestamp help a lot!
                So is it the API URL ? Why don't you post the URL masking any key...

                2.4.5-RELEASE-p1 (amd64)
                Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                R 1 Reply Last reply Reply Quote 0
                • R
                  revengineer @RonpfS
                  last edited by

                  @RonpfS I did not see anything with a time stamp. The URL is

                  http://data.phishtank.com/data/online-valid.csv

                  and if you have an API key, it is

                  http://data.phishtank.com/data/<your app key>/online-valid.csv
                  1 Reply Last reply Reply Quote 0
                  • RonpfSR
                    RonpfS
                    last edited by

                    Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
                    https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2

                    2.4.5-RELEASE-p1 (amd64)
                    Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                    Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                    R 1 Reply Last reply Reply Quote 0
                    • R
                      revengineer @RonpfS
                      last edited by

                      @RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.

                      RonpfSR 1 Reply Last reply Reply Quote 0
                      • RonpfSR
                        RonpfS @revengineer
                        last edited by

                        @revengineer yes try it.

                        Why don't you register to pull with the API key?

                        2.4.5-RELEASE-p1 (amd64)
                        Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                        Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                        R 1 Reply Last reply Reply Quote 0
                        • R
                          revengineer @RonpfS
                          last edited by

                          @RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.

                          1 Reply Last reply Reply Quote 0
                          • RonpfSR
                            RonpfS
                            last edited by RonpfS

                            @revengineer said in Phishtank list download fail:

                            http://data.phishtank.com/data/online-valid.csv

                            And the browser goes thru the same pfsense ?
                            Maybe you are on a block list, wait another 2 days before testing.
                            Try curl in a shell on the pfsense device to see more log.

                            2.4.5-RELEASE-p1 (amd64)
                            Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                            Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                            R 2 Replies Last reply Reply Quote 0
                            • R
                              revengineer @RonpfS
                              last edited by

                              @RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.

                              Let me ask you this: Is the PhishTank feed actually working for you?

                              1 Reply Last reply Reply Quote 0
                              • RonpfSR
                                RonpfS
                                last edited by

                                @revengineer said in Phishtank list download fail:

                                Let me ask you this: Is the PhishTank feed actually working for you?

                                Yes.

                                2.4.5-RELEASE-p1 (amd64)
                                Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                R 1 Reply Last reply Reply Quote 0
                                • R
                                  revengineer @RonpfS
                                  last edited by

                                  @RonpfS And yes, the browser is behind the same pfSense that I am trying to install the feed on.

                                  1 Reply Last reply Reply Quote 0
                                  • R
                                    revengineer @RonpfS
                                    last edited by

                                    @RonpfS I am stumped. Not sure what else to try.

                                    1 Reply Last reply Reply Quote 0
                                    • RonpfSR
                                      RonpfS
                                      last edited by

                                      Open a shell and try curl ...
                                      Anything in the /tmp folder ?

                                      2.4.5-RELEASE-p1 (amd64)
                                      Intel Core2 Quad CPU Q8400 @ 2.66GHz 8GB
                                      Backup 0.5_5, Bandwidthd 0.7.4_4, Cron 0.3.7_5, pfBlockerNG-devel 3.0.0_16, Status_Traffic_Totals 2.3.1_1, System_Patches 1.2_5

                                      R 1 Reply Last reply Reply Quote 0
                                      • R
                                        revengineer @RonpfS
                                        last edited by

                                        @RonpfS So tried curl and it returns to the commandline without downloading a file. The verbose output is

                                        *   Trying 104.17.177.85:80...
* TCP_NODELAY set
* Connected to data.phishtank.com (104.17.177.85) port 80 (#0)
> GET /data/online-valid.csv HTTP/1.1
> Host: data.phishtank.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 21 Nov 2020 01:31:10 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sat, 21 Nov 2020 02:31:10 GMT
< Location: https://data.phishtank.com/data/online-valid.csv
< cf-request-id: 068a0644d00000cee439007000000001
< Server: cloudflare
< CF-RAY: 5f56a64e1f94cee4-IAD
<
* Connection #0 to host data.phishtank.com left intact
                                        provelsP 1 Reply Last reply Reply Quote 0
                                        • provelsP
                                          provels @revengineer
                                          last edited by

                                          @revengineer
                                          Tried HTTPS? I was able to add the https://data.phishtank.com/data/online-valid.csv.bz2
                                          from the pfB Feeds page and for the heck of it registered for an API key and added it to the link. Ran w/o error on Force/Reload and Cron. That's the same list, just a different format, right?
                                          bf140d68-b617-417a-85ae-82d01d6f3927-image.png

                                          Peder

                                          MAIN - pfSense+ 24.11-RELEASE - Adlink MXE-5401, i7, 16 GB RAM, 64 GB SSD. 500 GB HDD for SyslogNG
                                          BACKUP - pfSense+ 23.01-RELEASE - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 3 GB RAM, 8GB VHDX (Dynamic)

                                          R 1 Reply Last reply Reply Quote 0
                                          • R
                                            revengineer @provels
                                            last edited by

                                            @provels Good point. I did not write but did indeed try many combinations. For the https case with .bz extension, I get

                                            curl -v https://data.phishtank.com/data/<api_key>/online-valid.csv.bz2
*   Trying 104.16.101.75:443...
* TCP_NODELAY set
* Connected to data.phishtank.com (104.16.101.75) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug 17 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2021 GMT
*  subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x803abb800)
> GET /data/<api_key>/online-valid.csv.bz2 HTTP/2
> Host: data.phishtank.com
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 302
< date: Sat, 21 Nov 2020 13:26:33 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: __cfduid=de8c92bb34f744f36bf09f5cfa9c6a7c21605965193; expires=Mon, 21-Dec-20 13:26:33 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure
< x-request-limit-interval: 10800 Seconds
< x-request-limit: 12
< x-request-count: 1
< location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605965203&Signature=efXCsFqG1q8UlLtJihn7Nj6fXJRyjTjXVq96b2gvsnAhyOiM9Kfv4mpuCfY...[ABBREVIATED]
< cf-cache-status: DYNAMIC
< cf-request-id: 068c953aa400002acc66ad6000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< server: cloudflare
< cf-ray: 5f5abe3ddf3e2acc-IAD
<
* Connection #0 to host data.phishtank.com left intact
                                            1 Reply Last reply Reply Quote 0
                                            • First post
                                              Last post
                                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.