Phishtank list download fail

revengineer

@RonpfS What log file are you talking about? The pfblockerng.log only shows

[ PhishTank ]		 Downloading update .. 403 Forbidden

 [ DNSBL_Phishing - PhishTank ] Download FAIL
  Firewall and/or IDS are not blocking download.

  Restoring previously downloaded file

The error.log only shows

 [ DNSBL_Phishing - PhishTank ] Download Fail
  Firewall and/or IDS are not blocking download.

Neither is very informative. If you know of more detailed logs, please let me know.

RonpfS

@revengineer said in Phishtank list download fail:

@RonpfS What log file are you talking about?

Well something with timestamp help a lot!
So is it the API URL ? Why don't you post the URL masking any key...

revengineer

@RonpfS I did not see anything with a time stamp. The URL is

http://data.phishtank.com/data/online-valid.csv

and if you have an API key, it is

http://data.phishtank.com/data/<your app key>/online-valid.csv

RonpfS

Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2

revengineer

@RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.

RonpfS

@revengineer yes try it.

Why don't you register to pull with the API key?

revengineer

@RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.

RonpfS

@revengineer said in Phishtank list download fail:

http://data.phishtank.com/data/online-valid.csv

And the browser goes thru the same pfsense ?
Maybe you are on a block list, wait another 2 days before testing.
Try curl in a shell on the pfsense device to see more log.

revengineer

@RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.

Let me ask you this: Is the PhishTank feed actually working for you?

RonpfS

@revengineer said in Phishtank list download fail:

Let me ask you this: Is the PhishTank feed actually working for you?

Yes.

revengineer

@RonpfS And yes, the browser is behind the same pfSense that I am trying to install the feed on.

revengineer

@RonpfS I am stumped. Not sure what else to try.

RonpfS

Open a shell and try curl ...
Anything in the /tmp folder ?

revengineer

@RonpfS So tried curl and it returns to the commandline without downloading a file. The verbose output is

*   Trying 104.17.177.85:80...
* TCP_NODELAY set
* Connected to data.phishtank.com (104.17.177.85) port 80 (#0)
> GET /data/online-valid.csv HTTP/1.1
> Host: data.phishtank.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 21 Nov 2020 01:31:10 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sat, 21 Nov 2020 02:31:10 GMT
< Location: https://data.phishtank.com/data/online-valid.csv
< cf-request-id: 068a0644d00000cee439007000000001
< Server: cloudflare
< CF-RAY: 5f56a64e1f94cee4-IAD
<
* Connection #0 to host data.phishtank.com left intact

provels

@revengineer
Tried HTTPS? I was able to add the https://data.phishtank.com/data/online-valid.csv.bz2
from the pfB Feeds page and for the heck of it registered for an API key and added it to the link. Ran w/o error on Force/Reload and Cron. That's the same list, just a different format, right?

revengineer

@provels Good point. I did not write but did indeed try many combinations. For the https case with .bz extension, I get

curl -v https://data.phishtank.com/data/<api_key>/online-valid.csv.bz2
*   Trying 104.16.101.75:443...
* TCP_NODELAY set
* Connected to data.phishtank.com (104.16.101.75) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug 17 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2021 GMT
*  subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x803abb800)
> GET /data/<api_key>/online-valid.csv.bz2 HTTP/2
> Host: data.phishtank.com
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 302
< date: Sat, 21 Nov 2020 13:26:33 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: __cfduid=de8c92bb34f744f36bf09f5cfa9c6a7c21605965193; expires=Mon, 21-Dec-20 13:26:33 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure
< x-request-limit-interval: 10800 Seconds
< x-request-limit: 12
< x-request-count: 1
< location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605965203&Signature=efXCsFqG1q8UlLtJihn7Nj6fXJRyjTjXVq96b2gvsnAhyOiM9Kfv4mpuCfY...[ABBREVIATED]
< cf-cache-status: DYNAMIC
< cf-request-id: 068c953aa400002acc66ad6000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< server: cloudflare
< cf-ray: 5f5abe3ddf3e2acc-IAD
<
* Connection #0 to host data.phishtank.com left intact

revengineer

I have tried downloading the feed from an Ubuntu server as well. The curl command fails in the same manner, wget command works.

@provels I think it may be time for me to contact the PhishTank folks. for comparison, can you post your "curl -v" output?

provels

@revengineer said in Phishtank list download fail:

curl -v

curl -v https://data.phishtank.com/data/<api-key>/online-valid.csv.bz2
*   Trying 2606:4700::6810:654b:443...
* TCP_NODELAY set
* Connected to data.phishtank.com (2606:4700::6810:654b) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /usr/local/share/certs/ca-root-nss.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
*  start date: Aug 17 00:00:00 2020 GMT
*  expire date: Aug 17 12:00:00 2021 GMT
*  subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x803aba800)
> GET /data/<api-key>/online-valid.csv.bz2 HTTP/2
> Host: data.phishtank.com
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 302
< date: Sat, 21 Nov 2020 15:14:17 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: __cfduid=db5c57b87a118488312b7047180d6f9101605971657; expires=Mon, 21-Dec-20 15:14:17 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure
< x-request-limit-interval: 10800 Seconds
< x-request-limit: 12
< x-request-count: 1
< location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605971667&Signature=iUZaI4nsb9LNji0tMhiEsrZW9fryn751OzXVP ... ETC, ETC.
< cf-cache-status: DYNAMIC
< cf-request-id: 068cf7dc0900007f68c79e9000000001
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< strict-transport-security: max-age=15552000
< server: cloudflare
< cf-ray: 5f5b5c0cd90f7f68-ORD
<
* Connection #0 to host data.phishtank.com left intact

revengineer

@provels This looks similar to my output. And after this you have a file called online-valid.csv.bz2 in the directory you called the command from?

provels

@revengineer said in Phishtank list download fail:

online-valid.csv.bz2

Well, um, no. Ran from /root, not there. Searched the tree, too. But like I said, worked OK using HTTPS from Force and Cron within pfB. Does curl actually do a GET? I'm no command line expert.