Phishtank list download fail

revengineer

There is a not on the PhishTank site that automatic downloads require an API key and I got one of those. I can download with a browser without problems, but pfblockerng fails. So it does not seem to be a quota issue. I have this set to download once a day but because it fails it tries updating every time the cron job runs, i.e., every hour.

I did more googling and it was mentioned somewhere that they are now using cloudflare for distribution and the 403 error could be a result of a captcha that is not visible. I cannot verify this and the captcha does not show up when using the browser.

@provels let me know if you get this to work. Otherwise, I am inclined to delete this from my list and go with other lists, e.g., shown here.

RonpfS

@revengineer Try using State Flex
You may have to disable the list for 48hrs then re-enable it to circumvent the blocking.

provels

@revengineer
Did not work. Will change to Flex and see how it goes.

[ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded

 [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
  Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ PhishTank_v4 ]

provels

@provels said in Phishtank list download fail:

@revengineer
Did not work. Will change to Flex and see how it goes.

[ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded

 [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
  Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ PhishTank_v4 ]

Failed with Flex as well, sorry.

RonpfS

@provels said in Phishtank list download fail:

Maybe post the log with Flex so we can see something.
What URL are we talking about ?

revengineer

@provels You are actually getting a different error than I did. Mine was a 403 error. The 509 may be the result of not using an API key, which seems to be required for automated downloads.

revengineer

@RonpfS What log file are you talking about? The pfblockerng.log only shows

[ PhishTank ]		 Downloading update .. 403 Forbidden

 [ DNSBL_Phishing - PhishTank ] Download FAIL
  Firewall and/or IDS are not blocking download.

  Restoring previously downloaded file

The error.log only shows

 [ DNSBL_Phishing - PhishTank ] Download Fail
  Firewall and/or IDS are not blocking download.

Neither is very informative. If you know of more detailed logs, please let me know.

RonpfS

@revengineer said in Phishtank list download fail:

@RonpfS What log file are you talking about?

Well something with timestamp help a lot!
So is it the API URL ? Why don't you post the URL masking any key...

revengineer

@RonpfS I did not see anything with a time stamp. The URL is

http://data.phishtank.com/data/online-valid.csv

and if you have an API key, it is

http://data.phishtank.com/data/<your app key>/online-valid.csv

RonpfS

Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2

revengineer

@RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.

RonpfS

@revengineer yes try it.

Why don't you register to pull with the API key?

revengineer

@RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.

RonpfS

@revengineer said in Phishtank list download fail:

http://data.phishtank.com/data/online-valid.csv

And the browser goes thru the same pfsense ?
Maybe you are on a block list, wait another 2 days before testing.
Try curl in a shell on the pfsense device to see more log.

revengineer

@RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.

Let me ask you this: Is the PhishTank feed actually working for you?

RonpfS

@revengineer said in Phishtank list download fail:

Let me ask you this: Is the PhishTank feed actually working for you?

Yes.

revengineer

@RonpfS And yes, the browser is behind the same pfSense that I am trying to install the feed on.

revengineer

@RonpfS I am stumped. Not sure what else to try.

RonpfS

Open a shell and try curl ...
Anything in the /tmp folder ?

revengineer

@RonpfS So tried curl and it returns to the commandline without downloading a file. The verbose output is

*   Trying 104.17.177.85:80...
* TCP_NODELAY set
* Connected to data.phishtank.com (104.17.177.85) port 80 (#0)
> GET /data/online-valid.csv HTTP/1.1
> Host: data.phishtank.com
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 21 Nov 2020 01:31:10 GMT
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: max-age=3600
< Expires: Sat, 21 Nov 2020 02:31:10 GMT
< Location: https://data.phishtank.com/data/online-valid.csv
< cf-request-id: 068a0644d00000cee439007000000001
< Server: cloudflare
< CF-RAY: 5f56a64e1f94cee4-IAD
<
* Connection #0 to host data.phishtank.com left intact