Phishtank list download fail

revengineer

I am using the Phishtank block list but have been receiving 403 download errors on an hourly basis even though the list is set to update daily. The error only occurs when the cron job executes or I manually select the "Cron" force update option. If I force "Reload All" then the list will download properly.

There are two issues here:

Why is the download failing? I have registered and am using the link with api key, which should not be limited.
Why is the download occurring hourly if the update frequency is set to daily?

Help much appreciated.

Thank you in advance.

revengineer

Pinging again on this topic. I am really puzzled because I can download the PhishTank blocklist without issues through any web browser but I get a 403 error using the pfBlockerNG cron job. Interestingly, PhishTank.ip and PhishTank.txt files do get created in the /var/db/pfblockerng/dnsbl directory and seem to have valid content. But I also see a file PhishTank.fail. Help greatly appreciated.

Gertjan

Probably the quota system, or the way their access works, can detect the difference between a user-browser download, and a 'curl' (whatever method the pfBlocker script use) type download.

Time to contact their support.

provels

@revengineer
I just added it to mine. I only update pfB once a day, so I'll see what happens tonight. Maybe you're blacklisting your own IP by checking too often?

But checking your link with a browser first time ever gives me this. Maybe their .csv is borked in some way.

You have exceeded the request rate limit for this method. Please see the response headers for usage stats. For more information about rate limiting on Phishtank, please see our developer site: http://www.phishtank.com/developer_info.php

revengineer

There is a not on the PhishTank site that automatic downloads require an API key and I got one of those. I can download with a browser without problems, but pfblockerng fails. So it does not seem to be a quota issue. I have this set to download once a day but because it fails it tries updating every time the cron job runs, i.e., every hour.

I did more googling and it was mentioned somewhere that they are now using cloudflare for distribution and the 403 error could be a result of a captcha that is not visible. I cannot verify this and the captcha does not show up when using the browser.

@provels let me know if you get this to work. Otherwise, I am inclined to delete this from my list and go with other lists, e.g., shown here.

RonpfS

@revengineer Try using State Flex
You may have to disable the list for 48hrs then re-enable it to circumvent the blocking.

provels

@revengineer
Did not work. Will change to Flex and see how it goes.

[ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded

 [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
  Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ PhishTank_v4 ]

provels

@provels said in Phishtank list download fail:

@revengineer
Did not work. Will change to Flex and see how it goes.

[ PhishTank_v4 ]		 Downloading update .. 509 Bandwidth Limit Exceeded

 [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ]
  Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ PhishTank_v4 ]

Failed with Flex as well, sorry.

RonpfS

@provels said in Phishtank list download fail:

Maybe post the log with Flex so we can see something.
What URL are we talking about ?

revengineer

@provels You are actually getting a different error than I did. Mine was a 403 error. The 509 may be the result of not using an API key, which seems to be required for automated downloads.

revengineer

@RonpfS What log file are you talking about? The pfblockerng.log only shows

[ PhishTank ]		 Downloading update .. 403 Forbidden

 [ DNSBL_Phishing - PhishTank ] Download FAIL
  Firewall and/or IDS are not blocking download.

  Restoring previously downloaded file

The error.log only shows

 [ DNSBL_Phishing - PhishTank ] Download Fail
  Firewall and/or IDS are not blocking download.

Neither is very informative. If you know of more detailed logs, please let me know.

RonpfS

@revengineer said in Phishtank list download fail:

@RonpfS What log file are you talking about?

Well something with timestamp help a lot!
So is it the API URL ? Why don't you post the URL masking any key...

revengineer

@RonpfS I did not see anything with a time stamp. The URL is

http://data.phishtank.com/data/online-valid.csv

and if you have an API key, it is

http://data.phishtank.com/data/<your app key>/online-valid.csv

RonpfS

Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2

revengineer

@RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.

RonpfS

@revengineer yes try it.

Why don't you register to pull with the API key?

revengineer

@RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.

RonpfS

@revengineer said in Phishtank list download fail:

http://data.phishtank.com/data/online-valid.csv

And the browser goes thru the same pfsense ?
Maybe you are on a block list, wait another 2 days before testing.
Try curl in a shell on the pfsense device to see more log.

revengineer

@RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.

Let me ask you this: Is the PhishTank feed actually working for you?

RonpfS

@revengineer said in Phishtank list download fail:

Let me ask you this: Is the PhishTank feed actually working for you?

Yes.