Phishtank list download fail
-
I am using the Phishtank block list but have been receiving 403 download errors on an hourly basis even though the list is set to update daily. The error only occurs when the cron job executes or I manually select the "Cron" force update option. If I force "Reload All" then the list will download properly.
There are two issues here:
-
Why is the download failing? I have registered and am using the link with api key, which should not be limited.
-
Why is the download occurring hourly if the update frequency is set to daily?
Help much appreciated.
Thank you in advance.
-
-
Pinging again on this topic. I am really puzzled because I can download the PhishTank blocklist without issues through any web browser but I get a 403 error using the pfBlockerNG cron job. Interestingly, PhishTank.ip and PhishTank.txt files do get created in the /var/db/pfblockerng/dnsbl directory and seem to have valid content. But I also see a file PhishTank.fail. Help greatly appreciated.
-
Probably the quota system, or the way their access works, can detect the difference between a user-browser download, and a 'curl' (whatever method the pfBlocker script use) type download.
Time to contact their support.
-
@revengineer
I just added it to mine. I only update pfB once a day, so I'll see what happens tonight. Maybe you're blacklisting your own IP by checking too often?But checking your link with a browser first time ever gives me this. Maybe their .csv is borked in some way.
You have exceeded the request rate limit for this method. Please see the response headers for usage stats. For more information about rate limiting on Phishtank, please see our developer site: http://www.phishtank.com/developer_info.php -
There is a not on the PhishTank site that automatic downloads require an API key and I got one of those. I can download with a browser without problems, but pfblockerng fails. So it does not seem to be a quota issue. I have this set to download once a day but because it fails it tries updating every time the cron job runs, i.e., every hour.
I did more googling and it was mentioned somewhere that they are now using cloudflare for distribution and the 403 error could be a result of a captcha that is not visible. I cannot verify this and the captcha does not show up when using the browser.
@provels let me know if you get this to work. Otherwise, I am inclined to delete this from my list and go with other lists, e.g., shown here.
-
@revengineer Try using State Flex
You may have to disable the list for 48hrs then re-enable it to circumvent the blocking. -
@revengineer
Did not work. Will change to Flex and see how it goes.[ PhishTank_v4 ] Downloading update .. 509 Bandwidth Limit Exceeded [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ] Firewall and/or IDS (Legacy mode only) are not blocking download. The Following List has been REMOVED [ PhishTank_v4 ]Peder
2.5.2 (amd64) ZFS - Hyper-V Virtual Machine, Gen 1, 2 v-CPUs, 2.5 GB RAM (Fixed), 8GB VHDX (Dynamic)
Packages : Cron, Iperf, Mailreport, Notes, Nut, OpenVPN, pfBlockerNG-devel, RRD_Summary, Service Watchdog, System_Patches -
@provels said in Phishtank list download fail:
@revengineer
Did not work. Will change to Flex and see how it goes.[ PhishTank_v4 ] Downloading update .. 509 Bandwidth Limit Exceeded [ pfB_CustomList_v4 - PhishTank_v4 ] Download FAIL [ 11/18/20 23:48:35 ] Firewall and/or IDS (Legacy mode only) are not blocking download. The Following List has been REMOVED [ PhishTank_v4 ]Failed with Flex as well, sorry.
-
@provels said in Phishtank list download fail:
Maybe post the log with Flex so we can see something.
What URL are we talking about ? -
@provels You are actually getting a different error than I did. Mine was a 403 error. The 509 may be the result of not using an API key, which seems to be required for automated downloads.
-
@RonpfS What log file are you talking about? The pfblockerng.log only shows
[ PhishTank ] Downloading update .. 403 Forbidden [ DNSBL_Phishing - PhishTank ] Download FAIL Firewall and/or IDS are not blocking download. Restoring previously downloaded fileThe error.log only shows
[ DNSBL_Phishing - PhishTank ] Download Fail Firewall and/or IDS are not blocking download.Neither is very informative. If you know of more detailed logs, please let me know.
-
@revengineer said in Phishtank list download fail:
@RonpfS What log file are you talking about?
Well something with timestamp help a lot!
So is it the API URL ? Why don't you post the URL masking any key... -
@RonpfS I did not see anything with a time stamp. The URL is
http://data.phishtank.com/data/online-valid.csvand if you have an API key, it is
http://data.phishtank.com/data/<your app key>/online-valid.csv -
Any luck with : https://data.phishtank.com/data/online-valid.csv.bz2
https://data.phishtank.com/data/API_KEY/online-valid.csv.bz2 -
@RonpfS Are you asking whether I have tried the .bz2 extension? The answer is not but I can try.
-
@revengineer yes try it.
Why don't you register to pull with the API key?
-
@RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.
-
@revengineer said in Phishtank list download fail:
http://data.phishtank.com/data/online-valid.csv
And the browser goes thru the same pfsense ?
Maybe you are on a block list, wait another 2 days before testing.
Try curl in a shell on the pfsense device to see more log. -
@RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.
Let me ask you this: Is the PhishTank feed actually working for you?
-
@revengineer said in Phishtank list download fail:
Let me ask you this: Is the PhishTank feed actually working for you?
Yes.
-
@RonpfS And yes, the browser is behind the same pfSense that I am trying to install the feed on.
-
@RonpfS I am stumped. Not sure what else to try.
-
Open a shell and try curl ...
Anything in the /tmp folder ? -
@RonpfS So tried curl and it returns to the commandline without downloading a file. The verbose output is
* Trying 104.17.177.85:80... * TCP_NODELAY set * Connected to data.phishtank.com (104.17.177.85) port 80 (#0) > GET /data/online-valid.csv HTTP/1.1 > Host: data.phishtank.com > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 301 Moved Permanently < Date: Sat, 21 Nov 2020 01:31:10 GMT < Transfer-Encoding: chunked < Connection: keep-alive < Cache-Control: max-age=3600 < Expires: Sat, 21 Nov 2020 02:31:10 GMT < Location: https://data.phishtank.com/data/online-valid.csv < cf-request-id: 068a0644d00000cee439007000000001 < Server: cloudflare < CF-RAY: 5f56a64e1f94cee4-IAD < * Connection #0 to host data.phishtank.com left intact -
@revengineer
Tried HTTPS? I was able to add the https://data.phishtank.com/data/online-valid.csv.bz2
from the pfB Feeds page and for the heck of it registered for an API key and added it to the link. Ran w/o error on Force/Reload and Cron. That's the same list, just a different format, right?
-
@provels Good point. I did not write but did indeed try many combinations. For the https case with .bz extension, I get
curl -v https://data.phishtank.com/data/<api_key>/online-valid.csv.bz2 * Trying 104.16.101.75:443... * TCP_NODELAY set * Connected to data.phishtank.com (104.16.101.75) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Aug 17 00:00:00 2020 GMT * expire date: Aug 17 12:00:00 2021 GMT * subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x803abb800) > GET /data/<api_key>/online-valid.csv.bz2 HTTP/2 > Host: data.phishtank.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! < HTTP/2 302 < date: Sat, 21 Nov 2020 13:26:33 GMT < content-type: text/html; charset=UTF-8 < set-cookie: __cfduid=de8c92bb34f744f36bf09f5cfa9c6a7c21605965193; expires=Mon, 21-Dec-20 13:26:33 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure < x-request-limit-interval: 10800 Seconds < x-request-limit: 12 < x-request-count: 1 < location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605965203&Signature=efXCsFqG1q8UlLtJihn7Nj6fXJRyjTjXVq96b2gvsnAhyOiM9Kfv4mpuCfY...[ABBREVIATED] < cf-cache-status: DYNAMIC < cf-request-id: 068c953aa400002acc66ad6000000001 < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < strict-transport-security: max-age=15552000 < server: cloudflare < cf-ray: 5f5abe3ddf3e2acc-IAD < * Connection #0 to host data.phishtank.com left intact -
I have tried downloading the feed from an Ubuntu server as well. The curl command fails in the same manner, wget command works.
@provels I think it may be time for me to contact the PhishTank folks. for comparison, can you post your "curl -v" output?
-
@revengineer said in Phishtank list download fail:
curl -v
curl -v https://data.phishtank.com/data/<api-key>/online-valid.csv.bz2 * Trying 2606:4700::6810:654b:443... * TCP_NODELAY set * Connected to data.phishtank.com (2606:4700::6810:654b) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Aug 17 00:00:00 2020 GMT * expire date: Aug 17 12:00:00 2021 GMT * subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x803aba800) > GET /data/<api-key>/online-valid.csv.bz2 HTTP/2 > Host: data.phishtank.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! < HTTP/2 302 < date: Sat, 21 Nov 2020 15:14:17 GMT < content-type: text/html; charset=UTF-8 < set-cookie: __cfduid=db5c57b87a118488312b7047180d6f9101605971657; expires=Mon, 21-Dec-20 15:14:17 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure < x-request-limit-interval: 10800 Seconds < x-request-limit: 12 < x-request-count: 1 < location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605971667&Signature=iUZaI4nsb9LNji0tMhiEsrZW9fryn751OzXVP ... ETC, ETC. < cf-cache-status: DYNAMIC < cf-request-id: 068cf7dc0900007f68c79e9000000001 < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < strict-transport-security: max-age=15552000 < server: cloudflare < cf-ray: 5f5b5c0cd90f7f68-ORD < * Connection #0 to host data.phishtank.com left intact -
@provels This looks similar to my output. And after this you have a file called online-valid.csv.bz2 in the directory you called the command from?
-
@revengineer said in Phishtank list download fail:
online-valid.csv.bz2
Well, um, no. Ran from /root, not there. Searched the tree, too. But like I said, worked OK using HTTPS from Force and Cron within pfB. Does curl actually do a GET? I'm no command line expert.
-
@provels I tried one more time using API key, https, and .bz2 ending and it's just not working. The log is below. A google search for "cloudflare 403 error" yields upwards of half a million hits, and the solution may be somewhere in there. I sent email to PhishTank but do not expect a response. Beyond that this seems like a waste of time to pursue as it seems to be specific to me.
CRON PROCESS START [ 11/21/20 13:44:12 ] [ BD_IPs ] Remote timestamp: Sat, 21 Nov 2020 18:32:31 GMT Local timestamp: Sat, 21 Nov 2020 17:32:30 GMT Update found [ FireHOL3_IPs ] Remote timestamp: Sat, 21 Nov 2020 18:33:37 GMT Local timestamp: Sat, 21 Nov 2020 17:18:10 GMT Update found [ PhishTank ] Previous download failed. Re-attempt download UPDATE PROCESS START [ 11/21/20 13:44:14 ] ===[ DNSBL Process ]================================================ [ EasyList ] exists. [ EasyPrivacy ] exists. [ Adaway ] exists. [ Cameleon ] exists. [ yoyo ] exists. [ MDS ] exists. [ MDL ] Downloading update .. 200 OK No Domains Found [ URLhaus ] exists. [ OpenPhish ] exists. [ 11/21/20 13:44:15 ] [ PhishingArmy ] exists. [ PhishTank ] Downloading update .. 403 Forbidden [ DNSBL_Phishing - PhishTank ] Download FAIL Firewall and/or IDS are not blocking download. [ oisd ] exists. [ DNSBL_IP ] Updating aliastable [ 11/21/20 13:44:20 ]... no changes. Total IP count = 307378 ===[ Continent Process ]============================================ [ pfB_Top_v4 ] exists. [ 11/21/20 13:44:21 ] [ pfB_Top_v6 ] exists. ===[ IPv4 Process ]================================================= [ BD_IPs ] Downloading update .. 200 OK. completed .. Reputation (Max=5) - Range(s) 182.138.158.|192.241.218.|192.241.219.|192.241.233.|192.241.234.|192.241.235.|192.241.236.|192.241.237.|192.241.238.|192.241.239.|124.235.138.|124.227.31.|196.52.43.|167.248.133.|92.118.161.|162.243.128.|87.251.70.|221.13.12.|81.161.63.|192.35.168.|170.130.187.|74.120.14.|185.202.1.|185.202.2.|104.140.188.|104.206.128.|102.165.30.|162.142.125.| Reputation -Max Stats ------------------------------ Blacklisted Match Ranges IPs Ranges IPs ------------------------------ 28 290 0 0 [ ET_Block_IPs ] exists. [ ET_Comp_IPs ] exists. [ FireHOL3_IPs ] Downloading update .. 200 OK. completed .. Reputation (Max=5) - Range(s) 124.156.50.|124.156.54.|124.156.55.|124.156.62.|124.156.64.|118.186.203.|194.187.19.|42.247.5.|59.177.77.|59.177.78.|59.177.79.|92.118.160.|92.118.161.|59.99.40.|23.129.64.|193.32.8.|49.51.160.|49.51.161.|170.130.187.|178.72.68.|178.72.69.|178.72.70.|178.72.71.|91.240.243.|199.195.251.|184.105.247.|103.52.216.|103.52.217.|164.52.24.|178.17.170.|89.33.193.|162.62.17.|162.62.26.|192.241.208.|5.45.207.|192.241.211.|192.241.212.|192.241.213.|192.241.214.|192.241.215.|192.241.216.|192.241.217.|192.241.218.|192.241.219.|192.241.220.|192.241.221.|192.241.222.|192.241.231.|192.241.232.|192.241.233.|192.241.236.|192.241.238.|37.9.13.|203.243.10.|121.169.34.|107.6.162.|107.6.168.|178.174.136.|178.174.137.|114.122.101.|114.122.107.|209.17.96.|209.17.97.|124.156.240.|124.156.241.|124.156.245.|45.113.70.|45.113.71.|185.191.171.|197.5.145.|49.51.8.|49.51.9.|170.106.36.|170.106.37.|170.106.38.|94.180.24.|94.180.25.|94.180.28.|170.106.76.|170.106.81.|89.46.223.|45.148.10.|192.35.168.|185.202.1.|178.175.132.|125.64.103.|200.73.128.|200.73.129.|104.206.128.|202.164.139.|102.165.30.|147.92.153.|109.188.125.|109.188.126.|65.49.20.|182.140.235.|202.114.176.|113.141.70.|213.180.203.|195.154.122.|195.154.123.|124.205.84.|192.42.116.|74.82.47.|121.147.227.|206.189.223.|114.119.128.|185.100.87.|114.119.135.|114.119.137.|114.119.158.|167.160.184.|176.212.104.|176.212.108.|87.236.208.|121.178.247.|199.249.230.|194.87.138.|91.192.103.|45.140.17.|150.109.167.|150.109.170.|150.109.180.|150.109.181.|150.109.182.|49.51.12.|162.247.74.|109.70.100.|58.216.176.|185.220.101.|185.220.102.|95.163.255.|218.89.77.|121.149.104.|172.172.26.|176.214.44.|172.172.30.|80.94.93.|192.99.175.|74.120.14.|104.140.188.|198.143.175.|13.66.139.|5.255.231.|162.142.125.| Reputation -Max Stats ------------------------------ Blacklisted Match Ranges IPs Ranges IPs ------------------------------ 145 1687 1 0 ===[ IPv6 Process ]================================================= ===[ Reputation - pMax ]====================================== Querying for repeat offenders ( pMax=50 ) [ 11/21/20 13:44:23 ] Reputation -pMax ( None ) ===[ Reputation - dMax ]====================================== Querying for repeat offenders ( dMax=5 ) [ 11/21/20 13:44:23 ] Classifying repeat offenders by GeoIP Processing [ Block ] IPs Removing [ Block ] IPs Removed the following IP ranges: 171.25.193.|216.83.57.|185.234.219.|164.52.24.|59.46.13.|193.169.252.|61.177.172.|222.192.88.|91.241.19.|184.105.247.|218.92.0.|192.241.209.|192.241.210.|192.241.215.|84.38.187.|205.185.127.|5.188.206.|45.87.43.|202.165.22.|213.202.233.|68.183.231.|2.57.122.|206.189.171.|209.141.54.|112.85.42.|118.244.128.|180.215.215.| Reputation - dMax Stats ------------------------------ Blacklisted Match Ranges IPs Ranges IPs ------------------------------ 27 25 1 0 ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload Updating: pfB_Top_v4 no changes. Updating: pfB_Top_v6 no changes. Updating: pfB_BinaryDefense 30 addresses added.37 addresses deleted. Updating: pfB_EmergingThreatsDShield 2 addresses added.3 addresses deleted. Updating: pfB_FireHOL3 10405 addresses added.57 addresses deleted. ===[ FINAL Processing ]===================================== [ Original IP count ] [ 64441 ] ===[ Deny List IP Counts ]=========================== 62227 total 22166 /var/db/pfblockerng/deny/FireHOL3_IPs.txt 19038 /var/db/pfblockerng/deny/pfB_Top_v4.txt 6944 /var/db/pfblockerng/deny/pfB_Top_v6.txt 6413 /var/db/pfblockerng/deny/ET_Comp_IPs.txt 5310 /var/db/pfblockerng/deny/BD_IPs.txt 2356 /var/db/pfblockerng/deny/ET_Block_IPs.txt ===[ DNSBL Domain/IP Counts ] =================================== 849719 total 378541 /var/db/pfblockerng/dnsbl/oisd.txt 307327 /var/db/pfblockerng/dnsbl/URLhaus.ip 91552 /var/db/pfblockerng/dnsbl/URLhaus.txt 25637 /var/db/pfblockerng/dnsbl/MDS.txt 17223 /var/db/pfblockerng/dnsbl/EasyList.txt 11980 /var/db/pfblockerng/dnsbl/PhishingArmy.txt 6712 /var/db/pfblockerng/dnsbl/Cameleon.txt 4093 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt 2901 /var/db/pfblockerng/dnsbl/OpenPhish.txt 2003 /var/db/pfblockerng/dnsbl/Adaway.txt 1699 /var/db/pfblockerng/dnsbl/yoyo.txt 21 /var/db/pfblockerng/dnsbl/OpenPhish.ip 20 /var/db/pfblockerng/dnsbl/EasyList.ip 10 /var/db/pfblockerng/dnsbl/EasyPrivacy.ip 0 /var/db/pfblockerng/dnsbl/PhishTank.fail ====================[ Last Updated List Summary ]============== Nov 20 00:30 ET_Block_IPs Nov 20 00:30 ET_Comp_IPs Nov 21 09:00 pfB_Top_v4 Nov 21 09:00 pfB_Top_v6 Nov 21 13:32 BD_IPs Nov 21 13:33 FireHOL3_IPs IPv4 alias tables IP count ----------------------------- 362661 IPv6 alias tables IP count ----------------------------- 6944 Alias table IP Counts ----------------------------- 369605 total 307378 /var/db/aliastables/pfB_DNSBLIP.txt 22166 /var/db/aliastables/pfB_FireHOL3.txt 19038 /var/db/aliastables/pfB_Top_v4.txt 8769 /var/db/aliastables/pfB_EmergingThreatsDShield.txt 6944 /var/db/aliastables/pfB_Top_v6.txt 5310 /var/db/aliastables/pfB_BinaryDefense.txt pfSense Table Stats ------------------- table-entries hard limit 2000000 Table Usage Count 486768 UPDATE PROCESS ENDED [ 11/21/20 13:44:27 ] -
@revengineer
Yeah, short of a typo on the URL line or a mistake in the API key, that's all I have. Sorry. -
@provels No worries, I thank you for your time to help me chase this down as far as we could.
There is one more question: Which version of pfBockerNG are you using? The devel version, or the older one. I still use the older (non-devel) version and that could be a difference.
-
@revengineer
I've only used the devel. Never had issues (at least not self-inflicted). -
@provels Thanks, I may update the version. I know that the author recommends the devel version for a long time, but for me this always sounded too much like "beta". Cheers!
