Phishtank list download fail
-
@revengineer yes try it.
Why don't you register to pull with the API key?
-
@RonpfS I do have an API key. I tried the link with .bz2 extension with and without API key and with and without FLEX setting. In all cases I get the 403 error. Each of these links work fine in a web browser.
-
@revengineer said in Phishtank list download fail:
http://data.phishtank.com/data/online-valid.csv
And the browser goes thru the same pfsense ?
Maybe you are on a block list, wait another 2 days before testing.
Try curl in a shell on the pfsense device to see more log. -
@RonpfS I tried that before as well. I actually forget that I turned it off and it was off for weeks. When I turned this feed back on the errors started right away.
Let me ask you this: Is the PhishTank feed actually working for you?
-
@revengineer said in Phishtank list download fail:
Let me ask you this: Is the PhishTank feed actually working for you?
Yes.
-
@RonpfS And yes, the browser is behind the same pfSense that I am trying to install the feed on.
-
@RonpfS I am stumped. Not sure what else to try.
-
Open a shell and try curl ...
Anything in the /tmp folder ? -
@RonpfS So tried curl and it returns to the commandline without downloading a file. The verbose output is
* Trying 104.17.177.85:80... * TCP_NODELAY set * Connected to data.phishtank.com (104.17.177.85) port 80 (#0) > GET /data/online-valid.csv HTTP/1.1 > Host: data.phishtank.com > User-Agent: curl/7.68.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 301 Moved Permanently < Date: Sat, 21 Nov 2020 01:31:10 GMT < Transfer-Encoding: chunked < Connection: keep-alive < Cache-Control: max-age=3600 < Expires: Sat, 21 Nov 2020 02:31:10 GMT < Location: https://data.phishtank.com/data/online-valid.csv < cf-request-id: 068a0644d00000cee439007000000001 < Server: cloudflare < CF-RAY: 5f56a64e1f94cee4-IAD < * Connection #0 to host data.phishtank.com left intact -
@revengineer
Tried HTTPS? I was able to add the https://data.phishtank.com/data/online-valid.csv.bz2
from the pfB Feeds page and for the heck of it registered for an API key and added it to the link. Ran w/o error on Force/Reload and Cron. That's the same list, just a different format, right?
-
@provels Good point. I did not write but did indeed try many combinations. For the https case with .bz extension, I get
curl -v https://data.phishtank.com/data/<api_key>/online-valid.csv.bz2 * Trying 104.16.101.75:443... * TCP_NODELAY set * Connected to data.phishtank.com (104.16.101.75) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Aug 17 00:00:00 2020 GMT * expire date: Aug 17 12:00:00 2021 GMT * subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x803abb800) > GET /data/<api_key>/online-valid.csv.bz2 HTTP/2 > Host: data.phishtank.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! < HTTP/2 302 < date: Sat, 21 Nov 2020 13:26:33 GMT < content-type: text/html; charset=UTF-8 < set-cookie: __cfduid=de8c92bb34f744f36bf09f5cfa9c6a7c21605965193; expires=Mon, 21-Dec-20 13:26:33 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure < x-request-limit-interval: 10800 Seconds < x-request-limit: 12 < x-request-count: 1 < location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605965203&Signature=efXCsFqG1q8UlLtJihn7Nj6fXJRyjTjXVq96b2gvsnAhyOiM9Kfv4mpuCfY...[ABBREVIATED] < cf-cache-status: DYNAMIC < cf-request-id: 068c953aa400002acc66ad6000000001 < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < strict-transport-security: max-age=15552000 < server: cloudflare < cf-ray: 5f5abe3ddf3e2acc-IAD < * Connection #0 to host data.phishtank.com left intact -
I have tried downloading the feed from an Ubuntu server as well. The curl command fails in the same manner, wget command works.
@provels I think it may be time for me to contact the PhishTank folks. for comparison, can you post your "curl -v" output?
-
@revengineer said in Phishtank list download fail:
curl -v
curl -v https://data.phishtank.com/data/<api-key>/online-valid.csv.bz2 * Trying 2606:4700::6810:654b:443... * TCP_NODELAY set * Connected to data.phishtank.com (2606:4700::6810:654b) port 443 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * successfully set certificate verify locations: * CAfile: /usr/local/share/certs/ca-root-nss.crt CApath: none * TLSv1.2 (OUT), TLS header, Certificate Status (22): * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-ECDSA-AES128-GCM-SHA256 * ALPN, server accepted to use h2 * Server certificate: * subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com * start date: Aug 17 00:00:00 2020 GMT * expire date: Aug 17 12:00:00 2021 GMT * subjectAltName: host "data.phishtank.com" matched cert's "*.phishtank.com" * issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3 * SSL certificate verify ok. * Using HTTP2, server supports multi-use * Connection state changed (HTTP/2 confirmed) * Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0 * Using Stream ID: 1 (easy handle 0x803aba800) > GET /data/<api-key>/online-valid.csv.bz2 HTTP/2 > Host: data.phishtank.com > user-agent: curl/7.68.0 > accept: */* > * Connection state changed (MAX_CONCURRENT_STREAMS == 256)! < HTTP/2 302 < date: Sat, 21 Nov 2020 15:14:17 GMT < content-type: text/html; charset=UTF-8 < set-cookie: __cfduid=db5c57b87a118488312b7047180d6f9101605971657; expires=Mon, 21-Dec-20 15:14:17 GMT; path=/; domain=.phishtank.com; HttpOnly; SameSite=Lax; Secure < x-request-limit-interval: 10800 Seconds < x-request-limit: 12 < x-request-count: 1 < location: https://d1750zhbc38ec0.cloudfront.net/datadumps/verified_online.csv.bz2?Expires=1605971667&Signature=iUZaI4nsb9LNji0tMhiEsrZW9fryn751OzXVP ... ETC, ETC. < cf-cache-status: DYNAMIC < cf-request-id: 068cf7dc0900007f68c79e9000000001 < expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct" < strict-transport-security: max-age=15552000 < server: cloudflare < cf-ray: 5f5b5c0cd90f7f68-ORD < * Connection #0 to host data.phishtank.com left intact -
@provels This looks similar to my output. And after this you have a file called online-valid.csv.bz2 in the directory you called the command from?
-
@revengineer said in Phishtank list download fail:
online-valid.csv.bz2
Well, um, no. Ran from /root, not there. Searched the tree, too. But like I said, worked OK using HTTPS from Force and Cron within pfB. Does curl actually do a GET? I'm no command line expert.
-
@provels I tried one more time using API key, https, and .bz2 ending and it's just not working. The log is below. A google search for "cloudflare 403 error" yields upwards of half a million hits, and the solution may be somewhere in there. I sent email to PhishTank but do not expect a response. Beyond that this seems like a waste of time to pursue as it seems to be specific to me.
CRON PROCESS START [ 11/21/20 13:44:12 ] [ BD_IPs ] Remote timestamp: Sat, 21 Nov 2020 18:32:31 GMT Local timestamp: Sat, 21 Nov 2020 17:32:30 GMT Update found [ FireHOL3_IPs ] Remote timestamp: Sat, 21 Nov 2020 18:33:37 GMT Local timestamp: Sat, 21 Nov 2020 17:18:10 GMT Update found [ PhishTank ] Previous download failed. Re-attempt download UPDATE PROCESS START [ 11/21/20 13:44:14 ] ===[ DNSBL Process ]================================================ [ EasyList ] exists. [ EasyPrivacy ] exists. [ Adaway ] exists. [ Cameleon ] exists. [ yoyo ] exists. [ MDS ] exists. [ MDL ] Downloading update .. 200 OK No Domains Found [ URLhaus ] exists. [ OpenPhish ] exists. [ 11/21/20 13:44:15 ] [ PhishingArmy ] exists. [ PhishTank ] Downloading update .. 403 Forbidden [ DNSBL_Phishing - PhishTank ] Download FAIL Firewall and/or IDS are not blocking download. [ oisd ] exists. [ DNSBL_IP ] Updating aliastable [ 11/21/20 13:44:20 ]... no changes. Total IP count = 307378 ===[ Continent Process ]============================================ [ pfB_Top_v4 ] exists. [ 11/21/20 13:44:21 ] [ pfB_Top_v6 ] exists. ===[ IPv4 Process ]================================================= [ BD_IPs ] Downloading update .. 200 OK. completed .. Reputation (Max=5) - Range(s) 182.138.158.|192.241.218.|192.241.219.|192.241.233.|192.241.234.|192.241.235.|192.241.236.|192.241.237.|192.241.238.|192.241.239.|124.235.138.|124.227.31.|196.52.43.|167.248.133.|92.118.161.|162.243.128.|87.251.70.|221.13.12.|81.161.63.|192.35.168.|170.130.187.|74.120.14.|185.202.1.|185.202.2.|104.140.188.|104.206.128.|102.165.30.|162.142.125.| Reputation -Max Stats ------------------------------ Blacklisted Match Ranges IPs Ranges IPs ------------------------------ 28 290 0 0 [ ET_Block_IPs ] exists. [ ET_Comp_IPs ] exists. [ FireHOL3_IPs ] Downloading update .. 200 OK. completed .. Reputation (Max=5) - Range(s) 124.156.50.|124.156.54.|124.156.55.|124.156.62.|124.156.64.|118.186.203.|194.187.19.|42.247.5.|59.177.77.|59.177.78.|59.177.79.|92.118.160.|92.118.161.|59.99.40.|23.129.64.|193.32.8.|49.51.160.|49.51.161.|170.130.187.|178.72.68.|178.72.69.|178.72.70.|178.72.71.|91.240.243.|199.195.251.|184.105.247.|103.52.216.|103.52.217.|164.52.24.|178.17.170.|89.33.193.|162.62.17.|162.62.26.|192.241.208.|5.45.207.|192.241.211.|192.241.212.|192.241.213.|192.241.214.|192.241.215.|192.241.216.|192.241.217.|192.241.218.|192.241.219.|192.241.220.|192.241.221.|192.241.222.|192.241.231.|192.241.232.|192.241.233.|192.241.236.|192.241.238.|37.9.13.|203.243.10.|121.169.34.|107.6.162.|107.6.168.|178.174.136.|178.174.137.|114.122.101.|114.122.107.|209.17.96.|209.17.97.|124.156.240.|124.156.241.|124.156.245.|45.113.70.|45.113.71.|185.191.171.|197.5.145.|49.51.8.|49.51.9.|170.106.36.|170.106.37.|170.106.38.|94.180.24.|94.180.25.|94.180.28.|170.106.76.|170.106.81.|89.46.223.|45.148.10.|192.35.168.|185.202.1.|178.175.132.|125.64.103.|200.73.128.|200.73.129.|104.206.128.|202.164.139.|102.165.30.|147.92.153.|109.188.125.|109.188.126.|65.49.20.|182.140.235.|202.114.176.|113.141.70.|213.180.203.|195.154.122.|195.154.123.|124.205.84.|192.42.116.|74.82.47.|121.147.227.|206.189.223.|114.119.128.|185.100.87.|114.119.135.|114.119.137.|114.119.158.|167.160.184.|176.212.104.|176.212.108.|87.236.208.|121.178.247.|199.249.230.|194.87.138.|91.192.103.|45.140.17.|150.109.167.|150.109.170.|150.109.180.|150.109.181.|150.109.182.|49.51.12.|162.247.74.|109.70.100.|58.216.176.|185.220.101.|185.220.102.|95.163.255.|218.89.77.|121.149.104.|172.172.26.|176.214.44.|172.172.30.|80.94.93.|192.99.175.|74.120.14.|104.140.188.|198.143.175.|13.66.139.|5.255.231.|162.142.125.| Reputation -Max Stats ------------------------------ Blacklisted Match Ranges IPs Ranges IPs ------------------------------ 145 1687 1 0 ===[ IPv6 Process ]================================================= ===[ Reputation - pMax ]====================================== Querying for repeat offenders ( pMax=50 ) [ 11/21/20 13:44:23 ] Reputation -pMax ( None ) ===[ Reputation - dMax ]====================================== Querying for repeat offenders ( dMax=5 ) [ 11/21/20 13:44:23 ] Classifying repeat offenders by GeoIP Processing [ Block ] IPs Removing [ Block ] IPs Removed the following IP ranges: 171.25.193.|216.83.57.|185.234.219.|164.52.24.|59.46.13.|193.169.252.|61.177.172.|222.192.88.|91.241.19.|184.105.247.|218.92.0.|192.241.209.|192.241.210.|192.241.215.|84.38.187.|205.185.127.|5.188.206.|45.87.43.|202.165.22.|213.202.233.|68.183.231.|2.57.122.|206.189.171.|209.141.54.|112.85.42.|118.244.128.|180.215.215.| Reputation - dMax Stats ------------------------------ Blacklisted Match Ranges IPs Ranges IPs ------------------------------ 27 25 1 0 ===[ Aliastables / Rules ]========================================== No changes to Firewall rules, skipping Filter Reload Updating: pfB_Top_v4 no changes. Updating: pfB_Top_v6 no changes. Updating: pfB_BinaryDefense 30 addresses added.37 addresses deleted. Updating: pfB_EmergingThreatsDShield 2 addresses added.3 addresses deleted. Updating: pfB_FireHOL3 10405 addresses added.57 addresses deleted. ===[ FINAL Processing ]===================================== [ Original IP count ] [ 64441 ] ===[ Deny List IP Counts ]=========================== 62227 total 22166 /var/db/pfblockerng/deny/FireHOL3_IPs.txt 19038 /var/db/pfblockerng/deny/pfB_Top_v4.txt 6944 /var/db/pfblockerng/deny/pfB_Top_v6.txt 6413 /var/db/pfblockerng/deny/ET_Comp_IPs.txt 5310 /var/db/pfblockerng/deny/BD_IPs.txt 2356 /var/db/pfblockerng/deny/ET_Block_IPs.txt ===[ DNSBL Domain/IP Counts ] =================================== 849719 total 378541 /var/db/pfblockerng/dnsbl/oisd.txt 307327 /var/db/pfblockerng/dnsbl/URLhaus.ip 91552 /var/db/pfblockerng/dnsbl/URLhaus.txt 25637 /var/db/pfblockerng/dnsbl/MDS.txt 17223 /var/db/pfblockerng/dnsbl/EasyList.txt 11980 /var/db/pfblockerng/dnsbl/PhishingArmy.txt 6712 /var/db/pfblockerng/dnsbl/Cameleon.txt 4093 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt 2901 /var/db/pfblockerng/dnsbl/OpenPhish.txt 2003 /var/db/pfblockerng/dnsbl/Adaway.txt 1699 /var/db/pfblockerng/dnsbl/yoyo.txt 21 /var/db/pfblockerng/dnsbl/OpenPhish.ip 20 /var/db/pfblockerng/dnsbl/EasyList.ip 10 /var/db/pfblockerng/dnsbl/EasyPrivacy.ip 0 /var/db/pfblockerng/dnsbl/PhishTank.fail ====================[ Last Updated List Summary ]============== Nov 20 00:30 ET_Block_IPs Nov 20 00:30 ET_Comp_IPs Nov 21 09:00 pfB_Top_v4 Nov 21 09:00 pfB_Top_v6 Nov 21 13:32 BD_IPs Nov 21 13:33 FireHOL3_IPs IPv4 alias tables IP count ----------------------------- 362661 IPv6 alias tables IP count ----------------------------- 6944 Alias table IP Counts ----------------------------- 369605 total 307378 /var/db/aliastables/pfB_DNSBLIP.txt 22166 /var/db/aliastables/pfB_FireHOL3.txt 19038 /var/db/aliastables/pfB_Top_v4.txt 8769 /var/db/aliastables/pfB_EmergingThreatsDShield.txt 6944 /var/db/aliastables/pfB_Top_v6.txt 5310 /var/db/aliastables/pfB_BinaryDefense.txt pfSense Table Stats ------------------- table-entries hard limit 2000000 Table Usage Count 486768 UPDATE PROCESS ENDED [ 11/21/20 13:44:27 ] -
@revengineer
Yeah, short of a typo on the URL line or a mistake in the API key, that's all I have. Sorry. -
@provels No worries, I thank you for your time to help me chase this down as far as we could.
There is one more question: Which version of pfBockerNG are you using? The devel version, or the older one. I still use the older (non-devel) version and that could be a difference.
-
@revengineer
I've only used the devel. Never had issues (at least not self-inflicted). -
@provels Thanks, I may update the version. I know that the author recommends the devel version for a long time, but for me this always sounded too much like "beta". Cheers!
