Feed Update Issue -- Talos

[ghkrauss]

Shown below is a recent update.
UPDATE PROCESS START [ 05/02/19 09:08:59 ]

===[ DNSBL Process ]================================================

Loading DNSBL Statistics... completed
Loading DNSBL Whitelist... completed

[ EasyList ] exists.
[ EasyPrivacy ] exists.
[ Adaway ] exists.
[ D_Me_ADs ] exists.
[ D_Me_Tracking ] exists.
[ hpHosts_ATS ] exists.
[ Cameleon ] exists.
[ SBL_ADs ] exists.
[ Yoyo ] exists.
[ Abuse_DOMBL ] exists.
[ Abuse_URLBL ] exists.
[ Abuse_Zeus_BD ] exists.
[ BBC_DC2 ] exists.
[ SWC ] exists. [ 05/02/19 09:09:00 ]
[ D_Me_Malv ] exists.
[ D_Me_Malw ] exists.
[ ISC_SDH ] exists.
[ MDS ] exists.
[ MDS_Immortal ] exists.
[ MDL ] exists.
[ MVPS ] exists.
[ Spam404 ] exists.
[ SFS_Toxic_BD ] exists.
Saving DNSBL database... completed

===[ GeoIP Process ]============================================

===[ IPv4 Process ]=================================================

[ Abuse_DYRE_v4 ] Downloading update .. 404 Not Found

[ pfB_PRI1_v4 - Abuse_DYRE_v4 ] Download FAIL
Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ Abuse_DYRE_v4 ]

[ Abuse_Feodo_C2_v4 ] exists.
[ Abuse_IPBL_v4 ] exists.
[ Abuse_SSLBL_v4 ] exists.
[ Abuse_Zeus_v4 ] exists.
[ BBC_C2_v4 ] exists.
[ CINS_army_v4 ] exists.
[ ET_Block_v4 ] exists.
[ ET_Comp_v4 ] exists.
[ ISC_1000_30_v4 ] exists.
[ ISC_Block_v4 ] exists.
[ Spamhaus_Drop_v4 ] exists.
[ Spamhaus_eDrop_v4 ] exists.
[ Talos_BL_v4 ] Downloading update .. 403 Forbidden

[ pfB_PRI1_v4 - Talos_BL_v4 ] Download FAIL
Firewall and/or IDS (Legacy mode only) are not blocking download.

The Following List has been REMOVED [ Talos_BL_v4 ]

===[ Aliastables / Rules ]==========================================

No changes to Firewall rules, skipping Filter Reload
No Changes to Aliases, Skipping pfctl Update

UPDATE PROCESS ENDED [ 05/02/19 09:09:04 ]

What is the solution to the Talos feed issue?

[Gertjan]

@ghkrauss said in Feed Update Issue -- Talos:

What is the solution to the Talos feed issue?

What is the issue ?

This :

@ghkrauss said in Feed Update Issue -- Talos:

[ Talos_BL_v4 ] Downloading update .. 403 Forbidden
[ pfB_PRI1_v4 - Talos_BL_v4 ] Download FAIL

Try the 'Talos' URL yourself in a web browser.
You should obtain some big list with IP's etc.
Or, the server that hosts the file is in a bad shape. It throws a "404" (the file was not found) in your face if it can't give you what you're asking for.
This happens. Servers go wako ones in a while. All depends on the admin of that site.
Maybe the file changed it's name ?

These lists, used by "pfBlockerNG", have to be maintained, also by you. Nothing is static, they can change.

[NogBadTheBad]

@Gertjan said in Feed Update Issue -- Talos:

These lists, used by "pfBlockerNG", have to be maintained, also by you. Nothing is static, they can change.

[ ISC_1000_30_v4 ]		 exists.
[ ISC_Block_v4 ]		 exists.
[ Spamhaus_Drop_v4 ]		 exists.
[ Spamhaus_eDrop_v4 ]		 exists.
[ Talos_BL_v4 ]			 Downloading update .. 403 Forbidden

 [ pfB_PRI1_v4 - Talos_BL_v4 ] Download FAIL
  Firewall and/or IDS (Legacy mode only) are not blocking download.

I wonder if its the redirect it's not liking, I just noticed this.

[Gertjan]

One point for @NogBadTheBad : you just discovered that a browser is probably somewhat smarter as the 'wget' or 'curl' used by 'pfBlockerNG'.

[linuxmanr4]

It's exactly the same for me.

I provisionally changed the url to Amazon hosted and it seems to work.

https://talos-intelligence-site.s3.amazonaws.com/production/document_files/files/000/066/901/original/ip_filter.blf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIXACIED2SPMSC7GA%2F20190502%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20190502T162159Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=3e1120e4e5e9b3d2b5e516f03adcfa299a9ef616c0aa859424a12d8d41c5d2d7

[ Spamhaus_Drop_v4 ]		 exists.
[ Spamhaus_eDrop_v4 ]		 exists.
[ Talos_BL_v4 ]			 Downloading update .. 200 OK. completed ..

I took note of the previous url in case it works again.

https://www.talosintelligence.com/feeds/ip-filter.blf

Greetings.

[provels]

@linuxmanr4 Your link doesn't seem to work anymore. I believe the extended information has caused it to expire. Same with me. Things are munged server-side.

[linuxmanr4]

That's right @provels , it worked for a while and then it did the same thing again.

I am going to report this problem to pfBlockerNG.

[RonpfS]

@linuxmanr4
There is an "Expires=3600" in the redirect URL

[BBcan177]

https://twitter.com/BBcan177/status/1124471820940468224

[neoaeon]

The user agent curlopt was resulting in a 403 from Cloudflare, seems they didn't like Google Chrome 43 circa 2015.

I changed my user agent to plain old 'curl' and everything is working again.

edit /usr/local/pkg/pfblockerng/pfblockerng.inc line 118:
from:

$pfb['curl_defaults'] = array(  CURLOPT_USERAGENT       => 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 Chrome/43.0.2357.65 Safari/537.36',

to:

$pfb['curl_defaults'] = array(  CURLOPT_USERAGENT       => 'curl',

edit /usr/local/pkg/pfblocker/pfblockerng_install.inc line 59:
from:

curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 Chrome/43.0.2357.65 Safari/537.36');

to:

curl_setopt($ch, CURLOPT_USERAGENT, 'curl');

-neo

P.s. @BBcan177 if you're going to fix this, while you're in there can you replace 1.1.1.1 with the RFC 5737 compliant 192.0.2.0 so we can use Cloudflare DNS w/o having to edit pfblockerng.inc and pfblockerng.sh please? :) (don't forget about the regex on pfblockerng.sh line 992)

Edit: BTW, not sure what's going on with caching, but restart php-fam didn't cause an update, I had to delete the /usr/local/pkg/pfblockerng/.pfblockerng.* files and then restart php-fam for the change to activate.

Edit2: diff for 2.1.4_17, fixes cloudflare DNS and Talos blacklists. pfblockerng_2.1.4_17.diff

1. scp/sftp the diff file to /usr/local/pkg/pfblockerng
2. run the following command from a shell:

cd /usr/local/pkg/pfblockerng ; patch -p0 < pfblockerng_2.1.4_17.diff

[Digital_ADHD]

This worked for me, Thanks!

[ Talos_BL_v4 ] Downloading update .. 200 OK. completed ..

@neoaeon said in Feed Update Issue -- Talos:

edit /usr/local/pkg/pfblockerng/pfblockerng.inc line 118:
from:
$pfb['curl_defaults'] = array( CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 Chrome/43.0.2357.65 Safari/537.36',

to:
$pfb['curl_defaults'] = array( CURLOPT_USERAGENT => 'curl',

edit /usr/local/pkg/pfblocker/pfblockerng_install.inc line 59:
from:
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 Chrome/43.0.2357.65 Safari/537.36');

to:
curl_setopt($ch, CURLOPT_USERAGENT, 'curl');

[linuxmanr4]

Thanks @neoaeon, after modifying the files the problem has been solved.

[RonpfS]

The feed now download without any modification to User agent.

[Digital_ADHD]

@RonpfS said in Feed Update Issue -- Talos:

The feed now download without any modification to User agent.

Thanks for the update!

[provels]

Looks like this feed is borked again. Worked fine for a while. Redid the useragent mods to fix.

[Digital_ADHD]

@provels updating useragent fixed this again for me

[neoaeon]

Zombie thread resurrection as this issue is back due to a regression.

Link to new thread: https://forum.netgate.com/topic/161817/pfblockerng-2-1x-fix-for-talos-feed-and-cloudflare-1-1-1-1-dns