PfBlocker list import problem ….

atrocity

Hello,

we use pfBlocker since months, we use it since the beta versions.
Now, we have 2 x PFSense : 2.3.2-RELEASE-p1 (amd64)
and pfBlockerNG version : 2.1.1_4

We have about 10 lists in our ip4 lists, and don't see the same number of IPs listed for the same exact lists …
But our main problem is that pfBlockerNG don't import 2 of our lists correctly :(
Her a exemple of a forced update status :

UPDATE PROCESS START [ 11/24/16 12:17:18 ]

Clearing all DNSBL Feeds…
** DNSBL Disabled **

===[ Continent Process ]============================================

….

===[ IPv4 Process ]=================================================

….

[ CSF ] Reload [ 11/24/16 12:17:26 ] . completed ..

[ CSF2 ] Reload . completed ..

===[ IPv6 Process ]=================================================

===[ Aliastables / Rules ]==========================================

No changes to Firewall rules, skipping Filter Reload

Updating: pfB_CSF
no changes.
Updating: pfB_CSF2
no changes.

===[ FINAL Processing ]=====================================

[ Original IP count ] [ 99559 ]

===[ Permit List IP Counts ]=========================

…..

===[ Deny List IP Counts ]===========================

57071 total
….
33 /var/db/pfblockerng/deny/CSF.txt
28 /var/db/pfblockerng/deny/CSF2.txt

====================[ Last Updated List Summary ]==============

….
Nov 24 12:00 CSF
Nov 24 12:15 CSF2
....

IPv4 alias tables IP count

94596

IPv6 alias tables IP count

4940

Alias table IP Counts

99536 total
....
33 /var/db/aliastables/pfB_CSF.txt
28 /var/db/aliastables/pfB_CSF2.txt

pfSense Table Stats

table-entries hard limit 1000000
Table Usage Count 184007

UPDATE PROCESS ENDED [ 11/24/16 12:17:30 ]

I removed the lines that don't concern the others lists, i leave only the CSF (ConfigServer firewall deny list, containing a simple list of IPs like :
61.12.1.200
115.248.248.85
195.134.76.233

There is no html formating and the IPs are one on each line.
As you can see, it import 2 lists. The 2 lists are the SAME EXACT list of IPs.
The fist one "CSF" show the it imported 33 IPs
The second one show it imported …. 28 IPs
:o :o :o

So i counted the 2 IPs lists, and on both i can only count ... 28 IPs ...
Then i removed some (the half) of this IPs, and the forced import show me the same exact result as on the top of this tread ....

33 /var/db/pfblockerng/deny/CSF.txt
28 /var/db/pfblockerng/deny/CSF2.txt

Well, well, well ....
I try many things ....
Deleted the content of the whole directory /var/db/pfblockerng/*
and did a try to import with force and without force. Same result ! :-\

Then i did a try to remove pfBlockerNG, reboot the firewall, install pfBlockerNG, redefine the ip4 lists.

A normal import (without force) work ! 8)

Then, cron should run the import after 1 hour. This is not working anymore, because he don't import the current and actualized lists again :'(

I see all this import problems with 2 PFSense firewalls, the first one is a Super Micro C2758, 8Gb ram, ssd, ....
the second one a old Watchguard
Everything works great on both, but not pfBlockerNG

PS: all our other lists are also not actualized correctly. For the sames lists, i can't see the same number of listed IPs, also if i start a forced update at the same exact moment on both firewalls.

What can be that wrong with this list import ?
Thanks for any idea.
Best regards
Atrocity 8)

atrocity

forgot to say that on the CSF list server, i SEE the Apache access ass successfull, and the lists are reachable in our whole network.
It can import it correctly 1 time, can't imagine that he can't do it later …

BBcan177

Is deduplication enabled? I would assume its due to that…

atrocity

Hello BBcan and thanks for your answer.
I checked de-duplication and here are the result of the 2 :

26 /var/db/aliastables/pfB_CSF.txt
1 /var/db/aliastables/pfB_CSF2.txt

33 /var/db/aliastables/pfB_CSF.txt

So also with de-duplication, the numbers of imported IPs are not the same.
Same thing for the other lists :

37525 /var/db/aliastables/pfB_Europe_v4.txt
30344 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
4940 /var/db/aliastables/pfB_Europe_v6.txt
3027 /var/db/aliastables/pfB_Blocklistde_SSH.txt
1342 /var/db/aliastables/pfB_Blocklistde_bots.txt
816 /var/db/aliastables/pfB_Spamhaus.txt
776 /var/db/aliastables/pfB_Rutgers_Attackers.txt
500 /var/db/aliastables/pfB_Blocklistde_ApacheDDOS.txt
409 /var/db/aliastables/pfB_Blocklistde_FTP21.txt
271 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt

46065 /var/db/aliastables/pfB_Europe_v4.txt
29329 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
6569 /var/db/aliastables/pfB_Europe_v6.txt
3369 /var/db/aliastables/pfB_Blocklistde_SSH.txt
1139 /var/db/aliastables/pfB_Blocklistde_bots.txt
816 /var/db/aliastables/pfB_Spamhaus.txt
252 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt
226 /var/db/aliastables/pfB_Rutgers_Attackers.txt

BBcan, i'm one of the guy that give you a access to our firewalls in Europe, some months ago during your beta ;-)
Thanks for any trick/idea to debug this.
Regards

BBcan177

Run a reload then execute the following command:

rm /var/db/pfblockerng/test1
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test1

Then after the next cron run:

rm /var/db/pfblockerng/test2
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test2

Then compare the differences:

awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

You could also send the comparison to a file for review:

awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

Then use a file editor like vi, ee, or nano to view the files
[ /var/db/pfblockerng/test1dif and /var/db/pfblockerng/test2dif ]

You can then grep one of the missing IPs and see if the IP is now blocked by a larger CIDR entry:

grep "1\.2\.3\.4"  /var/db/pfblockerng/deny/*
grep "1\.2\.3\."  /var/db/pfblockerng/deny/*
grep "1\.2\."  /var/db/pfblockerng/deny/*
grep "1\."  /var/db/pfblockerng/deny/*

You can use this link to see what a CIDR range includes:
https://www.ultratools.com/tools/netMask

You can send the results via email to make it easier to reply. (See general tab for email)

Please do your own testing first please :)

atrocity

Hello,

well, here are the results on of of the 2 firewalls :

[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

nothing found :(

Should i try to delete all the pfblocker/* and redo this tests ?
Thanks
Best regards

lispeedyg

Hi,
Has this ever been resolved??
I've been having the same issues and the OP. Currently on pfSense V 2.4.1 and pfBlockerNG v 2.1.2_1.
And, specifically, the only way I can actually have the list updated is to uninstall pfBlocker completely, i.e., Don't keep settings. And then re-start the process completely. All other facets work as intended. But, this has been a significant drawback.

Is there any further help anyone can provide?

BBcan177

Post some more details about what you're seeing … Keep in mind that Feeds won't update if the source of the Feed is unchanged (ie: previously downloaded feed is the same as the source feed), and the pfblockerng.log will show "Update not required".

lispeedyg

Hi,
Thanks for the quick reply..
My feed is actually a text file I host on the pfSense box at /usr/local/www/DenyIPs.txt
I add to the file manually as I'm informed of possible new threats.
I've pointed to the file source in two seperate ways, i.e., using the direct path above and also as http://[server-ip]/DenyIPs.txt

In both scenarios I can see the new IP entries in the file/browser. I then run both an Update and Reload for pfBlockerNG. And in all cases (other than completely deleting the application) the result is "no changes".

BBcan177

You need to use this format:

/usr/local/www/DenyIPs.txt

And keep in mind that Force Update|Reload just downloads the feed if its new (IE: wasn't previously downloaded).

You need to define the Frequency setting to say "once per hour" and then it will download the referenced Source Feed at that time via the Cron task.

lispeedyg

Thanks for that.
I've reverted to the original file pointer, i.e., /usr/local/www/DenyIPs.txt.
I've also revised the frequency to 1hour from weekly.
After the above, I manually ran Update/Cron/Reload. And, the new IP's are still not visible and the number of IP's are still the same as before the process. Do I need to wait the hour for it to run the CRON automatically? If not I think the problem still exists.

BBcan177

You can goto the Log Browser tab and delete this feed in the "Deny" folder.
or from the shell:

rm /var/db/pfblockerng/deny/DenyIPs.*

Then Force Update to get the changes immediately… But after that... it will update the file as per the frequency setting as designed...

lispeedyg

Thank you.
I'll give that a try and post back if there are other issues.
Thanks again.