PfBlocker list import problem ….

[atrocity]

Hello,

we use pfBlocker since months, we use it since the beta versions.
Now, we have 2 x PFSense : 2.3.2-RELEASE-p1 (amd64)
and pfBlockerNG version : 2.1.1_4

We have about 10 lists in our ip4 lists, and don't see the same number of IPs listed for the same exact lists …
But our main problem is that pfBlockerNG don't import 2 of our lists correctly :(
Her a exemple of a forced update status :

UPDATE PROCESS START [ 11/24/16 12:17:18 ]

Clearing all DNSBL Feeds…
** DNSBL Disabled **

===[  Continent Process  ]============================================

….

===[  IPv4 Process  ]=================================================

….

[ CSF ] Reload [ 11/24/16 12:17:26 ] . completed ..

[ CSF2 ] Reload . completed ..

===[  IPv6 Process  ]=================================================

===[  Aliastables / Rules  ]==========================================

No changes to Firewall rules, skipping Filter Reload

Updating: pfB_CSF
no changes.
Updating: pfB_CSF2
no changes.

===[ FINAL Processing ]=====================================

[ Original IP count  ]  [ 99559 ]

===[ Permit List IP Counts ]=========================

…..

===[ Deny List IP Counts ]===========================

57071 total
….
      33 /var/db/pfblockerng/deny/CSF.txt
      28 /var/db/pfblockerng/deny/CSF2.txt

====================[ Last Updated List Summary ]==============

….
Nov 24 12:00 CSF
Nov 24 12:15 CSF2
....

IPv4 alias tables IP count

94596

IPv6 alias tables IP count

4940

Alias table IP Counts

99536 total
....
      33 /var/db/aliastables/pfB_CSF.txt
      28 /var/db/aliastables/pfB_CSF2.txt

pfSense Table Stats

table-entries hard limit  1000000
Table Usage Count        184007

UPDATE PROCESS ENDED [ 11/24/16 12:17:30 ]

I removed the lines that don't concern the others lists, i leave only the CSF (ConfigServer firewall deny list, containing a simple list of IPs like :
61.12.1.200
115.248.248.85
195.134.76.233

There is no html formating and the IPs are one on each line.
As you can see, it import 2 lists. The 2 lists are the SAME EXACT list of IPs.
The fist one "CSF" show the it imported 33 IPs
The second one show it imported …. 28 IPs
:o :o :o

So i counted the 2 IPs lists, and on both i can only count ... 28 IPs ...
Then i removed some (the half) of this IPs, and the forced import show me the same exact result as on the top of this tread ....

33 /var/db/pfblockerng/deny/CSF.txt
      28 /var/db/pfblockerng/deny/CSF2.txt

Well, well, well ....
I try many things ....
Deleted the content of the whole directory /var/db/pfblockerng/*
and did a try to import with force and without force. Same result !  :-\

Then i did a try to remove pfBlockerNG, reboot the firewall, install pfBlockerNG, redefine the ip4 lists.

A normal import (without force) work !  8)

Then, cron should run the import after 1 hour. This is not working anymore, because he don't import the current and actualized lists again  :'(

I see all this import problems with 2 PFSense firewalls, the first one is a  Super Micro C2758, 8Gb ram, ssd, ....
the second one a old Watchguard
Everything works great on both, but not pfBlockerNG

PS: all our other lists are also not actualized correctly. For the sames lists, i can't see the same number of listed IPs, also if i start a forced update at the same exact moment on both firewalls.

What can be that wrong with this list import ?
Thanks for any idea.
Best regards
Atrocity  8)

[atrocity]

forgot to say that on the CSF list server, i SEE the Apache access ass successfull, and the lists are reachable in our whole network.
It can import it correctly 1 time, can't imagine that he can't do it later …

[BBcan177]

Is deduplication enabled? I would assume its due to that…

[atrocity]

Hello BBcan and thanks for your answer.
I checked de-duplication and here are the result of the 2 :

26 /var/db/aliastables/pfB_CSF.txt
      1 /var/db/aliastables/pfB_CSF2.txt

33 /var/db/aliastables/pfB_CSF.txt

So also with de-duplication, the numbers of imported IPs are not the same.
Same thing for the other lists :

37525 /var/db/aliastables/pfB_Europe_v4.txt
  30344 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
    4940 /var/db/aliastables/pfB_Europe_v6.txt
    3027 /var/db/aliastables/pfB_Blocklistde_SSH.txt
    1342 /var/db/aliastables/pfB_Blocklistde_bots.txt
    816 /var/db/aliastables/pfB_Spamhaus.txt
    776 /var/db/aliastables/pfB_Rutgers_Attackers.txt
    500 /var/db/aliastables/pfB_Blocklistde_ApacheDDOS.txt
    409 /var/db/aliastables/pfB_Blocklistde_FTP21.txt
    271 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt

46065 /var/db/aliastables/pfB_Europe_v4.txt
  29329 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
    6569 /var/db/aliastables/pfB_Europe_v6.txt
    3369 /var/db/aliastables/pfB_Blocklistde_SSH.txt
    1139 /var/db/aliastables/pfB_Blocklistde_bots.txt
    816 /var/db/aliastables/pfB_Spamhaus.txt
    252 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt
    226 /var/db/aliastables/pfB_Rutgers_Attackers.txt

BBcan, i'm one of the guy that give you a access to our firewalls in Europe, some months ago during your beta ;-)
Thanks for any trick/idea to debug this.
Regards

[BBcan177]

Run a reload then execute the following command:

rm /var/db/pfblockerng/test1
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test1

Then after the next cron run:

rm /var/db/pfblockerng/test2
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test2

Then compare the differences:

awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

You could also send the comparison to a file for review:

awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

Then use a file editor like vi, ee, or nano to view the files
[ /var/db/pfblockerng/test1dif    and    /var/db/pfblockerng/test2dif  ]

You can then grep one of the missing IPs and see if the IP is now blocked by a larger CIDR entry:

grep "1\.2\.3\.4"  /var/db/pfblockerng/deny/*
grep "1\.2\.3\."  /var/db/pfblockerng/deny/*
grep "1\.2\."  /var/db/pfblockerng/deny/*
grep "1\."  /var/db/pfblockerng/deny/*

You can use this link to see what a CIDR range includes:
https://www.ultratools.com/tools/netMask

You can send the results via email to make it easier to reply. (See general tab for email)

Please do your own testing first please :)

[atrocity]

Hello,

well, here are the results on of of the 2 firewalls :

[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

nothing found :(

Should i try to delete all the pfblocker/* and redo this tests ?
Thanks
Best regards

[lispeedyg]

Hi,
Has this ever been resolved??
I've been having the same issues and the OP.  Currently on pfSense V 2.4.1 and pfBlockerNG v 2.1.2_1.
And, specifically, the only way I can actually have the list updated is to uninstall pfBlocker completely, i.e., Don't keep settings.  And then re-start the process completely.  All other facets work as intended.  But, this has been a significant drawback.

Is there any further help anyone can provide?

[BBcan177]

Post some more details about what you're seeing … Keep in mind that Feeds won't update if the source of the Feed is unchanged (ie: previously downloaded feed is the same as the source feed), and the pfblockerng.log will show "Update not required".

[lispeedyg]

Hi,
Thanks for the quick reply..
My feed is actually a text file I host on the pfSense box at /usr/local/www/DenyIPs.txt
I add to the file manually as I'm informed of possible new threats.
I've pointed to the file source in two seperate ways, i.e., using the direct path above and also as http://[server-ip]/DenyIPs.txt

In both scenarios I can see the new IP entries in the file/browser.  I then run both an Update and Reload for pfBlockerNG.  And in all cases (other than completely deleting the application) the result is "no changes".

[BBcan177]

You need to use this format:

/usr/local/www/DenyIPs.txt

And keep in mind that Force Update|Reload just downloads the feed if its new (IE: wasn't previously downloaded).

You need to define the Frequency setting to say "once per hour" and then it will download the referenced  Source Feed at that time via the Cron task.

[lispeedyg]

Thanks for that.
I've reverted to the original file pointer, i.e., /usr/local/www/DenyIPs.txt.
I've also revised the frequency to 1hour from weekly.
After the above, I manually ran Update/Cron/Reload.  And, the new IP's are still not visible and the number of IP's are still the same as before the process.  Do I need to wait the hour for it to run the CRON automatically?  If not I think the problem still exists.

[BBcan177]

You can goto the Log Browser tab and delete this feed in the "Deny" folder.
or from the shell:

rm /var/db/pfblockerng/deny/DenyIPs.*

Then Force Update to get the changes immediately… But after that... it will update the file as per the frequency setting as designed...

[lispeedyg]

Thank you.
I'll give that a try and post back if there are other issues.
Thanks again.