PfBlocker list import problem ….

[atrocity]

Hello BBcan and thanks for your answer.
I checked de-duplication and here are the result of the 2 :

26 /var/db/aliastables/pfB_CSF.txt
      1 /var/db/aliastables/pfB_CSF2.txt

33 /var/db/aliastables/pfB_CSF.txt

So also with de-duplication, the numbers of imported IPs are not the same.
Same thing for the other lists :

37525 /var/db/aliastables/pfB_Europe_v4.txt
  30344 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
    4940 /var/db/aliastables/pfB_Europe_v6.txt
    3027 /var/db/aliastables/pfB_Blocklistde_SSH.txt
    1342 /var/db/aliastables/pfB_Blocklistde_bots.txt
    816 /var/db/aliastables/pfB_Spamhaus.txt
    776 /var/db/aliastables/pfB_Rutgers_Attackers.txt
    500 /var/db/aliastables/pfB_Blocklistde_ApacheDDOS.txt
    409 /var/db/aliastables/pfB_Blocklistde_FTP21.txt
    271 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt

46065 /var/db/aliastables/pfB_Europe_v4.txt
  29329 /var/db/aliastables/pfB_Blocklistde_SMTP25.txt
    6569 /var/db/aliastables/pfB_Europe_v6.txt
    3369 /var/db/aliastables/pfB_Blocklistde_SSH.txt
    1139 /var/db/aliastables/pfB_Blocklistde_bots.txt
    816 /var/db/aliastables/pfB_Spamhaus.txt
    252 /var/db/aliastables/pfB_Blocklistde_bruteforce.txt
    226 /var/db/aliastables/pfB_Rutgers_Attackers.txt

BBcan, i'm one of the guy that give you a access to our firewalls in Europe, some months ago during your beta ;-)
Thanks for any trick/idea to debug this.
Regards

[BBcan177]

Run a reload then execute the following command:

rm /var/db/pfblockerng/test1
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test1

Then after the next cron run:

rm /var/db/pfblockerng/test2
cat /var/db/pfblockerng/deny/*.txt >> /var/db/pfblockerng/test2

Then compare the differences:

awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n

You could also send the comparison to a file for review:

awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

Then use a file editor like vi, ee, or nano to view the files
[ /var/db/pfblockerng/test1dif    and    /var/db/pfblockerng/test2dif  ]

You can then grep one of the missing IPs and see if the IP is now blocked by a larger CIDR entry:

grep "1\.2\.3\.4"  /var/db/pfblockerng/deny/*
grep "1\.2\.3\."  /var/db/pfblockerng/deny/*
grep "1\.2\."  /var/db/pfblockerng/deny/*
grep "1\."  /var/db/pfblockerng/deny/*

You can use this link to see what a CIDR range includes:
https://www.ultratools.com/tools/netMask

You can send the results via email to make it easier to reply. (See general tab for email)

Please do your own testing first please :)

[atrocity]

Hello,

well, here are the results on of of the 2 firewalls :

[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test2 /var/db/pfblockerng/test1 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test1dif
[2.3.2-RELEASE][root@MyFirewallName]/var/db/pfblockerng: awk 'FNR==NR{a[$0];next}!($0 in a)' /var/db/pfblockerng/test1 /var/db/pfblockerng/test2 | sort -t . -k 1,1n -k 2,2n -k 3,3n -k 4,4n > /var/db/pfblockerng/test2dif

nothing found :(

Should i try to delete all the pfblocker/* and redo this tests ?
Thanks
Best regards

[lispeedyg]

Hi,
Has this ever been resolved??
I've been having the same issues and the OP.  Currently on pfSense V 2.4.1 and pfBlockerNG v 2.1.2_1.
And, specifically, the only way I can actually have the list updated is to uninstall pfBlocker completely, i.e., Don't keep settings.  And then re-start the process completely.  All other facets work as intended.  But, this has been a significant drawback.

Is there any further help anyone can provide?

[BBcan177]

Post some more details about what you're seeing … Keep in mind that Feeds won't update if the source of the Feed is unchanged (ie: previously downloaded feed is the same as the source feed), and the pfblockerng.log will show "Update not required".

[lispeedyg]

Hi,
Thanks for the quick reply..
My feed is actually a text file I host on the pfSense box at /usr/local/www/DenyIPs.txt
I add to the file manually as I'm informed of possible new threats.
I've pointed to the file source in two seperate ways, i.e., using the direct path above and also as http://[server-ip]/DenyIPs.txt

In both scenarios I can see the new IP entries in the file/browser.  I then run both an Update and Reload for pfBlockerNG.  And in all cases (other than completely deleting the application) the result is "no changes".

[BBcan177]

You need to use this format:

/usr/local/www/DenyIPs.txt

And keep in mind that Force Update|Reload just downloads the feed if its new (IE: wasn't previously downloaded).

You need to define the Frequency setting to say "once per hour" and then it will download the referenced  Source Feed at that time via the Cron task.

[lispeedyg]

Thanks for that.
I've reverted to the original file pointer, i.e., /usr/local/www/DenyIPs.txt.
I've also revised the frequency to 1hour from weekly.
After the above, I manually ran Update/Cron/Reload.  And, the new IP's are still not visible and the number of IP's are still the same as before the process.  Do I need to wait the hour for it to run the CRON automatically?  If not I think the problem still exists.

[BBcan177]

You can goto the Log Browser tab and delete this feed in the "Deny" folder.
or from the shell:

rm /var/db/pfblockerng/deny/DenyIPs.*

Then Force Update to get the changes immediately… But after that... it will update the file as per the frequency setting as designed...

[lispeedyg]

Thank you.
I'll give that a try and post back if there are other issues.
Thanks again.