PfBlocker - Block Hit Counter
-
pfBlocker - Block Hit Counter and SRI BOThunter Lookup Tool.
The following script reads the /var/log/filter.log file to count the number of blocks for each interface and report back the count and also to complete a SRI BotHunter lookup.
The script can be run from the EMAIL Reports Manager so you could get a Daily report or it can be manually run from the SSH Shell.
Here is what the output looks like -
[ Each Section contains ]
* It will report the Blocks from the highest to lowest count found.
* Lists all pfBlocker Lists that contain the Blocked IP Address. (It searches by the first 3 octets of the IP as the lists may contain CIDR or IP Ranges)
* The EVENT Detail [ SRC IP, src port, DST IP and dst port ]
* Total number of Blocked events
* SRI BOTHuner lookup of the addressThis will repeat for each blocked address seperated by Interface that are found.
If the address is not found; IP was blocked by a FW rule, or it was removed by an update to the pfBlocker list.[EXAMPLE]
========================================================== pfBlockerAtlas.txt:176.32.101.4/32 pfBlockerOther.txt:176.32.101.4/32 pfBlockerOther.txt:176.32.101.129/32 pfBlockerOther.txt:176.32.101.68/32 pfBlockerOther.txt:176.32.101.137/32 pfBlockerOther.txt:176.32.101.81/32 EVENT [ 10.x.x.x.43595 > 176.32.101.137.80 ] Blocked ip [ 176.32.101.137 ] on bge0, found [ 1 ] times --------------------------------------------------------- IP Address = 176.32.101.137 Threat Level = High Threat Category = Malware Propagator Threat Description = Malware drive-by exploit site Hostname = Service Provider = AMAZON DATA SERVICES IRELAND LTD Domain Name = AMAZON.COM ASN Number = ASN Name = Network Speed = COMP Country CC = IE Country = IRELAND Region = DUBLIN CITY City = DUBLIN Longitude = -6.26718997955322 Latitude = 53.3439903259277 ========================================================== pfBlockerAlienvault.txt:87.106.52.0/24 pfBlockerET.txt:87.106.52.92/32 pfBlockerOther.txt:87.106.52.92/32 pfBlockerSFS.txt:87.106.52.0/24 EVENT [ 87.106.52.183.58757 > 10.x.x.x.25 ] Blocked ip [ 87.106.52.183 ] on bce0, found [ 4 ] times --------------------------------------------------------- IP Address = 87.106.52.183 Threat Level = Unverified Threat Category = Threat Description = Hostname = Service Provider = 1&1 INTERNET AG Domain Name = 1AND1.CO.UK ASN Number = ASN Name = Network Speed = COMP Country CC = DE Country = GERMANY Region = BADEN-WURTTEMBERG City = KARLSRUHE Longitude = 8.38582992553711 Latitude = 49.0047187805176 ========================================================== pfBlockerAlienvault.txt:62.112.100.0/24 pfBlockerdBlock.txt:62.112.100.0/24 pfBlockerinfiltrated.txt:62.112.100.44/32 EVENT [ 62.112.100.44.60635 > 10.x.x.x.25 ] Blocked ip [ 62.112.100.44 ] on bce0, found [ 4 ] times --------------------------------------------------------- IP Address = 62.112.100.44 Threat Level = Unverified Threat Category = Malware Propagator Threat Description = Malware drive-by exploit site Hostname = spd44.rdc.ru Service Provider = IM Domain Name = - ASN Number = 25513 ASN Name = ASN-MGTS-USPD OJS Moscow city telephone network Network Speed = COMP Country CC = RU Country = RUSSIAN FEDERATION Region = - City = - Longitude = 37.5830001831055 Latitude = 55.75 ==========================================================NOTE
[1]
To use the script, the Firewall logs need to be formated to one line entries with a patch.
Jimp has provided a patch to allow pfSense to send its syslogs to a remote Syslog software.
This functionality allows the script to lookup the required data for each Block.From pfSense add "System Patches" from the "Avalable Packages" repository.
In the System:Patches menu, select "+" and add a new patchIf you're on 2.1, add this patch:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diffIf you're on 2.0.x, use this patch instead:
http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.0.x.diffOnce you have entered the patch details, you need to "Fetch" and than "Apply"
(HELP LINK) https://doc.pfsense.org/index.php/System_Patches
Finally. check the box on the system log settings to force the firewall logs to one line.
The script will not report any Blocked Addresses prior to this patch being made active.[2]
The easiest way to use the Script is to Install the pfSense package "Filer" using the Package Manager.
This way you can add the script to pfSense from the WEB GUI and will allow the script to be saved in the
pfsense .conf file.In "Filer", add a new entry, [File] enter the path /home/user/pfcount (Change "user" to your user folder)
[Desc] pfCount
[Perm] 755Paste the SCRIPT below into the "File Contents Box"
Select "Background and Save"[3]
Use the package "MAIL Report" to allow running the script and sending the data via email.
Add a new Report, Save, reopen the report and in the "Report Commands" add the path to the script as above.
Save, select "Send Now" to check if the script is functioning properly. You will get an email.
Please ensure you add your email credentials in Advanced:Notification on the Web GUI for the email function to work.Make sure the report is run at the end of the day as it will only report the current days events. This can be changed by removing lines with
grep "$event" so that it will report all IP addresses that are blocked in the filter.log[4]
The script can use a "Whitelist" file, so you can exclude IP address(es).
You will need to change this line in the Script, to use your local user folder
whitelist=/home/user/whitelist
(The whitelist file can also be edited in the "Filer" Program.
Here is the script
#!/bin/sh banner=" ==========================================================" event=$(date +%b" "%d) log=/var/log/filter.log lists=/var/db/aliastables/* whitelist=/home/user/whitelist int=$(grep -ao 'block in on.*: (' $log | sed 's/block in on//' | sed 's/: (//' | sort | uniq) block() { blockip=$(grep -a "block in on $1" $log | grep "$event" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" | sed -r 's/(10)(\.([2][0-5][0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){3}'/p.p.p.p/g | sed -r 's/(172)\.(1[6-9]|2[0-9]|3[0-1])(\.([2][0-5][0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){2}'/p.p.p.p/g | sed -r 's/(192)\.(168)(\.([2][0-5][0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){2}'/p.p.p.p/g | sed -r 's/224\.0\.0\..*'/p.p.p.p/g | sed -r 's/239\.255\.255\..*'/p.p.p.p/g | grep -v "p.p.p.p" | grep -vxf $whitelist | sort -nr | uniq -c | sort -nr | cut -c 6-) } for a in $int; do block "$a" echo "$banner" echo " The Following Address(es) were Blocked on Interface $a on $event" echo "$banner" echo echo -e "$blockip\n" echo for i in $blockip; do ii=$(echo $i | cut -d"." -f1-3) blist=$(grep ^$ii $lists | sed 's/\/var\/db\/aliastables\// /g') if [ -z "$blist" ]; then blist=" Address not found in any current lists"; fi dlist=$(grep -a $i $log | grep "$event" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}.*:" | sed 's/\://g' | sort | uniq -f 5) count=$(grep -a "block in on $a" $log | grep "$event" | grep -o "$i" | wc -l | sed -e 's/ *//') echo echo "$banner" echo -e "$blist\n" echo -e " EVENT [ $dlist ]\n" echo " Blocked IP [ $i ] on $a, found [ $count ] times" wget -qO- "http://kb.bothunter.net/ipInfo/nowait.php?IP=$i" | tail +3 | head -17 | sed -r 's/<\/{0,1}b>//g' echo done done -
Hi, is this still working? Tried it with 2.1.2 but no luck, even the patch from http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff applies without a error.
-
Hi, is this still working? Tried it with 2.1.2 but no luck, even the patch from http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff applies without a error.
Hi Bismarck,
The 2.1.1 patch is working for me on 2.1.2
What issue are you having, can you be more specific?
Thanks.
-
Thank you for the script!
I don;t think it is working for me, though. I applied the 2.1.1 patch on 2.1.3, enabled one-line logging in log settings, and touched the whitelist file. I did that about 3 hours ago. Still, all I get form the script is this:
==========================================================
The Following Address(es) were Blocked on Interface re0 on May 05==========================================================
The Following Address(es) were Blocked on Interface re1 on May 05I am running 2.1.3 x64 and I have a TON of block lists in PFBlocker.
Can you please assist in troubleshooting?
-
Hi Pistolero,
I think I see the issue, for some reason the logs have changed in pfSense or I didn't test it on a singular date. The Date function used a Zero filled value when it needs a single digit variable. Need to change the "d" date variable to an "e" variable.
Change this line
event=$(date +%b" "%d)
to
event=$(date +%b" "%e)
Save and try it again.
-
@BBcan17:
Hi Pistolero,
I think I see the issue, for some reason the logs have changed in pfSense or I didn't test it on a singular date. The Date function used a Zero filled value when it needs a single digit variable. Need to change the "d" date variable to an "e" variable.
Change this line
event=$(date +%b" "%d)
to
event=$(date +%b" "%e)
Save and try it again.
Hi BBcan17, I have the same problem as Pistolero which is no blocks shown. And where do we need to change those line? I cant fin that in pf-log-oneline-option-2.1.1.diff ???
-
Hi Bismarck,
You need to change that in the pfcount script. It's near the top of the script. Line 4.
If you followed the original instructions, you can edit it in the filer package.
-
Works great now! Thanky you!
Command output: pfBlocker Hit Counter (/stuff/pfcount)
==========================================================
The Following Address(es) were Blocked on Interface bce0 on May 661.174.51.200
213.113.206.131==========================================================
pfBlockerEmerging_OPFAS.txt:61.174.51.194/32
pfBlockerEmerging_OPFAS.txt:61.174.51.195/32
pfBlockerEmerging_OPFAS.txt:61.174.51.196/32
pfBlockerEmerging_OPFAS.txt:61.174.51.197/32
pfBlockerEmerging_OPFAS.txt:61.174.51.198/32
pfBlockerEmerging_OPFAS.txt:61.174.51.199/32
pfBlockerEmerging_OPFAS.txt:61.174.51.200/32
pfBlockerEmerging_OPFAS.txt:61.174.51.201/32
pfBlockerEmerging_OPFAS.txt:61.174.51.202/32
pfBlockerEmerging_OPFAS.txt:61.174.51.203/32
pfBlockerEmerging_OPFAS.txt:61.174.51.204/32
pfBlockerEmerging_OPFAS.txt:61.174.51.205/32
pfBlockerEmerging_OPFAS.txt:61.174.51.207/32
pfBlockerEmerging_OPFAS.txt:61.174.51.208/32
pfBlockerEmerging_OPFAS.txt:61.174.51.209/32
pfBlockerEmerging_OPFAS.txt:61.174.51.210/32
pfBlockerEmerging_OPFAS.txt:61.174.51.211/32
pfBlockerEmerging_OPFAS.txt:61.174.51.212/32
pfBlockerEmerging_OPFAS.txt:61.174.51.213/32
pfBlockerEmerging_OPFAS.txt:61.174.51.214/32
pfBlockerEmerging_OPFAS.txt:61.174.51.215/32
pfBlockerEmerging_OPFAS.txt:61.174.51.216/32
pfBlockerEmerging_OPFAS.txt:61.174.51.217/32
pfBlockerEmerging_OPFAS.txt:61.174.51.218/32
pfBlockerEmerging_OPFAS.txt:61.174.51.219/32
pfBlockerEmerging_OPFAS.txt:61.174.51.220/32
pfBlockerEmerging_OPFAS.txt:61.174.51.221/32
pfBlockerEmerging_OPFAS.txt:61.174.51.222/32
pfBlockerEmerging_OPFAS.txt:61.174.51.223/32
pfBlockerEmerging_OPFAS.txt:61.174.51.224/32
pfBlockerEmerging_OPFAS.txt:61.174.51.225/32
pfBlockerEmerging_OPFAS.txt:61.174.51.226/32
pfBlockerEmerging_OPFAS.txt:61.174.51.227/32
pfBlockerEmerging_OPFAS.txt:61.174.51.228/32
pfBlockerEmerging_OPFAS.txt:61.174.51.229/32
pfBlockerEmerging_OPFAS.txt:61.174.51.230/32
pfBlockerEmerging_OPFAS.txt:61.174.51.232/32
pfBlockerEmerging_OPFAS.txt:61.174.51.233/32
pfBlockerEmerging_OPFAS.txt:61.174.51.234/32
pfBlockerEmerging_OPFAS.txt:61.174.51.235/32EVENT [ 61.174.51.200.6000 > 213.252.49.13.22 ]
Blocked IP [ 61.174.51.200 ] on bce0, found [ 1 ] times
–-------------------------------------------------------
IP Address = 61.174.51.200
Threat Level = High
Threat Category = Malware Propagator
Threat Description = Malware scan and infect source
Hostname =
Service Provider = CHINANET-ZJ HUZHOU NODE NETWORK
Domain Name = CHINATELECOM.COM.CN
ASN Number =
ASN Name =
Network Speed = DSL
Country CC = CN
Country = CHINA
Region = ZHEJIANG
City = HANGZHOU
Longitude = 120.161422729492
Latitude = 30.2936496734619==========================================================
Address not found in any current listsEVENT [ 213.113.206.131.1248 > 213.252.49.13.23 ]
Blocked IP [ 213.113.206.131 ] on bce0, found [ 1 ] times
–-------------------------------------------------------
IP Address = 213.113.206.131
Threat Level = Unverified
Threat Category =
Threat Description =
Hostname =
Service Provider = B2 BREDBAND AB
Domain Name = BREDBAND.COM
ASN Number =
ASN Name =
Network Speed = DSL
Country CC = SE
Country = SWEDEN
Region = VARMLANDS LAN
City = KARLSTAD
Longitude = 13.5035696029663
Latitude = 59.3792991638184==========================================================
The Following Address(es) were Blocked on Interface bce1 on May 6==========================================================
The Following Address(es) were Blocked on Interface bce2 on May 6But it would be nice to have just the IPs listed, which are blocked and not all in that segment, see above.
Anyway thank you very much. :)
-
Damn, you're good! That did the trick! Thank you!
@BBcan17:
Hi Pistolero,
I think I see the issue, for some reason the logs have changed in pfSense or I didn't test it on a singular date. The Date function used a Zero filled value when it needs a single digit variable. Need to change the "d" date variable to an "e" variable.
Change this line
event=$(date +%b" "%d)
to
event=$(date +%b" "%e)
Save and try it again.
-
Hi BBcan,
I LOVE YOUR SCRIPT!!!
Can I humbly request the addition of an option to enable name resolution for the IPs listed in the report? It would help me greatly to troubleshoot a stupid app which will not work and I don't know what is blocking it.
Thank you sir!
-
Hi Pistolero,
Thanks. The BotHunter lookup should provide all of the Name Resolution. Typically if it doesn't report some of the details, then the IP is most likely malicious or the Bothunter doesnt have it in its database.
Can you post the IP and the Bothunter report for the one that's causing you an issue?
If you want a quick and dirty method to see what Blocklist has an certain IP, you can do the following command in an SSH shell or from the Diagnostics:Command Prompt.
grep "x.x.x.x" /var/db/aliastables/*
or just search with the (x.x.x.) First Three octets as the IP address you want to find as the address could be in a CIDR Range.
@Bismarck et al
You can modify the script with these lines so it will report less Lines in the output.
The first line has the ^$ii changed to ^$i and the next lines is new.
blist=$(grep ^$i $lists | sed 's//var/db/aliastables// /g')
if [ -z "$blist" ]; then blist=$(grep ^$ii $lists | sed 's//var/db/aliastables// /g'); fi -
Hi BBcan!
I am really missing your awesome script on 2.2! Any idea how to make it work?
Thanks!
-
Check out BBCans recent pfblockerNG package - I think that will do what you want.
-
Hi BBcan!
I am really missing your awesome script on 2.2! Any idea how to make it work?
Thanks!
Its been awhile since I looked at that… Are you using pfBlockerNG? Maybe that will provide all you need?
-
Hi BBcan!
I am really missing your awesome script on 2.2! Any idea how to make it work?
Thanks!
Its been awhile since I looked at that… Are you using pfBlockerNG? Maybe that will provide all you need?
I have tried pfBlockerNG, but I am currently unable to make it work (get the dreaded "-" in the packet list, and multiple errors downloading the lists. If time permits I'll make a post on the NG thread about my woes with it…
Thank you, sir!
-
I've installed then reinstalled pfBlockerNG so many times it would make you laugh. What are the errors your getting and what do the logs say? I started with BB's script, this is it, only better. Yell out and I can quickly get you up an running.
Hi BBcan!
I am really missing your awesome script on 2.2! Any idea how to make it work?
Thanks!
Its been awhile since I looked at that… Are you using pfBlockerNG? Maybe that will provide all you need?
I have tried pfBlockerNG, but I am currently unable to make it work (get the dreaded "-" in the packet list, and multiple errors downloading the lists. If time permits I'll make a post on the NG thread about my woes with it…
Thank you, sir!
-
hi wcrowder!
I really appreciate your offer to assist. I am overseas (In Medellin, Colombia, actually), and will be back home on March 5th. BBCan, as awesome as he is, also offered to help (you guys rock! thank you!).
If you can, It'd be awesome to have a TeamViewer or WebEx session on or after march 5th. If you are unable to, I'll gather logs and shoot 'em over your way. I'm sure it's not working due to something stupid I did :P
Again, thank you and have a great weekend, sir!
