PfBlocker - Block Hit Counter

[BBcan177]

pfBlocker - Block Hit Counter and SRI BOThunter Lookup Tool.

The following script reads the /var/log/filter.log file to count the number of blocks for each interface and report back the count and also to complete a SRI BotHunter lookup.

The script can be run from the EMAIL Reports Manager so you could get a Daily report or it can be manually run from the SSH Shell.

Here is what the output looks like -

[ Each Section contains ]

*  It will report the Blocks from the highest to lowest count found.
  *  Lists all pfBlocker Lists that contain the Blocked IP Address. (It searches by the first 3 octets of the IP as the lists may contain CIDR or IP Ranges)
  *  The EVENT Detail [ SRC IP, src port, DST IP and dst port ]
  *  Total number of Blocked events
  *  SRI BOTHuner lookup of the address

This will repeat for each blocked address seperated by Interface that are found.
If the address is not found; IP was blocked by a FW rule, or it was removed by an update to the pfBlocker list.

[EXAMPLE]

 ==========================================================
 pfBlockerAtlas.txt:176.32.101.4/32
 pfBlockerOther.txt:176.32.101.4/32
 pfBlockerOther.txt:176.32.101.129/32
 pfBlockerOther.txt:176.32.101.68/32
 pfBlockerOther.txt:176.32.101.137/32
 pfBlockerOther.txt:176.32.101.81/32

 EVENT [ 10.x.x.x.43595 > 176.32.101.137.80 ]

 Blocked ip [ 176.32.101.137 ] on bge0, found [ 1 ] times
 ---------------------------------------------------------
      IP Address         = 176.32.101.137
      Threat Level       = High
      Threat Category    = Malware Propagator
      Threat Description = Malware drive-by exploit site
      Hostname           =
      Service Provider   = AMAZON DATA SERVICES IRELAND LTD
      Domain Name        = AMAZON.COM
      ASN Number         =
      ASN Name           =
      Network Speed      = COMP
      Country CC         = IE
      Country            = IRELAND
      Region             = DUBLIN CITY
      City               = DUBLIN
      Longitude          = -6.26718997955322
      Latitude           = 53.3439903259277

 ==========================================================

 pfBlockerAlienvault.txt:87.106.52.0/24
 pfBlockerET.txt:87.106.52.92/32
 pfBlockerOther.txt:87.106.52.92/32
 pfBlockerSFS.txt:87.106.52.0/24

 EVENT [ 87.106.52.183.58757 > 10.x.x.x.25 ]

 Blocked ip [ 87.106.52.183 ] on bce0, found [ 4 ] times
 ---------------------------------------------------------
      IP Address         = 87.106.52.183
      Threat Level       = Unverified
      Threat Category    =
      Threat Description =
      Hostname           =
      Service Provider   = 1&1 INTERNET AG
      Domain Name        = 1AND1.CO.UK
      ASN Number         =
      ASN Name           =
      Network Speed      = COMP
      Country CC         = DE
      Country            = GERMANY
      Region             = BADEN-WURTTEMBERG
      City               = KARLSRUHE
      Longitude          = 8.38582992553711
      Latitude           = 49.0047187805176

 ========================================================== 
 pfBlockerAlienvault.txt:62.112.100.0/24 
 pfBlockerdBlock.txt:62.112.100.0/24 
 pfBlockerinfiltrated.txt:62.112.100.44/32  

 EVENT [ 62.112.100.44.60635 > 10.x.x.x.25 ]  

 Blocked ip [ 62.112.100.44 ] on bce0, found [ 4 ] times
 --------------------------------------------------------- 

 IP Address = 62.112.100.44 
 Threat Level = Unverified 
 Threat Category = Malware Propagator 
 Threat Description = Malware drive-by exploit site 
 Hostname = spd44.rdc.ru 
 Service Provider = IM 
 Domain Name = - 
 ASN Number = 25513 
 ASN Name = ASN-MGTS-USPD OJS Moscow city telephone network 
 Network Speed = COMP 
 Country CC = RU 
 Country = RUSSIAN FEDERATION 
 Region = - 
 City = - 
 Longitude = 37.5830001831055 
 Latitude = 55.75

 ==========================================================

NOTE

[1]

To use the script, the Firewall logs need to be formated to one line entries with a patch.

Jimp has provided a patch to allow pfSense to send its syslogs to a remote Syslog software.
  This functionality allows the script to lookup the required data for each Block.

From pfSense add "System Patches" from the "Avalable Packages" repository.
  In the System:Patches menu, select "+" and add a new patch

If you're on 2.1, add this patch:
  http://files.pfsense.org/jimp/patches/pf-log-oneline-option.diff

If you're on 2.0.x, use this patch instead:
  http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.0.x.diff

Once you have entered the patch details, you need to "Fetch" and than "Apply"

(HELP LINK) https://doc.pfsense.org/index.php/System_Patches

Finally. check the box on the system log settings to force the firewall logs to one line.
  The script will not report any Blocked Addresses prior to this patch being made active.

[2]

The easiest way to use the Script is to Install the pfSense package "Filer" using the Package Manager.
This way you can add the script to pfSense from the WEB GUI and will allow the script to be saved in the
pfsense .conf file.

In "Filer", add a new entry, [File] enter the path    /home/user/pfcount  (Change "user" to your user folder)
                                        [Desc] pfCount
                                        [Perm] 755

Paste the SCRIPT below into the "File Contents Box"
                                        Select "Background and Save"

[3]

Use the package "MAIL Report" to allow running the script and sending the data via email.
Add a new Report, Save, reopen the report and in the "Report Commands" add the path to the script as above.
Save, select "Send Now" to check if the script is functioning properly. You will get an email.
Please ensure you add your email credentials in Advanced:Notification on the Web GUI for the email function to work.

Make sure the report is run at the end of the day as it will only report the current days events. This can be changed by removing lines with
grep "$event" so that it will report all IP addresses that are blocked in the filter.log

[4]

The script can use a "Whitelist" file, so you can exclude IP address(es).

You will need to change this line in the Script, to use your local user folder

whitelist=/home/user/whitelist

(The whitelist file can also be edited in the "Filer" Program.

Here is the script

#!/bin/sh

banner=" =========================================================="
event=$(date +%b" "%d)

log=/var/log/filter.log
lists=/var/db/aliastables/*
whitelist=/home/user/whitelist

int=$(grep -ao 'block in on.*: (' $log | sed 's/block in on//' | sed 's/: (//' | sort | uniq)

block()
{
blockip=$(grep -a "block in on $1" $log | grep "$event" |
        grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}" |
        sed -r 's/(10)(\.([2][0-5][0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){3}'/p.p.p.p/g |
        sed -r 's/(172)\.(1[6-9]|2[0-9]|3[0-1])(\.([2][0-5][0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){2}'/p.p.p.p/g |
        sed -r 's/(192)\.(168)(\.([2][0-5][0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){2}'/p.p.p.p/g |
        sed -r 's/224\.0\.0\..*'/p.p.p.p/g |
        sed -r 's/239\.255\.255\..*'/p.p.p.p/g |
        grep -v "p.p.p.p" |
        grep -vxf $whitelist |
        sort -nr |
        uniq -c |
        sort -nr |
        cut -c 6-)
}

for a in $int; do
        block "$a"
        echo "$banner"
        echo " The Following Address(es) were Blocked on Interface $a on $event"
        echo "$banner"
        echo
        echo -e "$blockip\n"
        echo

        for i in $blockip; do
                ii=$(echo $i | cut -d"." -f1-3)
                blist=$(grep ^$ii $lists | sed 's/\/var\/db\/aliastables\// /g')
                if [ -z "$blist" ]; then blist=" Address not found in any current lists"; fi
                dlist=$(grep -a $i $log | grep "$event" | grep -oE "[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}.*:" | sed 's/\://g' | sort | uniq -f 5)
                count=$(grep -a "block in on $a" $log | grep "$event" | grep -o "$i" | wc -l | sed -e 's/ *//')
                echo
                echo "$banner"
                echo -e "$blist\n"
                echo -e " EVENT [ $dlist ]\n"
                echo " Blocked IP [ $i ] on $a, found [ $count ] times"
                wget -qO- "http://kb.bothunter.net/ipInfo/nowait.php?IP=$i" | tail +3 | head -17 | sed -r 's/<\/{0,1}b>//g'
                echo
        done
done

[Bismarck]

Hi, is this still working? Tried it with 2.1.2 but no luck, even the patch from http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff applies without a error.

[BBcan177]

@Bismarck:

Hi, is this still working? Tried it with 2.1.2 but no luck, even the patch from http://files.pfsense.org/jimp/patches/pf-log-oneline-option-2.1.1.diff applies without a error.

Hi Bismarck,

The 2.1.1 patch is working for me on 2.1.2

What issue are you having, can you be more specific?

Thanks.

[Pistolero]

Thank you for the script!

I don;t think it is working for me, though. I applied the 2.1.1 patch on 2.1.3, enabled one-line logging in log settings, and touched the whitelist file. I did that about 3 hours ago. Still, all I get form the script is this:

==========================================================
The Following Address(es) were Blocked on Interface re0 on May 05

==========================================================
The Following Address(es) were Blocked on Interface re1 on May 05

I am running 2.1.3 x64 and I have a TON of block lists in PFBlocker.

Can you please assist in troubleshooting?

[BBcan177]

Hi Pistolero,

I think I see the issue, for some reason the logs have changed in pfSense or I didn't test it on a singular date. The Date function used a Zero filled value when it needs a single digit variable. Need to change the "d" date variable to an "e" variable.

Change this line

event=$(date +%b" "%d)

to

event=$(date +%b" "%e)

Save and try it again.

[Bismarck]

@BBcan17:

Hi Pistolero,

I think I see the issue, for some reason the logs have changed in pfSense or I didn't test it on a singular date. The Date function used a Zero filled value when it needs a single digit variable. Need to change the "d" date variable to an "e" variable.

Change this line

event=$(date +%b" "%d)

to

event=$(date +%b" "%e)

Save and try it again.

Hi BBcan17, I have the same problem as Pistolero which is no blocks shown. And where do we need to change those line? I cant fin that in pf-log-oneline-option-2.1.1.diff ???

[BBcan177]

Hi Bismarck,

You need to change that in the pfcount script. It's near the top of the script. Line 4.

If you followed the original instructions, you can edit it in the filer package.

[Bismarck]

Works great now! Thanky you!

Command output: pfBlocker Hit Counter (/stuff/pfcount)

==========================================================
The Following Address(es) were Blocked on Interface bce0 on May  6

61.174.51.200
213.113.206.131

==========================================================
pfBlockerEmerging_OPFAS.txt:61.174.51.194/32
pfBlockerEmerging_OPFAS.txt:61.174.51.195/32
pfBlockerEmerging_OPFAS.txt:61.174.51.196/32
pfBlockerEmerging_OPFAS.txt:61.174.51.197/32
pfBlockerEmerging_OPFAS.txt:61.174.51.198/32
pfBlockerEmerging_OPFAS.txt:61.174.51.199/32
pfBlockerEmerging_OPFAS.txt:61.174.51.200/32
pfBlockerEmerging_OPFAS.txt:61.174.51.201/32
pfBlockerEmerging_OPFAS.txt:61.174.51.202/32
pfBlockerEmerging_OPFAS.txt:61.174.51.203/32
pfBlockerEmerging_OPFAS.txt:61.174.51.204/32
pfBlockerEmerging_OPFAS.txt:61.174.51.205/32
pfBlockerEmerging_OPFAS.txt:61.174.51.207/32
pfBlockerEmerging_OPFAS.txt:61.174.51.208/32
pfBlockerEmerging_OPFAS.txt:61.174.51.209/32
pfBlockerEmerging_OPFAS.txt:61.174.51.210/32
pfBlockerEmerging_OPFAS.txt:61.174.51.211/32
pfBlockerEmerging_OPFAS.txt:61.174.51.212/32
pfBlockerEmerging_OPFAS.txt:61.174.51.213/32
pfBlockerEmerging_OPFAS.txt:61.174.51.214/32
pfBlockerEmerging_OPFAS.txt:61.174.51.215/32
pfBlockerEmerging_OPFAS.txt:61.174.51.216/32
pfBlockerEmerging_OPFAS.txt:61.174.51.217/32
pfBlockerEmerging_OPFAS.txt:61.174.51.218/32
pfBlockerEmerging_OPFAS.txt:61.174.51.219/32
pfBlockerEmerging_OPFAS.txt:61.174.51.220/32
pfBlockerEmerging_OPFAS.txt:61.174.51.221/32
pfBlockerEmerging_OPFAS.txt:61.174.51.222/32
pfBlockerEmerging_OPFAS.txt:61.174.51.223/32
pfBlockerEmerging_OPFAS.txt:61.174.51.224/32
pfBlockerEmerging_OPFAS.txt:61.174.51.225/32
pfBlockerEmerging_OPFAS.txt:61.174.51.226/32
pfBlockerEmerging_OPFAS.txt:61.174.51.227/32
pfBlockerEmerging_OPFAS.txt:61.174.51.228/32
pfBlockerEmerging_OPFAS.txt:61.174.51.229/32
pfBlockerEmerging_OPFAS.txt:61.174.51.230/32
pfBlockerEmerging_OPFAS.txt:61.174.51.232/32
pfBlockerEmerging_OPFAS.txt:61.174.51.233/32
pfBlockerEmerging_OPFAS.txt:61.174.51.234/32
pfBlockerEmerging_OPFAS.txt:61.174.51.235/32

EVENT [ 61.174.51.200.6000 > 213.252.49.13.22 ]

Blocked IP [ 61.174.51.200 ] on bce0, found [ 1 ] times
–-------------------------------------------------------
      IP Address        = 61.174.51.200
      Threat Level      = High
      Threat Category    = Malware Propagator
      Threat Description = Malware scan and infect source
      Hostname          =
      Service Provider  = CHINANET-ZJ HUZHOU NODE NETWORK
      Domain Name        = CHINATELECOM.COM.CN
      ASN Number        =
      ASN Name          =
      Network Speed      = DSL
      Country CC        = CN
      Country            = CHINA
      Region            = ZHEJIANG
      City              = HANGZHOU
      Longitude          = 120.161422729492
      Latitude          = 30.2936496734619

==========================================================
Address not found in any current lists

EVENT [ 213.113.206.131.1248 > 213.252.49.13.23 ]

Blocked IP [ 213.113.206.131 ] on bce0, found [ 1 ] times
–-------------------------------------------------------
      IP Address        = 213.113.206.131
      Threat Level      = Unverified
      Threat Category    =
      Threat Description =
      Hostname          =
      Service Provider  = B2 BREDBAND AB
      Domain Name        = BREDBAND.COM
      ASN Number        =
      ASN Name          =
      Network Speed      = DSL
      Country CC        = SE
      Country            = SWEDEN
      Region            = VARMLANDS LAN
      City              = KARLSTAD
      Longitude          = 13.5035696029663
      Latitude          = 59.3792991638184

==========================================================
The Following Address(es) were Blocked on Interface bce1 on May  6

==========================================================
The Following Address(es) were Blocked on Interface bce2 on May  6

But it would be nice to have just the IPs listed, which are blocked and not all in that segment, see above.

Anyway thank you very much. :)

[Pistolero]

Damn, you're good! That did the trick! Thank you!

@BBcan17:

Hi Pistolero,

I think I see the issue, for some reason the logs have changed in pfSense or I didn't test it on a singular date. The Date function used a Zero filled value when it needs a single digit variable. Need to change the "d" date variable to an "e" variable.

Change this line

event=$(date +%b" "%d)

to

event=$(date +%b" "%e)

Save and try it again.

[Pistolero]

Hi BBcan,

I LOVE YOUR SCRIPT!!!

Can I humbly request the addition of an option to enable name resolution for the IPs listed in the report? It would help me greatly to troubleshoot a stupid app which will not work and I don't know what is blocking it.

Thank you sir!

[BBcan177]

Hi Pistolero,

Thanks. The BotHunter lookup should provide all of the Name Resolution. Typically if it doesn't report some of the details, then the IP is most likely malicious or the Bothunter doesnt have it in its database.

Can you post the IP and the Bothunter report for the one that's causing you an issue?

If you want a quick and dirty method to see what Blocklist has an certain IP, you can do the following command in an SSH shell or from the Diagnostics:Command Prompt.

grep "x.x.x.x" /var/db/aliastables/*

or just search with the (x.x.x.) First Three octets as the IP address you want to find as the address could be in a CIDR Range.

@Bismarck et al

You can modify the script with these lines so it will report less Lines in the output.

The first line has the ^$ii changed to ^$i and the next lines is new.

blist=$(grep ^$i $lists | sed 's//var/db/aliastables// /g')
                if [ -z "$blist" ]; then blist=$(grep ^$ii $lists | sed 's//var/db/aliastables// /g'); fi

[Pistolero]

Hi BBcan!

I am really missing your awesome script on 2.2! Any idea how to make it work?

Thanks!

[q54e3w]

Check out BBCans recent pfblockerNG package - I think that will do what you want.

[BBcan177]

@Pistolero:

Hi BBcan!

I am really missing your awesome script on 2.2! Any idea how to make it work?

Thanks!

Its been awhile since I looked at that… Are you using pfBlockerNG? Maybe that will provide all you need?

[Pistolero]

@BBcan177:

@Pistolero:

Hi BBcan!

I am really missing your awesome script on 2.2! Any idea how to make it work?

Thanks!

Its been awhile since I looked at that… Are you using pfBlockerNG? Maybe that will provide all you need?

I have tried pfBlockerNG, but I am currently unable to make it work (get the dreaded "-" in the packet list, and multiple errors downloading the lists. If time permits I'll make a post on the NG thread about my woes with it…

Thank you, sir!

[wcrowder]

I've installed then reinstalled pfBlockerNG so many times it would make you laugh. What are the errors your getting and what do the logs say? I started with BB's script, this is it, only better. Yell out and I can quickly get you up an running.

@Pistolero:

@BBcan177:

@Pistolero:

Hi BBcan!

I am really missing your awesome script on 2.2! Any idea how to make it work?

Thanks!

Its been awhile since I looked at that… Are you using pfBlockerNG? Maybe that will provide all you need?

I have tried pfBlockerNG, but I am currently unable to make it work (get the dreaded "-" in the packet list, and multiple errors downloading the lists. If time permits I'll make a post on the NG thread about my woes with it…

Thank you, sir!

[Pistolero]

hi wcrowder!

I really appreciate your offer to assist. I am overseas (In Medellin, Colombia, actually), and will be back home on March 5th. BBCan, as awesome as he is, also offered to help (you guys rock! thank you!).

If you can, It'd be awesome to have a TeamViewer or WebEx session on or after march 5th. If you are unable to, I'll gather logs and shoot 'em over your way. I'm sure it's not working due to something stupid I did :P

Again, thank you and have a great weekend, sir!