Firewall rules hitcount for pfSense 2.1.5 and 2.2.4
-
Hi all,
These days showing some pfsense features like rules flow, states, etc, I missed a field on rules gui showing rule hit count. These could help sysadmins to see what rules are "working", what rules has active states and what rules has a heavy traffic.
Why not include these feature that we see on almost all commercial firewalls to pfSense? 8)
Follow these steps via gui ( only one via console ) to get it working on pfSense.
-
Steps for pfSense 2.1.5
Step 1:
Install packages filer and System PatchesStep 2:
Create on filer package, the file /etc/inc/rule_count.inc with 0644 permission with following contents:function bd_nice_number($n) { // first strip any formatting; $n = (0+str_replace(",","",$n)); // is this a number? if(!is_numeric($n)) return false; // now filter it; if($n>1000000000000) return round(($n/1000000000000),1).'t'; else if($n>1000000000) return round(($n/1000000000),1).'g'; else if($n>1000000) return round(($n/1000000),1).'m'; else if($n>1000) return round(($n/1000),1).'k'; return number_format($n); } $rules_count_array=array(); exec("/sbin/pfctl -vvsr | /usr/bin/grep -A1 USER_RULE:",$cnt_pfctl); if (!empty($cnt_pfctl)) { foreach($cnt_pfctl as $line) { if (preg_match("/USER_RULE:\s+(\w+)/",$line,$m1)){ $cr=$m1[1]; continue; } if (preg_match("/Evaluations:\s+(\d+)\s+Packets:\s+(\d+)\s+Bytes:\s+(\d+)\s+States:\s+(\d+)\s+/",$line,$m2)){ if ($cr != "" && !isset($rules_count_array[$cr][Packets])){ $rules_count_array[$cr]=array('Evaluations'=>$m2[1],'Packets'=>bd_nice_number($m2[2]),'Bytes'=>bd_nice_number($m2[3]),'States'=>$m2[4]); } } } } ?>
Create on package filer, the file /root/create_rule_hashes.php with 0644 permission with the following contents
require_once("/etc/inc/util.inc"); require_once("/etc/inc/functions.inc"); require_once("/etc/inc/pkg-utils.inc"); require_once("/etc/inc/globals.inc"); global $config; //var_dump($config['filter']['rule'][0]); //var_dump($config['filter']); $new_rules=array(); $count=0; foreach($config['filter']['rule'] as $fr){ if ( !array_key_exists ( 'hash' , $fr )) { $fr['hash']= md5(rand(10000000,99999999)); $count++; } $new_rules[]=$fr; } if ($count > 0) { print "{$count} new hashes created.\n Please reload your rules.\n"; $config['filter']['rule']=$new_rules; write_config(); } ?>
Step 3:
exec the created file**/root/create_rule_hashes.php** on cosole/sshcp /conf/config.xml /root/config.bkp.xml && php /root/create_rule_hashes.php
Step 4:
Create, test and apply patches under next steps using system->patches.Description: filter_inc_patch
Patch Contents:--- filter.orig.inc 2015-08-11 16:40:29.000000000 +0000 +++ filter.inc 2015-08-11 16:40:33.000000000 +0000 @@ -1947,10 +1947,13 @@ $line = filter_generate_user_rule($rule); $ret['rule'] = $line; $ret['interface'] = $rule['interface']; - if($rule['descr'] != "" and $line != "") + if ($rule['hash'] != "" and $line != "") { + $ret['descr'] = "label \"" . fix_rule_label("USER_RULE: {$rule['hash']}") . "\""; + } elseif ($rule['descr'] != "" and $line != "") { $ret['descr'] = "label \"" . fix_rule_label("USER_RULE: {$rule['descr']}") . "\""; - else + } else { $ret['descr'] = "label \"USER_RULE\""; + } return $ret; }
Base Directory:/etc/inc/ <–-Don't forget this field!!!
Description:firewall_rules_patch
Patch Contents:--- firewall_rules.orig.php 2015-08-11 16:41:35.000000000 +0000 +++ firewall_rules.php 2015-08-11 16:47:46.000000000 +0000 @@ -45,6 +45,7 @@ require_once("functions.inc"); require_once("filter.inc"); require_once("shaper.inc"); +require_once("rule_count.inc"); $pgtitle = array(gettext("Firewall"),gettext("Rules")); $shortcut_section = "firewall"; @@ -352,7 +353,7 @@ - + "> pfSense_handle_custom_code("/usr/local/pkg/firewall_rules/pre_id_tablehead"); ?> @@ -744,9 +745,18 @@ $printicon = true; } } + if (isset($filterent['hash']) && is_array($rules_count_array) && array_key_exists($filterent['hash'],$rules_count_array)) { + $rules_count=$rules_count_array[$filterent['hash']]['Packets']."/".$rules_count_array[$filterent['hash']]['States']; + $rules_title=""; + foreach ($rules_count_array[$filterent['hash']] as $rck => $rcv) { + $rules_title.= "$rck: $rcv\n"; + } + } else { + $rules_count="0/0"; + } ?> - - + + pfSense_handle_custom_code("/usr/local/pkg/firewall_rules/pre_id_tr");
Base Directory:/usr/local/www/ <–-Don't forget this field!!!
Description:firewall_rules_edit_patch
Patch Contents:--- firewall_rules_edit.orig.php 2015-08-11 16:41:38.000000000 +0000 +++ firewall_rules_edit.php 2015-08-11 16:44:02.000000000 +0000 @@ -108,6 +108,10 @@ if ( isset($a_filter[$id]['updated']) && is_array($a_filter[$id]['updated']) ) $pconfig['updated'] = $a_filter[$id]['updated']; + if (isset($a_filter[$id]['hash']) && is_array($a_filter[$id]['hash'])) { + $pconfig['hash'] = $a_filter[$id]['hash']; + } + if (!isset($a_filter[$id]['type'])) $pconfig['type'] = "pass"; else @@ -730,6 +734,12 @@ if ( isset($a_filter[$id]['created']) && is_array($a_filter[$id]['created']) ) $filterent['created'] = $a_filter[$id]['created']; + if (isset($a_filter[$id]['hash']) && is_array($a_filter[$id]['hash'])) { + $filterent['hash'] = $a_filter[$id]['hash']; + } else { + $filterent['hash'] = md5(rand(10000000,99999999)); + } + $filterent['updated'] = make_config_revision_entry(); // Allow extending of the firewall edit page and include custom input validation @@ -1683,6 +1693,13 @@ + + + + + + +
Base Directory:/usr/local/www/ <–-Don't forget this field!!!
After creating/saving these patches, click test and if all goes fine, click apply.
Step 5:
Edit and save any rule to update pfctl and start getting rule count hit. -
Steps for pfSense 2.2.4 (updated 2015/08/21) v0.4
Step 1:
Install packages filer and System PatchesStep 2:
Create on filer package, the file /etc/inc/rule_count.inc with 0644 permission with following contents:$rules_count_array=array(); exec("/sbin/pfctl -vvsr | /usr/bin/grep -A1 ' label '",$cnt_pfctl); if (!empty($cnt_pfctl)) { foreach($cnt_pfctl as $line) { if (preg_match('/@(\d+)\W\d+.*label "(USER_RULE: |)(.*)"/',$line,$m1)){ $cr=$m1[3]; $rl=$m1[2].$m1[3]; $rid=$m1[1]; continue; } if ($cr != "" && preg_match("/Evaluations:\s+(\d+)\s+Packets:\s+(\d+)\s+Bytes:\s+(\d+)\s+States:\s+(\d+)\s+/",$line,$m2)){ if (isset($rules_count_array[$cr][Packets])){ $rules_count_array[$cr]['Packets'] += $m2[2]; $rules_count_array[$cr]['Evaluations'] += $m2[1]; $rules_count_array[$cr]['Bytes'] += $m2[3]; $rules_count_array[$cr]['States'] += $m2[4]; $rules_count_array[$cr]['RuleId'] .= "|$rid"; }else { $rules_count_array[$cr]=array('Evaluations'=>$m2[1],'Packets'=>$m2[2],'Bytes'=>$m2[3],'States'=>$m2[4],'Label'=>$rl,'RuleId'=>$rid); } } } } ?>
Step 3:
Create, test and apply patches under next steps using system->patches.Description: filter_inc_patch
Patch Contents:--- filter.orig.inc 2015-08-11 13:17:11.000000000 +0000 +++ filter.inc 2015-08-17 15:05:54.000000000 +0000 @@ -2168,10 +2168,13 @@ $line = filter_generate_user_rule($rule); $ret['rule'] = $line; $ret['interface'] = $rule['interface']; - if($rule['descr'] != "" and $line != "") + if ($rule['tracker'] != "" and $line != "") { + $ret['descr'] = "label \"" . fix_rule_label("USER_RULE: {$rule['tracker']}") . "\""; + } elseif ($rule['descr'] != "" and $line != "") { $ret['descr'] = "label \"" . fix_rule_label("USER_RULE: {$rule['descr']}") . "\""; - else + } else { $ret['descr'] = "label \"USER_RULE\""; + } return $ret; }
Base Directory:/etc/inc/
Description:firewall_rules_patch
Patch Contents:--- firewall_rules.orig.php 2015-08-11 13:27:15.000000000 +0000 +++ firewall_rules.php 2015-08-21 17:13:30.000000000 +0000 @@ -46,10 +46,72 @@ require_once("functions.inc"); require_once("filter.inc"); require_once("shaper.inc"); +require_once("rule_count.inc"); $pgtitle = array(gettext("Firewall"),gettext("Rules")); $shortcut_section = "firewall"; +//Get rule Hit count +function get_rule_ht($tracker,$sum_ht=array()){ + global $g,$rules_count_array; + //check if there is previous values + $packets=(isset($sum_ht['packets']) ? $sum_ht['packets'] : 0); + $states=(isset($sum_ht['bytes']) ? $sum_ht['bytes'] : 0); + $rules_title=(isset($sum_ht['title']) ? $sum_ht['title'] : ""); + $rules_id=array(); + if (preg_match("/\s+/",$tracker)){ + $rules_title.="$tracker\n"; + } + if (is_array($rules_count_array) && array_key_exists($tracker,$rules_count_array)) { + $packets +=$rules_count_array[$tracker]['Packets']; + $states +=$rules_count_array[$tracker]['States']; + foreach ($rules_count_array[$tracker] as $rck => $rcv) { + switch($rck){ + case "Label": + $label=$rcv; + break; + case "RuleId": + $ruleid=$rcv; + break; + default: + $rules_title.= "$rck: ".bd_nice_number($rcv)."\n"; + } + } + } + $hitcount=bd_nice_number($packets)."/".bd_nice_number($states); + if ($states > 0){ + //icon_log.gif + ///themes/pfsense_ng/images/icons/icon_log_d.gif + //$html= + $html="{$hitcount} "; + $html.="![](\"./themes/{$g['theme']}/images/icons/icon_block.gif\")"; + //$resp_info="Kill Rule States {$rulelabel}" . $resp; + } else { + $html="{$hitcount}"; + } + return (array( 'title'=> $rules_title, + 'packets' =>$packets, + 'states' =>$states, + 'html' =>$html + )); +} + +function bd_nice_number($n) { + // first strip any formatting; + $n = (0+str_replace(",","",$n)); + + // is this a number? + if(!is_numeric($n)) return false; + + // now filter it; + if($n>1000000000000) return round(($n/1000000000000),1).'t'; + else if($n>1000000000) return round(($n/1000000000),1).'g'; + else if($n>1000000) return round(($n/1000000),1).'m'; + else if($n>1000) return round(($n/1000),1).'k'; + + return number_format($n); +} + function delete_nat_association($id) { global $config; @@ -135,6 +197,70 @@ $savemsg = sprintf(gettext("The settings have been applied. The firewall rules are now reloading in the background. You can also %s monitor %s the reload progress"),"[","](status_filter_reload.php)"); } + /* handle AJAX operations */ + if($_POST['action'] == "KillRuleStates") { + if (isset($_POST['label'])){ + $rulelabel=html_entity_decode($_POST['label']); + $cnt_pfctlk=array(); + exec("/sbin/pfctl -k label -k \"{$rulelabel}\" 2>&1",$cnt_pfctlk); + } else { + echo gettext("invalid input"); + } + if (!empty($cnt_pfctlk)) { + foreach($cnt_pfctlk as $line) { + print "$line\n"; + } + } + return; + } + if($_POST['action'] == "ShowRuleStates") { + $td1=""; + $td2=""; + //$td2=""; + $resp = ""; + $resp .= ""; + $resp .= $td1 . gettext("Proto") . ""; + $resp .= $td1 . gettext("Source -> Router -> Destination") . ""; + $resp .= $td1 . gettext("State") . ""; + $resp .= $td1 . gettext("Packets") . ""; + $resp .= $td1 . gettext("bytes") . ""; + $resp .= ""; + $state_count=0; + /*if (isset($_POST['label'])){ + $rulelabel=html_entity_decode($_POST['label']); + } + if (isset($_POST['packets'])){ + $packets=html_entity_decode($_POST['packets']); + } + */ + if (isset($_POST['hitcount'])){ + $hitcount=html_entity_decode($_POST['hitcount']); + } + if (isset($_POST['ruleid'])){ + $ruleid=html_entity_decode($_POST['ruleid']); + $cnt_pfctls=array(); + exec("/sbin/pfctl -vvss | /usr/bin/grep -EB2 \"rule ({$ruleid})\"",$cnt_pfctls); + } else { + echo gettext("invalid input"); + } + if (!empty($cnt_pfctls)) { + foreach($cnt_pfctls as $line) { + if (preg_match("/^\w+\s+(\w+)\s+(.*)(.-.)(.*)\s+(\w+:\w+)/",$line,$mcon)) { + $state_count++; + $resp .= "{$td2}{$mcon[1]}{$td2}{$mcon[2]}{$mcon[3]}{$mcon[4]}{$td2}{$mcon[5]}"; + } + elseif (preg_match("/age.*, (\S+) pkts, (\S+) bytes, rule (\d+)/",$line,$mrule)) { + list($pkt1,$pkt2)=split(":",$mrule[1],2); + list($bt1,$bt2)=split(":",$mrule[2],2); + $resp .= "{$td2}".bd_nice_number($pkt1)." / ".bd_nice_number($pkt2)."{$td2}".bd_nice_number($bt1)." / ".bd_nice_number($bt2).""; + } + } + } + $resp .= " "; + $html.="<u>{$hitcount}</u>"; + print ($html); + return; + } } if ($_GET['act'] == "del") { @@ -233,7 +359,66 @@ <form action="firewall_rules.php" method="post"> + @@ -272,7 +457,7 @@ - + "> pfSense_handle_custom_code("/usr/local/pkg/firewall_rules/pre_id_tablehead"); ?> @@ -319,11 +504,12 @@ || ((count($config['interfaces']) == 1) && ($if == 'wan')))): $alports = implode(' ', filter_get_antilockout_ports(true)); + $rule_hit_count=get_rule_ht("anti-lockout rule"); ?> ![pass](./themes/<?= $g['theme']; ?>/images/icons/icon_pass.gif) - + pfSense_handle_custom_code("/usr/local/pkg/firewall_rules/pre_id_tr_antilockout"); ?> @@ -351,11 +537,15 @@ - ++ $rule_hit_count=get_rule_ht("Block private networks from " . strtoupper($if) . " block 192.168/16"); + $rule_hit_count=get_rule_ht("Block private networks from " . strtoupper($if) . " block 127/8",$rule_hit_count); + $rule_hit_count=get_rule_ht("Block private networks from " . strtoupper($if) . " block 172.16/12",$rule_hit_count); + $rule_hit_count=get_rule_ht("Block private networks from " . strtoupper($if) . " block 10/8",$rule_hit_count);?> ![block](./themes/<?= $g['theme']; ?>/images/icons/icon_block.gif) - + * * @@ -379,11 +569,14 @@ - ++ $rule_hit_count=get_rule_ht("block bogon IPv4 networks from ".strtoupper($if)); + $rule_hit_count=get_rule_ht("block bogon IPv6 networks from ".strtoupper($if),$rule_hit_count); + ?> ![block](./themes/<?= $g['theme']; ?>/images/icons/icon_block.gif) - + * * @@ -606,9 +799,10 @@ $printicon = true; } } + $rule_hit_count=get_rule_ht($filterent['tracker']); ?> - - + + pfSense_handle_custom_code("/usr/local/pkg/firewall_rules/pre_id_tr"); @@ -816,6 +1010,7 @@ </form> +
Base Directory:/usr/local/www/
After creating/saving these patches, click test and if all goes fine, click apply.
Step 4:
Edit any rule and save to update pfctl and start getting hit counts on gui -
This looks great! Any plans to get this into 2.2.5 or 2.3?
-
Would need some work before being integrated (would have to be 2.3)
Why use a hash there? In 2.2.x all rules have a unique tracker ID already, no need for an extra hash.
2.1.x is dead, not worth adding things for to maintain compatibility at this stage.
-
Why use a hash there? In 2.2.x all rules have a unique tracker ID already, no need for an extra hash.
Hi Jimp, Thank's for the feedback. I'm testing with tracker right now.
It looks like rules created by nat does not have this tracker(testing with 2.2.4).
Using tracker field, it reduced to just one new file and two patches.
<rule><source> <any><interface>wan</interface> <protocol>tcp</protocol> <destination><address>127.0.0.1</address> <port>80</port></destination> <associated-rule-id>nat_55d21c4e8742b7.64890792</associated-rule-id> <created><time>1439833166</time> <username>NAT Port Forward</username></created></any></rule>
-
I've updated the code for 2.2.4 to use tracker and included counter to all rules on gui. See the topic above for instructions.
-
Thanks, works great 2.2.4.
One question though, shouldn't we rather use 644 permissions? At least for /etc/inc files ? -
One question though, shouldn't we rather use 644 permissions? At least for /etc/inc files ?
Yeah, you should. Also, you can diff the new file against /dev/null, put it into System Patches and forget about Filer altogether.
-
One question though, shouldn't we rather use 644 permissions? At least for /etc/inc files ?
Updated to 0644, thanks for the feedback.
-
Something seems odd with pfSense handling auto-added Port Forward rules. They get a name like "USER_RULE: NAT …" and there's no tracker ID. As I use Multi-WAN and some Port Forwards were just copied from WAN1 to WAN2, pfctl -vvsr shows two rules with the same name. That breaks the new counters...
Edit: Put in unique descriptions, still no luck. -
On to the next problem, Port Aliases. While pf let's us write single-line rules with something like
port { 25 465 587 }
it automatically creates a separate rule for every single port. pfctl will show three rules for the example above whilst our ruleset only has a single rule for this.
-
Updated the code for 2.2.4 to include kill states option for specific rule.
-
On to the next problem, Port Aliases. While pf let's us write single-line rules with something like
port { 25 465 587 }
it automatically creates a separate rule for every single port. pfctl will show three rules for the example above whilst our ruleset only has a single rule for this.
I'll try to simulate it.
-
Updated the code for 2.2.4 to include kill states option for specific rule.
Wasn't able to try it out yet. But if that means, the byte counter does not show anymore when hovering there with the mouse, that would be too bad. Maybe the Hits field could show
Packets(Bytes)/States
or something similar? -
Wasn't able to try it out yet. But if that means, the byte counter does not show anymore when hovering there with the mouse, that would be too bad. Maybe the Hits field could show
Packets(Bytes)/States
or something similar?I'ts still working inside the td with mouse over, I've just included the option to kill when there are active states and only over the numbers.
-
Something seems odd with pfSense handling auto-added Port Forward rules. They get a name like "USER_RULE: NAT …" and there's no tracker ID.
Edit and save rule created by nat to force the trackerid
-
On to the next problem, Port Aliases. While pf let's us write single-line rules with something like
port { 25 465 587 }
it automatically creates a separate rule for every single port. pfctl will show three rules for the example above whilst our ruleset only has a single rule for this.
check if new code fixes it. (v0.31)
@78(1439883226) pass in quick on em0 reply-to (em0 172.17.12.1) inet proto tcp from any to (self:3) port = http flags S/SA keep state label "USER_RULE: 1439883226" [ Evaluations: 17 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: pid 23535 State Creations: 3319624344] @79(1439883226) pass in quick on em0 reply-to (em0 172.17.12.1) inet proto tcp from any to (self:3) port = https flags S/SA keep state label "USER_RULE: 1439883226" [ Evaluations: 17 Packets: 763 Bytes: 442121 States: 6 ] [ Inserted: pid 23535 State Creations: 3319624408] @80(1439883226) pass in quick on em0 reply-to (em0 172.17.12.1) inet proto tcp from any to (self:3) port = ssh flags S/SA keep state label "USER_RULE: 1439883226" [ Evaluations: 1 Packets: 555 Bytes: 81812 States: 1 ] [ Inserted: pid 23535 State Creations: 3321614424] @81(1439883226) pass in quick on em0 reply-to (em0 172.17.12.1) inet proto tcp from any to (self:3) port = 8443 flags S/SA keep state label "USER_RULE: 1439883226" [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: pid 23535 State Creations: 3321614400] @82(1439883226) pass in quick on em0 reply-to (em0 172.17.12.1) inet proto tcp from any to (self:3) port = domain flags S/SA keep state label "USER_RULE: 1439883226" [ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ] [ Inserted: pid 23535 State Creations: 3321614376]
-
Thanks, works great.
But one odd thing I noticed is, that rules with multiple ports like those you fixed seem to get their counter cleared on every filter reload. At the same time, my other "normal" rules do not suffer from this.
Are you seeing this too, or am I doing something wrong? -
More features on version 0.4 for pfsense 2.2, now we can view hit count, list and kill States(with ajax to keep it light and fast).