Duplicated firewall rules



  • Hi,

    I have the following problem with my pfsense 2.2.6:

    2.2.6-RELEASE (amd64)
    built on Mon Dec 21 14:50:08 CST 2015
    FreeBSD 10.1-RELEASE-p25

    • dual WAN (load balancing)
    • multiple vlan interfaces on the lan side

    firewall rules from both wan interfaces are multiplied by at least 1000 times (4-5 rules multiplied 1000 times)

    how can delete the duplicated rules from both interfaces without reinstall? the pfsense is in production and I cannot afford a downtime for the moment.

    any useful  help needed.

    Liviu



  • Select all the ones you want to delete and click the X at the bottom of the screen.

    Any idea how you ended up with duplicated rules?



  • @cmb:

    Any idea how you ended up with duplicated rules?

    Indeed. That's what would worry me more than the task of removing the duplicates.



  • Wonder if someone held down F5 and re-posted a creation event.



  • @cmb:

    Select all the ones you want to delete and click the X at the bottom of the screen.

    Any idea how you ended up with duplicated rules?

    hi, same problem duplicated multiple rules 512 times, config file xml has 110.000 lines , when accesing webgui php-fpm goes 100% on cpu and timeout…is there a  way to delete rules from ssh ....or restore good config without webgui ?

    thanks



  • +1

    My System

    pfSense CE v2.2.3
    FreeBSD gateway.mlan.local 10.3-RELEASE-p5 FreeBSD 10.3-RELEASE-p5 #0 7307492(RELENG_2_3_2): Tue Jul 19 13:29:35 CDT 2016    root@ce23-amd64-builder:/builder/pfsense-232/tmp/obj/builder/pfsense-232/tmp/FreeBSD-src/sys/pfSense  amd64

    Packages Installed

    • Avahi
    • Squid
    • Lightsquid
    • pfBlockerNG

    Observations

    • The number of rules listed against the WAN interface grew to 1000+, causing php-fpm to peg at 100% CPU usage any time a config change was made in the web-gui
    • A free install of pfSense and a re-load of the old config caused the issue to re-occur (config had 1000+ rules)
    • After removing all of the duplicates, re-installing pfSense and then loading the 'cleaned' config the issue re-occurs
    • 9am: pfSense re-installed (35 rules - counted using 'grep -c tracker /conf/config.xml')
    • 6pm: Now 155 rules (all dupes in the WAN section - counted using 'grep -c tracker /conf/config.xml')
    • 9am following day: Now 283 rules (all dupes in the WAN section - counted using 'grep -c tracker /conf/config.xml')
    • The duplicates all have the same <tracker>values as their originals
    • The multiplier of duplicates I get is always a multiple of 2 (32, 64, 256, 512, 1024, etc)
    
    grep tracker /conf/config.xml  | sort | uniq -c | sort -n | tail -10
       1                    <tracker>1455574900</tracker>
       1                    <tracker>1470906216</tracker>
      32                    <tracker>1422442622</tracker>
      32                    <tracker>1422442623</tracker>
      32                    <tracker>1447961159</tracker>
      32                    <tracker>1447961390</tracker>
      32                    <tracker>1447961411</tracker>
      32                    <tracker>1448888848</tracker>
      32                    <tracker>1448888874</tracker>
      32                    <tracker>1448888893</tracker>
    
    

    Theories Tested:

    • Thought it might be a browser issue or plugin. Tried with Firefox, Chrome and event 'Lynx' - same issue
    • Thought it might be pfBlockerNG. Disabled it. Issue still occurs

    Sample Rule:

    
     <rule><id><tracker>1448888893</tracker>
        <type>pass</type>
        <interface>wan</interface>
        <ipprotocol>inet</ipprotocol>
        <tag><tagged><max><max-src-nodes><max-src-conn><max-src-states><statetimeout><statetype>keep state</statetype>
        <os><protocol>tcp</protocol>
        <source>
            <any><destination><address>192.168.1.40</address>
    
            <port>1234</port></destination> 
    
        <updated><time>1448888893</time>
            <username>admin@192.168.1.254</username></updated> 
        <created><time>1448888893</time>
            <username>admin@192.168.1.254</username></created></any></os></statetimeout></max-src-states></max-src-conn></max-src-nodes></max></tagged></tag></id></rule> 
    
    ```</tracker>


  • For anyone having the same issue you can temporarily fix the issue (and cleanup the rules) using the following steps if you access to a Linux box

    You can install the necessary utils on Ubuntu with "sudo apt-get install xgrep libxml2-utils"

    1. Build a list of unique rules in your config

    
    xmllint --c14n --noblanks config-<date>.xml \
        | xgrep -x '/pfsense/filter/rule' -t \
        | sort \
        | uniq \
        | xmllint --c14n --noblanks -</date> 
    

    2. Replace all the text between the tags in your config file with the output from the command above and save it for future use (eg. config-cooked.xml)

    3. Using the Diagnostics -> Backup & Restore menu. Select your 'cooked' config and load only the Firewall Rules section from the config



  • You can also determine how many instances of the duplicate rules you have with the command below;

    grep tracker /conf/config.xml  | sort | uniq -c | sort -n | tail -15
    

    No Problems Here!

    [root@gateway ~]# grep tracker /conf/config.xml  | sort | uniq -c | sort -n | tail -15
       1                    <tracker>1470906216</tracker>
       1                    <tracker>1770008480</tracker>
       1                    <tracker>1770008555</tracker>
       1                    <tracker>1770008584</tracker>
       1                    <tracker>1770008659</tracker>
       1                    <tracker>1770008755</tracker>
       1                    <tracker>1770008797</tracker>
       1                    <tracker>1770008859</tracker>
       1                    <tracker>1770008861</tracker>
       1                    <tracker>1770008901</tracker>
       1                    <tracker>1770008941</tracker>
       1                    <tracker>1770008946</tracker>
       1                    <tracker>1770008965</tracker>
       1                    <tracker>1770009045</tracker>
       1                    <tracker>1770009050</tracker>
    
    

    Time to Fix the Rules!

    
    [root@gateway ~]# grep tracker /conf/config.xml  | sort | uniq -c | sort -n | tail -15
       1                    <tracker>1770008901</tracker>
       1                    <tracker>1770008941</tracker>
       1                    <tracker>1770008946</tracker>
       1                    <tracker>1770008965</tracker>
       1                    <tracker>1770009045</tracker>
       1                    <tracker>1770009050</tracker>
     512                    <tracker>1422442622</tracker>
     512                    <tracker>1422442623</tracker>
     512                    <tracker>1422442625</tracker>
     512                    <tracker>1447961159</tracker>
     512                    <tracker>1447961390</tracker>
     512                    <tracker>1447961411</tracker>
     512                    <tracker>1448888848</tracker>
     512                    <tracker>1448888874</tracker>
     512                    <tracker>1448888893</tracker>
    
    


  • Possibly Fixed with the complete removal of pfblockerNG

    • I've upgraded to v2.3.2_1 [https://blog.pfsense.org/?p=2122]
    • I then un-installed pfBlockerNG
    • Restored my Firewall Rules from 'good' config
    • Removed the pfBlockerNG rules (no aliases any more) from Firewall
    • Reboot.

    No re-occurrences in 9hrs and counting (always had at least 1 occurrence after a reboot)


Log in to reply