Firewall rule toggle fails after several on/off cycles : 2.0.3 release



  • 4 days checking, searched forums etc. no direct references found. Nonetheless, apologies if this is a known problem.

    Synopsis:
    If a rule is added on the "Firewall->Rules" page of the web GUI and then repeatedly toggled on/off via the icon to the left of the rule, with "Apply changes" clicked in between, a block rule will be displayed as enabled but effectively it is lost, even though the white-on-red icon is dispayed.
    ("Diagnostics->States->Reset States" used to reset states initially after each rule toggle to "on", but turns out to be irrelevant to problem.)

    Test setup:
    pfSence 2.0.3-RELEASE (i386) iso installed to (and booting from) HDD on Dell PE750 with 2 x Gb ethernet
    "factory default" configuration plus:
    "Block private networks" disabled on WAN
    Manual anti-lockout rule on WAN
    ICMP block rule on LAN: single LAN IP -> single WAN IP (placed after auto-anti-lockout rule and before default allow LAN to any rule)

    WAN is on 10.0.0.0/8, DHCP lease on 10.0.0.10, ethernet->ADSL router->Internet
    LAN is on 192.168.0.0/16 with default 192.168.1.1
    1 unmanaged switch on LAN NIC
    1 Win7 laptop ethernet -> switch

    Procedure:
    From LAN PC set up a:
    ping -t {WAN-host}
    Create new ICMP block rule for above, save it and apply changes (rule is effective after states are reset).
    Toggle the rule off via icon (disabled) and apply changes (wait for rules to reload).
    Wait for ping replies to resume
    Toggle the rule on (enabled) and apply changes (wait for rules to reload).
    Reset states via Diagnostics->States (wait to see if ping replies stop).

    Repeate the above cycle until the rule is displayed as toggled on (enabled) but the state reset fails, and the ping continues to pass through the firewall and receive replies.

    Please note: A rule may have to be toggled disable/enable and a different GUI page loaded repeatedly before the error manifests itself. In testing here it can take up to 6 retries before the FW rules page "loses" the rule, but it also happens after just 2 cycles.

    If the rule is toggled thus: enable/disable/enable (with changes applied in between, but without leaving the rules page) then the error is cleared and the rule is sent to the server (see below). A state reset then blocks any current or new connections.

    Entering the rule's individual config page and re-saving without any changes also clears the error without changing the enable/disable box.

    I have tested the above with 4 browsers on 3 OS's: FF 21.0 and IE 9. Linux, WinXP, Win7. Also using Linux box on LAN instead of Win7.
    I have also tested with TCP 80 and SQUID TCP 3128 rules and used the F5 key for repeated page reloads. The same error arrises but is obviously easier to do with a simple ICMP echo repeat.

    I have verified that the in-memory and /tmp/rules.debug rule set are both missing the ICMP block rule when the error state is reached via the shell: pfctl -f /tmp/rules.debug  and  pfctl -sa  are both missing the ICMP block rule completely. The rule reappears once the error state is cleared using the enable/disable/enable procedure.
    This can be easily reproduced, and the state reload is irrelevant since pfSense is unaware of the block rule's existance.
    (! I forgot to stat /tmp/rules.debug in my tests… will do this later today!)

    4th July 2013: I should no better at my age  ;) ... don't guess, test, and then re-test. Strike the following and see below:
    [guesswork:]
    It may be that the FW rule array in the browser's javascript is not correctly reset at some point after repeatedly cycling the toggle? So it cannot be transferred to the pfSense server? The fact that saving the same rule unaltered clears the error suggests this?

    4th July 2013:
    If the web GUI is left displaying the block rule as enabled, but the packets are still passed despite state resets, and then the pfSense server is rebooted the rule reappears in /tmp/rules.debug and output of pfcntl -sa
      ….
    No more guessing (publicly).....

    Otherwise: thanks for developing pfSense! I can only assume it took me years to find it because of a hefty bias towards Linux :)

    Mark



  • Update:

    I have verified the problem and have produced the same error by booting the same PE750 server from a live CD (same 2.0.3 RELEASE (i386)) instead of HDD.

    All settings are factory default, with the only addition being the one ICMP block rule as described above.

    A few more hours testing revealed something very strange, so I decided to check on another server, just to rule out hardware faults.
    Booting a Dell PE1650 from the live CD produces the same results. The error is repeatedly produced.

    (In case you're wondering: Both Dell PE servers are in good condition and only decomissioned as part of a move to VMs.)

    Schroedingers cat is still in the box, and still breathing, maybe:
    The strange revelation (actually probably a hint at the underlying cause) is that running shell commands that stat files (from the console) in between the cycling of the rule toggle prevents the error from appearing (up to 20 cycles with no errors).

    e.g. a simple "ls" like this:
    ls -lTtrA /tmp/rules.*
    in between toggling the rule between enabled/disabled is sufficient for the rules to be committed and reloaded correctly.

    Toggling purely from the GUI without console interaction soon reproduces the error again.

    (@devs: Tried a straight "sync" from shell on the HDD install but this does not have the fix-effect. Another pointer I hope.)

    Mark



  • Hi Mark,

    Just experienced the problem you're describing on my build too. I am running 2.1-RC0 (i386) built on Wed Jun 26 04:29:44 EDT 2013.

    I was experimenting with rule blocking TCP/UDP packets. I disabled rule clicking tiny red x icon adjacent to the tick box. Then re-enabled it the same way after about 15-20 seconds. Rule still doesn't get enabled. All the traffic that was blocked initially (before disabling rule) can still pass although tiny icon changed color from gray back to red.

    I looked at your bug report and it is rejected for some reason. I find it hard to believe that no one else came across this bug. I think people just work around it and don't bother posting on forum.



  • Same problem here, created floating ICMP pass rule for all networks, starts pinging between networks. Then disable rule and nothing happens. Only reboot helps. Release 2.15.



  • Diagnostics, states, reset does the job  :-X


Log in to reply