Floating rules: outbound block rules not silently dropping packets



  • I have set up a lab environment with pfSense 2.1.0-BETA0 snapshot (Built On: Thu Dec  6 18:43:41 EST 2012) and two lab servers. I'm not sure if this applies specifically to 2.1 or if this is a general problem with floating rules, so apologies if this is in the wrong forum, I guess the topic could be moved by a moderator.

    First, a little background to what I'm trying to achieve and why. We operate pfSense as a firewall / NAT solution isolating customer servers from the Internet (except for selected services) and from each others.

    • Each customer may have one or more servers.
    • Every customer is assigned one (in some rare cases more than one) VLAN.
    • Every customer is assigned IP space so that no two of our customers have the same IP address.
    • Customers should not be able to talk to each other, except if the traffic is permitted by a rule that would have permitted the Internet at large to talk to the server.
    • Customers should be able to talk to any host on the Internet

    I'm finding that traditional filtering on inbound traffic is not workable here, because there's no "good" way of setting up a rule there that would permit all traffic to "the internet" while not permitting traffic to other customers. For this reason I want to use floating rules for all of these policies, so that they may apply on all interfaces. Then I want to add a floating default-deny for all outbound traffic, and then add a default-permit for traffic going towards the WAN interface specifically.

    I have this almost just fine in a lab setup. A diagram of the setup has been attached.

    And here is the /tmp/rules.debug to show what rules I have set up. (I have redacted my public IP address replacing the first three octets with x.y.z)

    set limit tables 3000
    set optimization normal
    set limit states 98000
    set limit src-nodes 98000
    
    #System aliases
    
    loopback = "{ lo0 }"
    WAN = "{ em0 }"
    LAN = "{ em1 }"
    OPT1 = "{ em2 }"
    
    #SSH Lockout Table
    table <sshlockout>persist
    table <webconfiguratorlockout>persist
    #Snort tables
    table <snort2c>table <virusprot># User Aliases 
    
    # Gateways
    GWGW_WAN = " route-to ( em0 x.y.z.1 ) "
    GWWAN_DHCP6 = "  "
    
    set loginterface em1
    
    set skip on pfsync0
    
    scrub on $WAN all    fragment reassemble
    scrub on $LAN all    fragment reassemble
    scrub on $OPT1 all    fragment reassemble
    
    no nat proto carp
    no rdr proto carp
    nat-anchor "natearly/*"
    nat-anchor "natrules/*"
    
    # Outbound NAT rules
    nat on $WAN  from 10.254.0.0/24 to any -> x.y.z.33/32 port 1024:65535  
    nat on $WAN  from 10.255.0.0/24 to any -> x.y.z.33/32 port 1024:65535  
    
    # Load balancing anchor
    rdr-anchor "relayd/*"
    # TFTP proxy
    rdr-anchor "tftp-proxy/*"
    table <negate_networks>{}
    # NAT Inbound Redirects
    rdr on em0 proto tcp from any to x.y.z.33 port 25422 -> 10.254.0.50 port 22
    # Reflection redirect
    rdr on { em1 em2 } proto tcp from any to x.y.z.33 port 25422 -> 10.254.0.50 port 22
    rdr on em0 proto tcp from any to x.y.z.33 port 25522 -> 10.255.0.50 port 22
    # Reflection redirect
    rdr on { em1 em2 } proto tcp from any to x.y.z.33 port 25522 -> 10.255.0.50 port 22
    # UPnPd rdr anchor
    rdr-anchor "miniupnpd"
    
    anchor "relayd/*"
    anchor "openvpn/*"
    anchor "ipsec/*"
    #---------------------------------------------------------------------------
    # default deny rules
    #---------------------------------------------------------------------------
    block in log inet all label "Default deny rule IPv4"
    block out log inet all label "Default deny rule IPv4"
    block in log inet6 all label "Default deny rule IPv6"
    block out log inet6 all label "Default deny rule IPv6"
    
    # IPv6 ICMP is not auxilary, it is required for operation
    # See man icmp6(4)
    # 1    unreach         Destination unreachable
    # 2    toobig          Packet too big
    # 128  echoreq         Echo service request
    # 129  echorep         Echo service reply
    # 133  routersol       Router solicitation
    # 134  routeradv       Router advertisement
    # 135  neighbrsol      Neighbor solicitation
    # 136  neighbradv      Neighbor advertisement
    pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} keep state
    
    # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep)
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} keep state
    pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} keep state
    pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} keep state
    
    # We use the mighty pf, we cannot be fooled.
    block quick inet proto { tcp, udp } from any port = 0 to any
    block quick inet proto { tcp, udp } from any to any port = 0
    block quick inet6 proto { tcp, udp } from any port = 0 to any
    block quick inet6 proto { tcp, udp } from any to any port = 0
    
    # Snort package
    block quick from <snort2c>to any label "Block snort2c hosts"
    block quick from any to <snort2c>label "Block snort2c hosts"
    
    # SSH lockout
    block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout"
    
    # webConfigurator lockout
    block in log quick proto tcp from <webconfiguratorlockout>to any port 443 label "webConfiguratorlockout"
    block in quick from <virusprot>to any label "virusprot overload table"
    table <bogons>persist file "/etc/bogons"
    table <bogonsv6>persist file "/etc/bogonsv6"
    # block bogon networks
    # http://www.cymru.com/Documents/bogon-bn-nonagg.txt
    # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt
    block in log quick on $WAN from <bogons>to any label "block bogon IPv4 networks from WAN"
    block in log quick on $WAN from <bogonsv6>to any label "block bogon IPv6 networks from WAN"
    antispoof for em0
    # block anything from private networks on interfaces with the option set
    antispoof for $WAN
    block in log quick on $WAN from 10.0.0.0/8 to any label "Block private networks from WAN block 10/8"
    block in log quick on $WAN from 127.0.0.0/8 to any label "Block private networks from WAN block 127/8"
    block in log quick on $WAN from 100.64.0.0/10 to any label "Block private networks from WAN block 100.64/10"
    block in log quick on $WAN from 172.16.0.0/12 to any label "Block private networks from WAN block 172.16/12"
    block in log quick on $WAN from 192.168.0.0/16 to any label "Block private networks from WAN block 192.168/16"
    block in log quick on $WAN from fc00::/7 to any label "Block ULA networks from WAN block fc00::/7"
    
    # allow our DHCPv6 client out to the WAN
    pass in quick on $WAN proto udp from fe80::/10 port = 546 to fe80::/10 port = 546 label "allow dhcpv6 client in WAN"
    pass in quick on $WAN proto udp from any port = 547 to any port = 546 label "allow dhcpv6 client in WAN"
    pass out quick on $WAN proto udp from any port = 546 to any port = 547 label "allow dhcpv6 client out WAN"
    antispoof for em1
    
    # allow access to DHCP server on LAN
    pass in quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $LAN proto udp from any port = 68 to 10.254.0.1 port = 67 label "allow access to DHCP server"
    pass out quick on $LAN proto udp from 10.254.0.1 port = 67 to any port = 68 label "allow access to DHCP server"
    
    # allow access to DHCPv6 server on LAN
    anchor "dhcpv6serverLAN"
    # We need inet6 icmp for stateless autoconfig and dhcpv6
    pass quick on $LAN inet6 proto udp from fe80::/10 to fe80::/10 port = 546 label "allow access to DHCPv6 server"
    pass quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 546 label "allow access to DHCPv6 server"
    pass quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 547 label "allow access to DHCPv6 server"
    pass quick on $LAN inet6 proto udp from ff02::/16 to fe80::/10 port = 547 label "allow access to DHCPv6 server"
    pass in quick on $LAN inet6 proto udp from fe80::/10 to  port = 546 label "allow access to DHCPv6 server"
    pass out quick on $LAN inet6 proto udp from  port = 547 to fe80::/10 label "allow access to DHCPv6 server"
    antispoof for em2
    
    # allow access to DHCP server on OPT1
    pass in quick on $OPT1 proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server"
    pass in quick on $OPT1 proto udp from any port = 68 to 10.255.0.1 port = 67 label "allow access to DHCP server"
    pass out quick on $OPT1 proto udp from 10.255.0.1 port = 67 to any port = 68 label "allow access to DHCP server"
    
    # loopback
    pass in on $loopback inet all label "pass IPv4 loopback"
    pass out on $loopback inet all label "pass IPv4 loopback"
    pass in on $loopback inet6 all label "pass IPv6 loopback"
    pass out on $loopback inet6 all label "pass IPv6 loopback"
    # let out anything from the firewall host itself and decrypted IPsec traffic
    pass out inet all keep state allow-opts label "let out anything IPv4 from firewall host itself"
    pass out inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself"
    # make sure the user cannot lock himself out of the webConfigurator or SSH
    pass in quick on em1 proto tcp from any to (em1) port { 443 80 22 } keep state label "anti-lockout rule"
    # NAT Reflection rules
    pass in inet tagged PFREFLECT keep state label "NAT REFLECT: Allow traffic to localhost"
    
    # User-defined rules follow
    
    anchor "userrules/*"
    block  out log inet from any to any  label "USER_RULE: Default deny outbound traffic"
    pass  out  on {  em0  } inet from any to any keep state  label "USER_RULE: Default permit traffic exiting on WAN"
    pass log inet proto tcp  from any to   10.254.0.50 port 22  flags S/SA keep state  label "USER_RULE: Allow ssh to kund1 server (NATed or not NATed)"
    pass log inet proto tcp  from any to   10.255.0.50 port 22  flags S/SA keep state  label "USER_RULE: Allow ssh to kund2 server (NATed or not NATed)"
    pass  in  quick  on $WAN inet from   x.y.z.49 to   x.y.z.33 keep state  label "USER_RULE: Allow administrative traffic"
    pass  in  quick  on $LAN inet from any to any keep state  label "USER_RULE: Outbound traffic is filtered by floating rules"
    pass  in  quick  on $OPT1 inet from any to any keep state  label "USER_RULE: Outbound traffic is filtered by floating rules"
    
    # Automatic Pass rules for any delegated IPv6 prefixes through dynamic IPv6 clients
    
    # VPN Rules
    anchor "tftp-proxy/*"</bogonsv6></bogons></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></virusprot></snort2c></webconfiguratorlockout></sshlockout> 
    

    This is working, with one strange caveat. Observe:

    
    pvz@srv1:~$ ifconfig eth0
    eth0      Link encap:Ethernet  HWaddr 00:50:56:97:39:88
              inet addr:10.254.0.50  Bcast:10.254.0.255  Mask:255.255.255.0
              inet6 addr: fe80::250:56ff:fe97:3988/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:129754 errors:0 dropped:0 overruns:0 frame:0
              TX packets:22657 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:175569470 (175.5 MB)  TX bytes:11916745 (11.9 MB)
    
    pvz@srv1:~$ sudo tcpdump -n -v -i eth0 'icmp and host 10.254.0.1' &
    [1] 7246
    pvz@srv1:~$ tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes
    
    pvz@srv1:~$ ping -c 1 10.255.0.50
    PING 10.255.0.50 (10.255.0.50) 56(84) bytes of data.
    16:57:52.030661 IP (tos 0x0, ttl 64, id 59938, offset 0, flags [DF], proto ICMP (1), length 56)
        10.254.0.1 > 10.254.0.50: ICMP host 10.255.0.50 unreachable, length 36
            IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84)
        10.254.0.50 > 10.255.0.50: ICMP echo request, id 7248, seq 1, length 64
    From 10.254.0.1 icmp_seq=1 Destination Host Unreachable
    
    --- 10.255.0.50 ping statistics ---
    1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms
    
    pvz@srv1:~$ nc -v 10.255.0.50 12345
    16:58:07.514405 IP (tos 0x0, ttl 64, id 19012, offset 0, flags [DF], proto ICMP (1), length 88)
        10.254.0.1 > 10.254.0.50: ICMP host 10.255.0.50 unreachable, length 68
            IP (tos 0x0, ttl 64, id 51115, offset 0, flags [DF], proto TCP (6), length 60)
        10.254.0.50.42934 > 10.255.0.50.12345: Flags [s], cksum 0x3251 (correct), seq 3858768486, win 14600, options [mss 1460,sackOK,TS val 2483661 ecr 0,nop,wscale 4], length 0
    nc: connect to 10.255.0.50 port 12345 (tcp) failed: No route to host
    pvz@srv1:~$ nc -v 10.255.0.50 22
    Connection to 10.255.0.50 22 port [tcp/ssh] succeeded!
    SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
    ^C
    pvz@srv1:~$
    
    The connections are blocked and passed as expected, but instead of just timing out, there is an ICMP error thrown by the firewall that can give an attacker a clue that the traffic is being blocked. I have made sure I've chosen "block" rather than "reject" in the floating rule that ends up blocking the pattern, as can be seen in /tmp/rules.debug
    
    So what gives? I'm sure I can concoct some kind of ugly ugly workaround in which I use an outbound policy to block ICMP host unreachable messages from the firewall, but it really ought not to be neccessary if the system is working as intended.
    ![lab.png](/public/_imported_attachments_/1/lab.png)
    ![lab.png_thumb](/public/_imported_attachments_/1/lab.png_thumb)[/s]
    


  • First, I cannot tell which are floating rules and which are not, but I can tell you that by default floating rules are last matching wins unless you specify the "quick" option.
    Also, by default, the first rules in the pre-programmed are the floating block rules. This means they all traffic is blocked unless there is a pass rule.



  • @podilarius:

    First, I cannot tell which are floating rules and which are not, but I can tell you that by default floating rules are last matching wins unless you specify the "quick" option.
    Also, by default, the first rules in the pre-programmed are the floating block rules. This means they all traffic is blocked unless there is a pass rule.

    All inbound traffic is blocked by default. But all outbound traffic is passed by default.

    In this specific example, the floating rules are the user-defined rules without the "quick" keyword in them. (In my case, I have no "quick" floating rules.)



  • It is passed only because there is a rule created on LAN. You can change this rule. If you delete this rule, all access to the internet will stop. This is because of the unseen default block rules. Rules create in LAN, WAN and OPTx are all quick rules, this means first matching rule wins. So a block before and allow will always block the traffic. Floating is the opposite unless the quick option is checked.



  • @podilarius:

    It is passed only because there is a rule created on LAN. You can change this rule. If you delete this rule, all access to the internet will stop. This is because of the unseen default block rules. Rules create in LAN, WAN and OPTx are all quick rules, this means first matching rule wins. So a block before and allow will always block the traffic. Floating is the opposite unless the quick option is checked.

    Yes, I know this already. The inbound rules are traversed first. Only if the inbound rules permit the packet to pass, will the outbound rules be traversed. I'm also aware about the difference between quick and non-quick.

    This doesn't get us any closer to explaining why I'm getting an ICMP Host Unreachable back from the firewall when I'm doing something I'm not supposed to, rather than the firewall just dropping the packet on the floor, which was the actual subject of this thread.



  • I have just checked on 2.0.1-RELEASE, the same exact problem is present there. Could some friendly mod please move this topic to the Firewalling forum?



  • It turns out that there's some "undocumented behaviour" in which where a packet that has been passed on the incoming side is then blocked on the outgoing side causes pf to emit an ICMP host unreachable message. I've confirmed this on a lot of different systems, including the latest pf upstream (OpenBSD 5.2).

    But I was able to work around this issue with an ugly hack. I discovered that by using a policy route to null route the packets instead of just "dropping" them, I could get the behaviour I wanted.

    So I just patched the pfsense rule generation code to do just that.

    This will probably break about 99 different things, but it seems to work for me in my test environment. If anyone else is having these issues and wants to give it a go, here are my local changes to pfsense (run the patch inside /etc/inc)

    
    --- filter.inc.orig     2012-12-11 02:41:15.000000000 +0100
    +++ filter.inc  2012-12-11 02:52:53.000000000 +0100
    @@ -2342,6 +2342,18 @@
                            " label \"NEGATE_ROUTE: Negate policy routing for destination\"\n";
    
            }
    +
    +       /* ugly hack to convert "block" rules to "pass route-to 0.0.0.0" rules to work around a bug which causes floating rules applied on exit
    +          interfaces to improperly send an ICMP unreachable instead of dropping the packet on the floor. this workaround fixes that. */
    +       if ($type == "block" && isset($rule['ipprotocol']) && ($rule['ipprotocol'] == "inet" || $rule['ipprotocol'] == "inet6")) {
    +               $aline['type'] = "pass ";
    +                if ($rule['ipprotocol'] == "inet") {
    +                       $aline['route'] = " route-to 0.0.0.0 ";
    +               } else {
    +                       $aline['route'] = " route-to :: ";
    +               }
    +       }
    +
            /* piece together the actual user rule */
            $line .= $aline['type'] . $aline['direction'] . $aline['log'] . $aline['quick'] . $aline['interface'] .
                    $aline['reply'] . $aline['route'] . $aline['ipprotocol'] . $aline['prot'] . $aline['src'] . $aline['os'] . $aline['dst'] .
    --- filter_log.inc.orig 2012-12-11 03:16:15.000000000 +0100
    +++ filter_log.inc      2012-12-11 03:15:44.000000000 +0100
    @@ -159,6 +159,13 @@
                    $flent['proto'] = "none";
            }
    
    +       /* check for the hack where we're null routing. */
    +        list($rulenum, $junk) = explode('/', $rule);
    +        $rule2 = find_rule_by_number($rulenum, $flent['act']);
    +       if (!(strpos($rule2, " route-to 0.0.0.0 ") === FALSE) || !(strpos($rule2, " route-to :: ") === FALSE)) {
    +               $flent['act'] = "block";
    +       }
    +
            /* If there is a src, a dst, and a time, then the line should be usable/good */
            if (!((trim($flent['src']) == "") || (trim($flent['dst']) == "") || (trim($flent['time']) == ""))) {
                    return $flent;
    
    

    This is most emphatically not the "right" way to do it. But it's a practical solution. More practical than digging into the internals of pf. At least for me. :-)



  • While I may have a much simpler setup but I do have a similar concept. I have three interfaces on my pfsense box: wan - lan - opt1. I keep traffic between lan and opt 1 separate entirely, but both are allowed to use the internet. I believe the same is for you with the addition of vlans. Right now I do not permit any traffic but DNS/DHCP requests and traffic to the internet.


  • Rebel Alliance Developer Netgate

    @pv2b - you've discovered one of the many reasons it's best to block the traffic as close to the source as possible.

    Sure an outbound floating rule shortcut sounds convenient, but ultimately it's best to block on the individual tabs (or both, just to be extra sure).



  • I don't agree that it's neccessarilly better to block as close to the source as possible. There's a compelling argument to the opposite in fact: it's generally better to block as close to the target as possible, because otherwise you have to trust all the intervening nodes not to send malicious traffic. That's why we have personal firewalls filtering our incoming traffic instead of relying on everyone else to do egress filtering so no evil packets escape their networks.

    Either way the argument for blocking like this isn't convenience, it's security. It's far too easy to screw up when you have to duplicate the same rule on a lot of interfaces.



  • @pv2b:

    It's far too easy to screw up when you have to duplicate the same rule on a lot of interfaces.

    That's never a requirement, use interface groups if you need to do that.


  • Rebel Alliance Developer Netgate

    @pv2b:

    I don't agree that it's neccessarilly better to block as close to the source as possible. There's a compelling argument to the opposite in fact: it's generally better to block as close to the target as possible, because otherwise you have to trust all the intervening nodes not to send malicious traffic. That's why we have personal firewalls filtering our incoming traffic instead of relying on everyone else to do egress filtering so no evil packets escape their networks.

    Either way the argument for blocking like this isn't convenience, it's security. It's far too easy to screw up when you have to duplicate the same rule on a lot of interfaces.

    Not quite. It's better to block as close to the source as possible under your control. Why let traffic any farther into your network than it needs to go? Your point is valid but only when the traffic is sourced remotely, on networks you do not control. When you control the source, there's no reason to let it pass in when you know it shouldn't.

    Yes, blocking on the way out is also good but because of the way stateful firewalls work, combining the two philosophies will lead to issues like the one you discovered. You allowed the traffic in one way, and blocked it out another. That is never as good as blocking it before it comes in.

    By using aliases, interface groups, etc, you can reduce the likelihood of making an error or over-duplicating.


Locked