Floating rules: outbound block rules not silently dropping packets
-
I have set up a lab environment with pfSense 2.1.0-BETA0 snapshot (Built On: Thu Dec 6 18:43:41 EST 2012) and two lab servers. I'm not sure if this applies specifically to 2.1 or if this is a general problem with floating rules, so apologies if this is in the wrong forum, I guess the topic could be moved by a moderator.
First, a little background to what I'm trying to achieve and why. We operate pfSense as a firewall / NAT solution isolating customer servers from the Internet (except for selected services) and from each others.
- Each customer may have one or more servers.
- Every customer is assigned one (in some rare cases more than one) VLAN.
- Every customer is assigned IP space so that no two of our customers have the same IP address.
- Customers should not be able to talk to each other, except if the traffic is permitted by a rule that would have permitted the Internet at large to talk to the server.
- Customers should be able to talk to any host on the Internet
I'm finding that traditional filtering on inbound traffic is not workable here, because there's no "good" way of setting up a rule there that would permit all traffic to "the internet" while not permitting traffic to other customers. For this reason I want to use floating rules for all of these policies, so that they may apply on all interfaces. Then I want to add a floating default-deny for all outbound traffic, and then add a default-permit for traffic going towards the WAN interface specifically.
I have this almost just fine in a lab setup. A diagram of the setup has been attached.
And here is the /tmp/rules.debug to show what rules I have set up. (I have redacted my public IP address replacing the first three octets with x.y.z)
set limit tables 3000 set optimization normal set limit states 98000 set limit src-nodes 98000 #System aliases loopback = "{ lo0 }" WAN = "{ em0 }" LAN = "{ em1 }" OPT1 = "{ em2 }" #SSH Lockout Table table <sshlockout>persist table <webconfiguratorlockout>persist #Snort tables table <snort2c>table <virusprot># User Aliases # Gateways GWGW_WAN = " route-to ( em0 x.y.z.1 ) " GWWAN_DHCP6 = " " set loginterface em1 set skip on pfsync0 scrub on $WAN all fragment reassemble scrub on $LAN all fragment reassemble scrub on $OPT1 all fragment reassemble no nat proto carp no rdr proto carp nat-anchor "natearly/*" nat-anchor "natrules/*" # Outbound NAT rules nat on $WAN from 10.254.0.0/24 to any -> x.y.z.33/32 port 1024:65535 nat on $WAN from 10.255.0.0/24 to any -> x.y.z.33/32 port 1024:65535 # Load balancing anchor rdr-anchor "relayd/*" # TFTP proxy rdr-anchor "tftp-proxy/*" table <negate_networks>{} # NAT Inbound Redirects rdr on em0 proto tcp from any to x.y.z.33 port 25422 -> 10.254.0.50 port 22 # Reflection redirect rdr on { em1 em2 } proto tcp from any to x.y.z.33 port 25422 -> 10.254.0.50 port 22 rdr on em0 proto tcp from any to x.y.z.33 port 25522 -> 10.255.0.50 port 22 # Reflection redirect rdr on { em1 em2 } proto tcp from any to x.y.z.33 port 25522 -> 10.255.0.50 port 22 # UPnPd rdr anchor rdr-anchor "miniupnpd" anchor "relayd/*" anchor "openvpn/*" anchor "ipsec/*" #--------------------------------------------------------------------------- # default deny rules #--------------------------------------------------------------------------- block in log inet all label "Default deny rule IPv4" block out log inet all label "Default deny rule IPv4" block in log inet6 all label "Default deny rule IPv6" block out log inet6 all label "Default deny rule IPv6" # IPv6 ICMP is not auxilary, it is required for operation # See man icmp6(4) # 1 unreach Destination unreachable # 2 toobig Packet too big # 128 echoreq Echo service request # 129 echorep Echo service reply # 133 routersol Router solicitation # 134 routeradv Router advertisement # 135 neighbrsol Neighbor solicitation # 136 neighbradv Neighbor advertisement pass quick inet6 proto ipv6-icmp from any to any icmp6-type {1,2,135,136} keep state # Allow only bare essential icmpv6 packets (NS, NA, and RA, echoreq, echorep) pass out quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {129,133,134,135,136} keep state pass out quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {129,133,134,135,136} keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to fe80::/10 icmp6-type {128,133,134,135,136} keep state pass in quick inet6 proto ipv6-icmp from ff02::/16 to fe80::/10 icmp6-type {128,133,134,135,136} keep state pass in quick inet6 proto ipv6-icmp from fe80::/10 to ff02::/16 icmp6-type {128,133,134,135,136} keep state # We use the mighty pf, we cannot be fooled. block quick inet proto { tcp, udp } from any port = 0 to any block quick inet proto { tcp, udp } from any to any port = 0 block quick inet6 proto { tcp, udp } from any port = 0 to any block quick inet6 proto { tcp, udp } from any to any port = 0 # Snort package block quick from <snort2c>to any label "Block snort2c hosts" block quick from any to <snort2c>label "Block snort2c hosts" # SSH lockout block in log quick proto tcp from <sshlockout>to any port 22 label "sshlockout" # webConfigurator lockout block in log quick proto tcp from <webconfiguratorlockout>to any port 443 label "webConfiguratorlockout" block in quick from <virusprot>to any label "virusprot overload table" table <bogons>persist file "/etc/bogons" table <bogonsv6>persist file "/etc/bogonsv6" # block bogon networks # http://www.cymru.com/Documents/bogon-bn-nonagg.txt # http://www.team-cymru.org/Services/Bogons/fullbogons-ipv6.txt block in log quick on $WAN from <bogons>to any label "block bogon IPv4 networks from WAN" block in log quick on $WAN from <bogonsv6>to any label "block bogon IPv6 networks from WAN" antispoof for em0 # block anything from private networks on interfaces with the option set antispoof for $WAN block in log quick on $WAN from 10.0.0.0/8 to any label "Block private networks from WAN block 10/8" block in log quick on $WAN from 127.0.0.0/8 to any label "Block private networks from WAN block 127/8" block in log quick on $WAN from 100.64.0.0/10 to any label "Block private networks from WAN block 100.64/10" block in log quick on $WAN from 172.16.0.0/12 to any label "Block private networks from WAN block 172.16/12" block in log quick on $WAN from 192.168.0.0/16 to any label "Block private networks from WAN block 192.168/16" block in log quick on $WAN from fc00::/7 to any label "Block ULA networks from WAN block fc00::/7" # allow our DHCPv6 client out to the WAN pass in quick on $WAN proto udp from fe80::/10 port = 546 to fe80::/10 port = 546 label "allow dhcpv6 client in WAN" pass in quick on $WAN proto udp from any port = 547 to any port = 546 label "allow dhcpv6 client in WAN" pass out quick on $WAN proto udp from any port = 546 to any port = 547 label "allow dhcpv6 client out WAN" antispoof for em1 # allow access to DHCP server on LAN pass in quick on $LAN proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server" pass in quick on $LAN proto udp from any port = 68 to 10.254.0.1 port = 67 label "allow access to DHCP server" pass out quick on $LAN proto udp from 10.254.0.1 port = 67 to any port = 68 label "allow access to DHCP server" # allow access to DHCPv6 server on LAN anchor "dhcpv6serverLAN" # We need inet6 icmp for stateless autoconfig and dhcpv6 pass quick on $LAN inet6 proto udp from fe80::/10 to fe80::/10 port = 546 label "allow access to DHCPv6 server" pass quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 546 label "allow access to DHCPv6 server" pass quick on $LAN inet6 proto udp from fe80::/10 to ff02::/16 port = 547 label "allow access to DHCPv6 server" pass quick on $LAN inet6 proto udp from ff02::/16 to fe80::/10 port = 547 label "allow access to DHCPv6 server" pass in quick on $LAN inet6 proto udp from fe80::/10 to port = 546 label "allow access to DHCPv6 server" pass out quick on $LAN inet6 proto udp from port = 547 to fe80::/10 label "allow access to DHCPv6 server" antispoof for em2 # allow access to DHCP server on OPT1 pass in quick on $OPT1 proto udp from any port = 68 to 255.255.255.255 port = 67 label "allow access to DHCP server" pass in quick on $OPT1 proto udp from any port = 68 to 10.255.0.1 port = 67 label "allow access to DHCP server" pass out quick on $OPT1 proto udp from 10.255.0.1 port = 67 to any port = 68 label "allow access to DHCP server" # loopback pass in on $loopback inet all label "pass IPv4 loopback" pass out on $loopback inet all label "pass IPv4 loopback" pass in on $loopback inet6 all label "pass IPv6 loopback" pass out on $loopback inet6 all label "pass IPv6 loopback" # let out anything from the firewall host itself and decrypted IPsec traffic pass out inet all keep state allow-opts label "let out anything IPv4 from firewall host itself" pass out inet6 all keep state allow-opts label "let out anything IPv6 from firewall host itself" # make sure the user cannot lock himself out of the webConfigurator or SSH pass in quick on em1 proto tcp from any to (em1) port { 443 80 22 } keep state label "anti-lockout rule" # NAT Reflection rules pass in inet tagged PFREFLECT keep state label "NAT REFLECT: Allow traffic to localhost" # User-defined rules follow anchor "userrules/*" block out log inet from any to any label "USER_RULE: Default deny outbound traffic" pass out on { em0 } inet from any to any keep state label "USER_RULE: Default permit traffic exiting on WAN" pass log inet proto tcp from any to 10.254.0.50 port 22 flags S/SA keep state label "USER_RULE: Allow ssh to kund1 server (NATed or not NATed)" pass log inet proto tcp from any to 10.255.0.50 port 22 flags S/SA keep state label "USER_RULE: Allow ssh to kund2 server (NATed or not NATed)" pass in quick on $WAN inet from x.y.z.49 to x.y.z.33 keep state label "USER_RULE: Allow administrative traffic" pass in quick on $LAN inet from any to any keep state label "USER_RULE: Outbound traffic is filtered by floating rules" pass in quick on $OPT1 inet from any to any keep state label "USER_RULE: Outbound traffic is filtered by floating rules" # Automatic Pass rules for any delegated IPv6 prefixes through dynamic IPv6 clients # VPN Rules anchor "tftp-proxy/*"</bogonsv6></bogons></bogonsv6></bogons></virusprot></webconfiguratorlockout></sshlockout></snort2c></snort2c></negate_networks></virusprot></snort2c></webconfiguratorlockout></sshlockout>
This is working, with one strange caveat. Observe:
pvz@srv1:~$ ifconfig eth0 eth0 Link encap:Ethernet HWaddr 00:50:56:97:39:88 inet addr:10.254.0.50 Bcast:10.254.0.255 Mask:255.255.255.0 inet6 addr: fe80::250:56ff:fe97:3988/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:129754 errors:0 dropped:0 overruns:0 frame:0 TX packets:22657 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:175569470 (175.5 MB) TX bytes:11916745 (11.9 MB) pvz@srv1:~$ sudo tcpdump -n -v -i eth0 'icmp and host 10.254.0.1' & [1] 7246 pvz@srv1:~$ tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 65535 bytes pvz@srv1:~$ ping -c 1 10.255.0.50 PING 10.255.0.50 (10.255.0.50) 56(84) bytes of data. 16:57:52.030661 IP (tos 0x0, ttl 64, id 59938, offset 0, flags [DF], proto ICMP (1), length 56) 10.254.0.1 > 10.254.0.50: ICMP host 10.255.0.50 unreachable, length 36 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto ICMP (1), length 84) 10.254.0.50 > 10.255.0.50: ICMP echo request, id 7248, seq 1, length 64 From 10.254.0.1 icmp_seq=1 Destination Host Unreachable --- 10.255.0.50 ping statistics --- 1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms pvz@srv1:~$ nc -v 10.255.0.50 12345 16:58:07.514405 IP (tos 0x0, ttl 64, id 19012, offset 0, flags [DF], proto ICMP (1), length 88) 10.254.0.1 > 10.254.0.50: ICMP host 10.255.0.50 unreachable, length 68 IP (tos 0x0, ttl 64, id 51115, offset 0, flags [DF], proto TCP (6), length 60) 10.254.0.50.42934 > 10.255.0.50.12345: Flags [s], cksum 0x3251 (correct), seq 3858768486, win 14600, options [mss 1460,sackOK,TS val 2483661 ecr 0,nop,wscale 4], length 0 nc: connect to 10.255.0.50 port 12345 (tcp) failed: No route to host pvz@srv1:~$ nc -v 10.255.0.50 22 Connection to 10.255.0.50 22 port [tcp/ssh] succeeded! SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1 ^C pvz@srv1:~$ The connections are blocked and passed as expected, but instead of just timing out, there is an ICMP error thrown by the firewall that can give an attacker a clue that the traffic is being blocked. I have made sure I've chosen "block" rather than "reject" in the floating rule that ends up blocking the pattern, as can be seen in /tmp/rules.debug So what gives? I'm sure I can concoct some kind of ugly ugly workaround in which I use an outbound policy to block ICMP host unreachable messages from the firewall, but it really ought not to be neccessary if the system is working as intended. ![lab.png](/public/_imported_attachments_/1/lab.png) ![lab.png_thumb](/public/_imported_attachments_/1/lab.png_thumb)[/s]
-
First, I cannot tell which are floating rules and which are not, but I can tell you that by default floating rules are last matching wins unless you specify the "quick" option.
Also, by default, the first rules in the pre-programmed are the floating block rules. This means they all traffic is blocked unless there is a pass rule. -
First, I cannot tell which are floating rules and which are not, but I can tell you that by default floating rules are last matching wins unless you specify the "quick" option.
Also, by default, the first rules in the pre-programmed are the floating block rules. This means they all traffic is blocked unless there is a pass rule.All inbound traffic is blocked by default. But all outbound traffic is passed by default.
In this specific example, the floating rules are the user-defined rules without the "quick" keyword in them. (In my case, I have no "quick" floating rules.)
-
It is passed only because there is a rule created on LAN. You can change this rule. If you delete this rule, all access to the internet will stop. This is because of the unseen default block rules. Rules create in LAN, WAN and OPTx are all quick rules, this means first matching rule wins. So a block before and allow will always block the traffic. Floating is the opposite unless the quick option is checked.
-
It is passed only because there is a rule created on LAN. You can change this rule. If you delete this rule, all access to the internet will stop. This is because of the unseen default block rules. Rules create in LAN, WAN and OPTx are all quick rules, this means first matching rule wins. So a block before and allow will always block the traffic. Floating is the opposite unless the quick option is checked.
Yes, I know this already. The inbound rules are traversed first. Only if the inbound rules permit the packet to pass, will the outbound rules be traversed. I'm also aware about the difference between quick and non-quick.
This doesn't get us any closer to explaining why I'm getting an ICMP Host Unreachable back from the firewall when I'm doing something I'm not supposed to, rather than the firewall just dropping the packet on the floor, which was the actual subject of this thread.
-
I have just checked on 2.0.1-RELEASE, the same exact problem is present there. Could some friendly mod please move this topic to the Firewalling forum?
-
It turns out that there's some "undocumented behaviour" in which where a packet that has been passed on the incoming side is then blocked on the outgoing side causes pf to emit an ICMP host unreachable message. I've confirmed this on a lot of different systems, including the latest pf upstream (OpenBSD 5.2).
But I was able to work around this issue with an ugly hack. I discovered that by using a policy route to null route the packets instead of just "dropping" them, I could get the behaviour I wanted.
So I just patched the pfsense rule generation code to do just that.
This will probably break about 99 different things, but it seems to work for me in my test environment. If anyone else is having these issues and wants to give it a go, here are my local changes to pfsense (run the patch inside /etc/inc)
--- filter.inc.orig 2012-12-11 02:41:15.000000000 +0100 +++ filter.inc 2012-12-11 02:52:53.000000000 +0100 @@ -2342,6 +2342,18 @@ " label \"NEGATE_ROUTE: Negate policy routing for destination\"\n"; } + + /* ugly hack to convert "block" rules to "pass route-to 0.0.0.0" rules to work around a bug which causes floating rules applied on exit + interfaces to improperly send an ICMP unreachable instead of dropping the packet on the floor. this workaround fixes that. */ + if ($type == "block" && isset($rule['ipprotocol']) && ($rule['ipprotocol'] == "inet" || $rule['ipprotocol'] == "inet6")) { + $aline['type'] = "pass "; + if ($rule['ipprotocol'] == "inet") { + $aline['route'] = " route-to 0.0.0.0 "; + } else { + $aline['route'] = " route-to :: "; + } + } + /* piece together the actual user rule */ $line .= $aline['type'] . $aline['direction'] . $aline['log'] . $aline['quick'] . $aline['interface'] . $aline['reply'] . $aline['route'] . $aline['ipprotocol'] . $aline['prot'] . $aline['src'] . $aline['os'] . $aline['dst'] . --- filter_log.inc.orig 2012-12-11 03:16:15.000000000 +0100 +++ filter_log.inc 2012-12-11 03:15:44.000000000 +0100 @@ -159,6 +159,13 @@ $flent['proto'] = "none"; } + /* check for the hack where we're null routing. */ + list($rulenum, $junk) = explode('/', $rule); + $rule2 = find_rule_by_number($rulenum, $flent['act']); + if (!(strpos($rule2, " route-to 0.0.0.0 ") === FALSE) || !(strpos($rule2, " route-to :: ") === FALSE)) { + $flent['act'] = "block"; + } + /* If there is a src, a dst, and a time, then the line should be usable/good */ if (!((trim($flent['src']) == "") || (trim($flent['dst']) == "") || (trim($flent['time']) == ""))) { return $flent;
This is most emphatically not the "right" way to do it. But it's a practical solution. More practical than digging into the internals of pf. At least for me. :-)
-
While I may have a much simpler setup but I do have a similar concept. I have three interfaces on my pfsense box: wan - lan - opt1. I keep traffic between lan and opt 1 separate entirely, but both are allowed to use the internet. I believe the same is for you with the addition of vlans. Right now I do not permit any traffic but DNS/DHCP requests and traffic to the internet.
-
@pv2b - you've discovered one of the many reasons it's best to block the traffic as close to the source as possible.
Sure an outbound floating rule shortcut sounds convenient, but ultimately it's best to block on the individual tabs (or both, just to be extra sure).
-
I don't agree that it's neccessarilly better to block as close to the source as possible. There's a compelling argument to the opposite in fact: it's generally better to block as close to the target as possible, because otherwise you have to trust all the intervening nodes not to send malicious traffic. That's why we have personal firewalls filtering our incoming traffic instead of relying on everyone else to do egress filtering so no evil packets escape their networks.
Either way the argument for blocking like this isn't convenience, it's security. It's far too easy to screw up when you have to duplicate the same rule on a lot of interfaces.
-
It's far too easy to screw up when you have to duplicate the same rule on a lot of interfaces.
That's never a requirement, use interface groups if you need to do that.
-
I don't agree that it's neccessarilly better to block as close to the source as possible. There's a compelling argument to the opposite in fact: it's generally better to block as close to the target as possible, because otherwise you have to trust all the intervening nodes not to send malicious traffic. That's why we have personal firewalls filtering our incoming traffic instead of relying on everyone else to do egress filtering so no evil packets escape their networks.
Either way the argument for blocking like this isn't convenience, it's security. It's far too easy to screw up when you have to duplicate the same rule on a lot of interfaces.
Not quite. It's better to block as close to the source as possible under your control. Why let traffic any farther into your network than it needs to go? Your point is valid but only when the traffic is sourced remotely, on networks you do not control. When you control the source, there's no reason to let it pass in when you know it shouldn't.
Yes, blocking on the way out is also good but because of the way stateful firewalls work, combining the two philosophies will lead to issues like the one you discovered. You allowed the traffic in one way, and blocked it out another. That is never as good as blocking it before it comes in.
By using aliases, interface groups, etc, you can reduce the likelihood of making an error or over-duplicating.