[SOLVED] Curious Floating Rules Behavior

[wussupi83]

Hi, first off, what I'm about ask is for research purposes only to answer the question "why are the floating rules acting this way?". In other words, i'm just testing the behavior of the rules not asking for other suggestions on how to accomplish the task, I know there are other ways to do what I'm about to bring up. Thank you.

Scenario: I have LAN clients accessing OpenDNS servers directly. i.e. no DNS resolver or forwarder. LAN clients have the OpenDNS IP addresses directly assigned by the DHCP server.

Set-up: On the floating rules tab, I opened up DNS ports using 3 different methods. Methods 1 and 2 work, method 3 does not.

METHOD 1: A single "pass" Floating Rule (This method works.)
          -Both WAN and LAN Interface.
          -Both Incoming and Outgoing
          -Source: Any
          -Destination: Port 53

Method 2: Two "pass" Floating Rules (This method works.)
          Floating Rule #1
              -WAN Interface
              -Both Incoming and Outgoing
              -Source: Any
              -Destination: Port 53
          Floating Rule #2
              -LAN Interface
              -Both Incoming and Outgoing
              -Source: Any
              -Destination: Port 53

Method 3: Four "pass" Floating Rules (This method does not work.)
          Floating Rule #1
              -WAN Interface
              -Incoming
              -Source: Any
              -Destination: Port 53
          Floating Rule #2
              -WAN Interface
              -Outgoing
              -Source: Any
              -Destination: Port 53
        Floating Rule #3
              -LAN Interface
              -Incoming
              -Source: Any
              -Destination: Port 53
          Floating Rule #4
              -LAN Interface
              -Outgoing
              -Source: Any
              -Destination: Port 53

I've tried the rules both with and without the "quick" option. I've also tried the rules in different orders with the same result.

I've detailed what happens below:

1.) The LAN Client sends the DNS packet to OpenDNS Servers.
2.) PFsense sends the DNS packet out the WAN interface.
3.) OpenDNS sends the DNS response packet back to the WAN interface.
4.) PFsense sends the DNS response packet addressed to the LAN Client back out the WAN interface (instead of the LAN interface where it belongs).

Attached is what the packet capture looks like on the WAN and LAN. You will see PFS appears to be sending out the LAN client addressed packet back out the WAN interface.

If this had something to do with the order of the rules or the quick option, I would expect different steps to fail in this process depending on the order. In addition, I would not expect to see the LAN client addressed packet from OpenDNS to be seen on the WAN interface.

Does anyone have any thoughts or insight for this behavior?


[Animosity022]

I wouldn't use a floating rule at all.

I'd use a PASS rule on my LAN Interface for TCP/UDP 53 as it's cleaner.

If you want to dig deeper, you can setup logging on the rules and see what's not passing in the Status->System Logs.

[wussupi83]

@Animosity022:

I wouldn't use a floating rule at all.

I'd use a PASS rule on my LAN Interface for TCP/UDP 53 as it's cleaner.

If you want to dig deeper, you can setup logging on the rules and see what's not passing in the Status->System Logs.

Thank you for the tip about setting up logging. What this revealed was that the INBOUND rules were never getting triggered, despite having incoming packets.

I tried the following to get the rules to trigger -

1.) Adding Complimentary Inbound floating rules on WAN and LAN with "Pass" Source Port: 53 and Destination: Any. (Although this seems unnecessary because DNS works just fine without these rules set-up with methods 1 and 2.)

2.) Removed the OUTBOUND rules, since these are mostly unnecessary as we all know. DNS worked after removing the WAN OUTBOUND rule. The LAN OUTBOUND rule caused no harm on the process. Therefore, it was the WAN OUTBOUND rule that was causing the processing issue.

3.) Left an OUTBOUND WAN floating rule, but opened the INBOUND ports in both the LAN and WAN firewall rule tabs (not floating rule). (This did not work. Therefore, further confirming it was the OUTBOUND WAN floating rule causing the issue.)

4.) I retested the same scenario using ports 80, 443 and ANY and had the same results.

MY CURRENT HYPOTHESIS: Without further input from someone who might understand the back-end processing of these rule better than I. The best uneducated guess I can make at this time is that this is a BUG with, at the very least, my PFSense install. It appears, to me at this time, that anytime a WAN OUTBOUND floating rule is added without including the LAN interface or INBOUND direction, it causes other INBOUND rules with the same port defined in the OUTBOUND floating rule to stop functioning (Keep in mind I have tested this with and without the QUICK option). Therefore, no INBOUND packet processing takes place with the same port defined in a lone WAN OUTBOUND Floating Rule.

P.S. - I have to mention - this only occurs after I have defined an OUTBOUND WAN floating rule and then RESTARTED PFsense. It does not occur before a restart.

P.P.S - This is NOT a VM.  I do have a second PFsense instance running in a VM, I will see if this has the same issue and report back.

[Derelict]

Very likely not a bug.

https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

Create a scenario that does NOT work and post the screenshots of the actual rules you have created (or post your /tmp/rules.debug file) and someone will tell you why it is not working.

Also explain your method of testing.

[wussupi83]

@Derelict:

Very likely not a bug.

https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

Create a scenario that does NOT work and post the screenshots of the actual rules you have created (or post your /tmp/rules.debug file) and someone will tell you why it is not working.

Also explain your method of testing.

You are correct sir it was not a bug. Thank you for pointing me to /temp/rules.debug, that had the nugget I needed to answer my question.

For anyone who finds this in the future and are curious, below is how the back-end logic of PFSense works when creating floating rules.

Whenever you create a floating "pass" rule that includes both INBOUND and OUTBOUND, PFSense uses logic called "pass on".

Whenever you create a floating "pass" rule that includes only INBOUND, PFSense uses logic called "pass in".

Whenever you create a floating "pass" rule that includes only OUTBOUND, PFSense uses logic called "pass out".
Note: Make sure you define a source interface other than the interface you are creating an OUTBOUND only floating rule on. Otherwise, INBOUND packets on the same interface will also be checked against the OUTBOUND rule. This means they will be passed back out the same interface they arrived on if they meet the criteria defined in the OUTBOUND rule.

Thank you Derelict and Animosity022 for guiding me to the rule logic I was seeking.

[Kryptos1]

@wussupi83:

@Derelict:

Very likely not a bug.

https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

Create a scenario that does NOT work and post the screenshots of the actual rules you have created (or post your /tmp/rules.debug file) and someone will tell you why it is not working.

Also explain your method of testing.

You are correct sir it was not a bug. Thank you for pointing me to /temp/rules.debug, that had the nugget I needed to answer my question.

For anyone who finds this in the future and are curious, below is how the back-end logic of PFSense works when creating floating rules.

Whenever you create a floating "pass" rule that includes both INBOUND and OUTBOUND, PFSense uses logic called "pass on".

Whenever you create a floating "pass" rule that includes only INBOUND, PFSense uses logic called "pass in".

Whenever you create a floating "pass" rule that includes only OUTBOUND, PFSense uses logic called "pass out".
Note: Make sure you define a source interface other than the interface you are creating an OUTBOUND only floating rule on. Otherwise, INBOUND packets on the same interface will also be checked against the OUTBOUND rule. This means they will be passed back out the same interface they arrived on if they meet the criteria defined in the OUTBOUND rule.

Thank you Derelict and Animosity022 for guiding me to the rule logic I was seeking.

Hello,
I conducted some extensive testing on my own regarding floating rules. I think pfSense lingo regarding floating rules should change from "Inbound" and "Outbound" to "Inside" and "Outside" (for floating rules only). For example, when you say "Whenever you create a floating "pass" rule that includes only OUTBOUND, PFSense uses logic called "pass out", it leads people to believe that the rule is controlling "OUTBOUND" traffic (packets leaving the interface). In reality it is controlling packets that are "OUTSIDE", relative to the interface(s) selected. I encourage anyone to perform the same testing I did to see the difference yourself. The difference is substantial.

[Derelict]

No, it controls traffic in the outbound direction on that interface. It has zero to do with whether the interface is considered to be "inside" or "outside" as is generally referred to in firewall/ASA terms.

[wussupi83]

"pass out" is the exact logic expression written in the /temp/rules.debug file when you create an OUTBOUND only floating point rule.

[Derelict]

Right. Pass traffic going out an interface.

It is usually used to block traffic (or match traffic for shaping, etc) since if the traffic is passed into pfsense in the first place it is passed on the way out unless blocked. It is generally accepted that the best place to block traffic is before it enters the firewall at all. (on the inbound interface).

[Kryptos1]

@Derelict:

No, it controls traffic in the outbound direction on that interface. It has zero to do with whether the interface is considered to be "inside" or "outside" as is generally referred to in firewall/ASA terms.

The second example in that test sheet confirms pfsense does not behave that way. Anyone can prove it to themselves by simply 1) starting with a factory install of pfsense 2), place two machines on two different interfaces, 3) create a floating rule specifying the LAN only (this is key because you're testing how pfsense processes OUT vs IN  packets), all Ipv4, enabled quick match, and action of drop, 4) on the OPT interface, create a normal rule under the interface tab to pass all traffic ANY to ANY (which ensures pfSense's default policy of drop doesn't interfere with the test). Then simply test all three directions (all, in, out) by passing a packets from the OPT station to the LAN and vice versa.

When testing the OUT direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they are OUT[side] relative the the LAN.

When testing the ANY direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they include packets that are OUT[side] relative the the LAN.

When testing the IN direction on the floating rule, packets originating from the OPT to the LAN station PASS - because they are not OUT[side] relative the the LAN (i.e. the rule simply doesn't apply).

Remember, the floating rule didn't even specify the OPT at all - thus "OUT[bound]" couldn't possibly be "OUT[bound]". The second test proves this, that the floating rule does not process packets as being "OUT[bound]" but rather OUT[side] (i.e. the direction of the packet relative to the interface that's been selected in the floating rule's config.)

[Kryptos1]

@Derelict:

Right. Pass traffic going out an interface.

It is usually used to block traffic (or match traffic for shaping, etc) since if the traffic is passed into pfsense in the first place it is passed on the way out unless blocked. It is generally accepted that the best place to block traffic is before it enters the firewall at all. (on the inbound interface).

Pehaps that's the intent but is not how pfsense behaves.

[johnpoz]

"Remember, the floating rule didn't even specify the OPT at all - thus "OUT[bound]" couldn't possibly be "OUT[bound]"."

Its not outbound of the opt interface its OUTBOUND of the LAN interface…  Why does this concept cause people so much pain??

Pfsense has interface - lan.. If the traffic is from the lan toward pfsense then its INBOUND or ingress to that interface... If its into the LAN from direction of pfsense then its outbound from the interface, or egress..  This is how every single interface works on any device be it router or switch..  Please look up ingress and egress in networking terms..

Attached is simple drawing...

The red arrows are inbound traffic to the specific interface (ingress), green  is outbound (egress)..

So if you have PC on lan that sends SYN to IP on OPT..  That syn would be in to the lan nic..  And out of the opt nic.. These are the 2 places where you could put rules..  To do the inbound rule just place the traffic on the lan interface.  If you want to stop traffic from going out the opt, then you would need to put that on floating tab picking out of the opt interface..

Its always best to block the traffic before it actually enters the firewall.. Why process the packet all the way through pfsense just to stop it from leaving the opt interface.. Just drop it as it tries to enter at the lan..

if you wanted to save yourself time in creating rules.. And you knew say that you didn't want lan, opt, optX, optY etc.. going out to internet on 53... Then you could put a floating rule on wan interface outbound direction blocking 53.. Now if lan or lop sent traffic it be dropped on the oubound direction of the wan interface.

[Kryptos1]

@johnpoz:

"Remember, the floating rule didn't even specify the OPT at all - thus "OUT[bound]" couldn't possibly be "OUT[bound]"."

Its not outbound of the opt interface its OUTBOUND of the LAN interface…  Why does this concept cause people so much pain??

Pfsense has interface - lan.. If the traffic is from the lan toward pfsense then its INBOUND or ingress to that interface... If its into the LAN from direction of pfsense then its outbound from the interface, or egress..  This is how every single interface works on any device be it router or switch..  Please look up ingress and egress in networking terms..

Attached is simple drawing...

The red arrows are inbound traffic to the specific interface (ingress), green  is outbound (egress)..

So if you have PC on lan that sends SYN to IP on OPT..  That syn would be in to the lan nic..  And out of the opt nic.. These are the 2 places where you could put rules..  To do the inbound rule just place the traffic on the lan interface.  If you want to stop traffic from going out the opt, then you would need to put that on floating tab picking out of the opt interface..

Its always best to block the traffic before it actually enters the firewall.. Why process the packet all the way through pfsense just to stop it from leaving the opt interface.. Just drop it as it tries to enter at the lan..

if you wanted to save yourself time in creating rules.. And you knew say that you didn't want lan, opt, optX, optY etc.. going out to internet on 53... Then you could put a floating rule on wan interface outbound direction blocking 53.. Now if lan or lop sent traffic it be dropped on the oubound direction of the wan interface.

This is my lab hanging on my wall in which I conducted these tests. The raspberry pi sits on the OPT1 (circled) and a laptop (unseen in the photo) is the LAN. When packets from the PI (OPT1) were sent to the laptop (LAN), they dropped per the rules I described because the direction they traveled were OUTside relative the LAN (remember the LAN is the only interface selected in the Floating rule for these tests). "OUT" cannot possibly be "OUTbound"

[Kryptos1]

@johnpoz:

The red arrows are inbound traffic to the specific interface (ingress), green  is outbound (egress)..

Per the drawing you posted, the green arrow on the LAN supports my tests and it's conclusion that packets are treated per the direction of their movement relative to the interface selected. This is the exact opposite of citing the packets as "egressing" per your own drawing. I agree 100% with the drawing you posted, but realize that the green arrow you have for the LAN is NOT "egress" at all - its OUTside (i.e. relative to the interface).

Specifically, citing "green  is outbound (egress).." is a total contradiction to the drawing you posted. That green arrow is not "OUTbound", it is OUTside relative the the LAN.

[Derelict]

When testing the OUT direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they are OUT[side] relative the the LAN.

When testing the ANY direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they include packets that are OUT[side] relative the the LAN.

When testing the IN direction on the floating rule, packets originating from the OPT to the LAN station PASS - because they are not OUT[side] relative the the LAN (i.e. the rule simply doesn't apply).

You are simply not getting it. The reason the packets are dropped in the first two examples is because your floating rule catches the traffic (actually the state creation) as it leaves the LAN interface OUTBOUND.

The reason the packets are not dropped in the third case is because the state creation is inbound on OPT1 (passed by the any/any rule there) and outbound on LAN. Your floating rule here is for LAN inbound. That would catch states created BY LAN hosts (LAN in), not TO LAN hosts (LAN out).

[Kryptos1]

@Derelict:

The reason the packets are dropped in the first two examples is because your floating rule catches the traffic (actually the state creation) as it leaves the LAN interface OUTBOUND.

Cannot possibly be correct because the packet didnt "leave" the LAN at all - that was the whole point of the test. They "left" the OPT1 and matched the floating rule because it was OUTside relative to the interface.

[Derelict]

Of course they didn't leave LAN. They were blocked by the firewall so the state was never created.

Let's get some terminology clear:

inside/outside

LAN/Trusted –- inside --- FIREWALL --- outside --- Internet/Untrusted

inbound (ingress) / outbound (egress)

inbound  --->|
            | Interface
outbound <--

[Kryptos1]

@Derelict:

Of course they didn't leave LAN. They were blocked by the firewall so the state was never created.

Let's get some terminology clear:

inside/outside

LAN/Trusted –- inside --- FIREWALL --- outside --- Internet/Untrusted

inbound (ingress) / outbound (egress)

inbound  --->|
            | Interface
outbound <--

The drawing posted by johnpoz is spot on. However, it seems you guys both believe that "OUT" means "OUTbound" (or egress). "OUT" is the direction of packets relative to the interface, its is not "egress" at all.. So in my case, packets sent from the raspberry pi were OUTside relative to the LAN.

[Derelict]

OUTSIDE is a location
OUTBOUND is a direction

[Kryptos1]

@Derelict:

OUTSIDE is a location
OUTBOUND is a direction

Whatever the case, "OUT" is neither egress traffic nor is it "OUTbound" traffic. It is traffic that is relative to the interface thats been selected.  I agree with johnpoz's drawing 100%.