[SOLVED] Curious Floating Rules Behavior



  • Hi, first off, what I'm about ask is for research purposes only to answer the question "why are the floating rules acting this way?". In other words, i'm just testing the behavior of the rules not asking for other suggestions on how to accomplish the task, I know there are other ways to do what I'm about to bring up. Thank you.

    Scenario: I have LAN clients accessing OpenDNS servers directly. i.e. no DNS resolver or forwarder. LAN clients have the OpenDNS IP addresses directly assigned by the DHCP server.

    Set-up: On the floating rules tab, I opened up DNS ports using 3 different methods. Methods 1 and 2 work, method 3 does not.

    METHOD 1: A single "pass" Floating Rule (This method works.)
              -Both WAN and LAN Interface.
              -Both Incoming and Outgoing
              -Source: Any
              -Destination: Port 53

    Method 2: Two "pass" Floating Rules (This method works.)
              Floating Rule #1
                  -WAN Interface
                  -Both Incoming and Outgoing
                  -Source: Any
                  -Destination: Port 53
              Floating Rule #2
                  -LAN Interface
                  -Both Incoming and Outgoing
                  -Source: Any
                  -Destination: Port 53

    Method 3: Four "pass" Floating Rules (This method does not work.)
              Floating Rule #1
                  -WAN Interface
                  -Incoming
                  -Source: Any
                  -Destination: Port 53
              Floating Rule #2
                  -WAN Interface
                  -Outgoing
                  -Source: Any
                  -Destination: Port 53
            Floating Rule #3
                  -LAN Interface
                  -Incoming
                  -Source: Any
                  -Destination: Port 53
              Floating Rule #4
                  -LAN Interface
                  -Outgoing
                  -Source: Any
                  -Destination: Port 53

    I've tried the rules both with and without the "quick" option. I've also tried the rules in different orders with the same result.

    I've detailed what happens below:

    1.) The LAN Client sends the DNS packet to OpenDNS Servers.
    2.) PFsense sends the DNS packet out the WAN interface.
    3.) OpenDNS sends the DNS response packet back to the WAN interface.
    4.) PFsense sends the DNS response packet addressed to the LAN Client back out the WAN interface (instead of the LAN interface where it belongs).

    Attached is what the packet capture looks like on the WAN and LAN. You will see PFS appears to be sending out the LAN client addressed packet back out the WAN interface.

    If this had something to do with the order of the rules or the quick option, I would expect different steps to fail in this process depending on the order. In addition, I would not expect to see the LAN client addressed packet from OpenDNS to be seen on the WAN interface.

    Does anyone have any thoughts or insight for this behavior?

    ![Unsuccessful WAN Packet Capture.png](/public/imported_attachments/1/Unsuccessful WAN Packet Capture.png)
    ![Unsuccessful WAN Packet Capture.png_thumb](/public/imported_attachments/1/Unsuccessful WAN Packet Capture.png_thumb)
    ![Unsuccessful LAN Packet Capture.png](/public/imported_attachments/1/Unsuccessful LAN Packet Capture.png)
    ![Unsuccessful LAN Packet Capture.png_thumb](/public/imported_attachments/1/Unsuccessful LAN Packet Capture.png_thumb)



  • I wouldn't use a floating rule at all.

    I'd use a PASS rule on my LAN Interface for TCP/UDP 53 as it's cleaner.

    If you want to dig deeper, you can setup logging on the rules and see what's not passing in the Status->System Logs.



  • @Animosity022:

    I wouldn't use a floating rule at all.

    I'd use a PASS rule on my LAN Interface for TCP/UDP 53 as it's cleaner.

    If you want to dig deeper, you can setup logging on the rules and see what's not passing in the Status->System Logs.

    Thank you for the tip about setting up logging. What this revealed was that the INBOUND rules were never getting triggered, despite having incoming packets.

    I tried the following to get the rules to trigger -

    1.) Adding Complimentary Inbound floating rules on WAN and LAN with "Pass" Source Port: 53 and Destination: Any. (Although this seems unnecessary because DNS works just fine without these rules set-up with methods 1 and 2.)

    2.) Removed the OUTBOUND rules, since these are mostly unnecessary as we all know. DNS worked after removing the WAN OUTBOUND rule. The LAN OUTBOUND rule caused no harm on the process. Therefore, it was the WAN OUTBOUND rule that was causing the processing issue.

    3.) Left an OUTBOUND WAN floating rule, but opened the INBOUND ports in both the LAN and WAN firewall rule tabs (not floating rule). (This did not work. Therefore, further confirming it was the OUTBOUND WAN floating rule causing the issue.)

    4.) I retested the same scenario using ports 80, 443 and ANY and had the same results.

    MY CURRENT HYPOTHESIS: Without further input from someone who might understand the back-end processing of these rule better than I. The best uneducated guess I can make at this time is that this is a BUG with, at the very least, my PFSense install. It appears, to me at this time, that anytime a WAN OUTBOUND floating rule is added without including the LAN interface or INBOUND direction, it causes other INBOUND rules with the same port defined in the OUTBOUND floating rule to stop functioning (Keep in mind I have tested this with and without the QUICK option). Therefore, no INBOUND packet processing takes place with the same port defined in a lone WAN OUTBOUND Floating Rule.

    P.S. - I have to mention - this only occurs after I have defined an OUTBOUND WAN floating rule and then RESTARTED PFsense. It does not occur before a restart.

    P.P.S - This is NOT a VM.  I do have a second PFsense instance running in a VM, I will see if this has the same issue and report back.


  • LAYER 8 Netgate

    Very likely not a bug.

    https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

    Create a scenario that does NOT work and post the screenshots of the actual rules you have created (or post your /tmp/rules.debug file) and someone will tell you why it is not working.

    Also explain your method of testing.



  • @Derelict:

    Very likely not a bug.

    https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

    Create a scenario that does NOT work and post the screenshots of the actual rules you have created (or post your /tmp/rules.debug file) and someone will tell you why it is not working.

    Also explain your method of testing.

    You are correct sir it was not a bug. Thank you for pointing me to /temp/rules.debug, that had the nugget I needed to answer my question.

    For anyone who finds this in the future and are curious, below is how the back-end logic of PFSense works when creating floating rules.

    Whenever you create a floating "pass" rule that includes both INBOUND and OUTBOUND, PFSense uses logic called "pass on".

    Whenever you create a floating "pass" rule that includes only INBOUND, PFSense uses logic called "pass in".

    Whenever you create a floating "pass" rule that includes only OUTBOUND, PFSense uses logic called "pass out".
    Note: Make sure you define a source interface other than the interface you are creating an OUTBOUND only floating rule on. Otherwise, INBOUND packets on the same interface will also be checked against the OUTBOUND rule. This means they will be passed back out the same interface they arrived on if they meet the criteria defined in the OUTBOUND rule.

    Thank you Derelict and Animosity022 for guiding me to the rule logic I was seeking.



  • @wussupi83:

    @Derelict:

    Very likely not a bug.

    https://doc.pfsense.org/index.php/Firewall_Rule_Processing_Order

    Create a scenario that does NOT work and post the screenshots of the actual rules you have created (or post your /tmp/rules.debug file) and someone will tell you why it is not working.

    Also explain your method of testing.

    You are correct sir it was not a bug. Thank you for pointing me to /temp/rules.debug, that had the nugget I needed to answer my question.

    For anyone who finds this in the future and are curious, below is how the back-end logic of PFSense works when creating floating rules.

    Whenever you create a floating "pass" rule that includes both INBOUND and OUTBOUND, PFSense uses logic called "pass on".

    Whenever you create a floating "pass" rule that includes only INBOUND, PFSense uses logic called "pass in".

    Whenever you create a floating "pass" rule that includes only OUTBOUND, PFSense uses logic called "pass out".
    Note: Make sure you define a source interface other than the interface you are creating an OUTBOUND only floating rule on. Otherwise, INBOUND packets on the same interface will also be checked against the OUTBOUND rule. This means they will be passed back out the same interface they arrived on if they meet the criteria defined in the OUTBOUND rule.

    Thank you Derelict and Animosity022 for guiding me to the rule logic I was seeking.

    Hello,
    I conducted some extensive testing on my own regarding floating rules. I think pfSense lingo regarding floating rules should change from "Inbound" and "Outbound" to "Inside" and "Outside" (for floating rules only). For example, when you say "Whenever you create a floating "pass" rule that includes only OUTBOUND, PFSense uses logic called "pass out", it leads people to believe that the rule is controlling "OUTBOUND" traffic (packets leaving the interface). In reality it is controlling packets that are "OUTSIDE", relative to the interface(s) selected. I encourage anyone to perform the same testing I did to see the difference yourself. The difference is substantial.



  • LAYER 8 Netgate

    No, it controls traffic in the outbound direction on that interface. It has zero to do with whether the interface is considered to be "inside" or "outside" as is generally referred to in firewall/ASA terms.



  • "pass out" is the exact logic expression written in the /temp/rules.debug file when you create an OUTBOUND only floating point rule.


  • LAYER 8 Netgate

    Right. Pass traffic going out an interface.

    It is usually used to block traffic (or match traffic for shaping, etc) since if the traffic is passed into pfsense in the first place it is passed on the way out unless blocked. It is generally accepted that the best place to block traffic is before it enters the firewall at all. (on the inbound interface).



  • @Derelict:

    No, it controls traffic in the outbound direction on that interface. It has zero to do with whether the interface is considered to be "inside" or "outside" as is generally referred to in firewall/ASA terms.

    The second example in that test sheet confirms pfsense does not behave that way. Anyone can prove it to themselves by simply 1) starting with a factory install of pfsense 2), place two machines on two different interfaces, 3) create a floating rule specifying the LAN only (this is key because you're testing how pfsense processes OUT vs IN  packets), all Ipv4, enabled quick match, and action of drop, 4) on the OPT interface, create a normal rule under the interface tab to pass all traffic ANY to ANY (which ensures pfSense's default policy of drop doesn't interfere with the test). Then simply test all three directions (all, in, out) by passing a packets from the OPT station to the LAN and vice versa.

    When testing the OUT direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they are OUT[side] relative the the LAN.

    When testing the ANY direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they include packets that are OUT[side] relative the the LAN.

    When testing the IN direction on the floating rule, packets originating from the OPT to the LAN station PASS - because they are not OUT[side] relative the the LAN (i.e. the rule simply doesn't apply).

    Remember, the floating rule didn't even specify the OPT at all - thus "OUT[bound]" couldn't possibly be "OUT[bound]". The second test proves this, that the floating rule does not process packets as being "OUT[bound]" but rather OUT[side] (i.e. the direction of the packet relative to the interface that's been selected in the floating rule's config.)




  • @Derelict:

    Right. Pass traffic going out an interface.

    It is usually used to block traffic (or match traffic for shaping, etc) since if the traffic is passed into pfsense in the first place it is passed on the way out unless blocked. It is generally accepted that the best place to block traffic is before it enters the firewall at all. (on the inbound interface).

    Pehaps that's the intent but is not how pfsense behaves.


  • LAYER 8 Global Moderator

    "Remember, the floating rule didn't even specify the OPT at all - thus "OUT[bound]" couldn't possibly be "OUT[bound]"."

    Its not outbound of the opt interface its OUTBOUND of the LAN interface…  Why does this concept cause people so much pain??

    Pfsense has interface - lan.. If the traffic is from the lan toward pfsense then its INBOUND or ingress to that interface... If its into the LAN from direction of pfsense then its outbound from the interface, or egress..  This is how every single interface works on any device be it router or switch..  Please look up ingress and egress in networking terms..

    Attached is simple drawing...

    The red arrows are inbound traffic to the specific interface (ingress), green  is outbound (egress)..

    So if you have PC on lan that sends SYN to IP on OPT..  That syn would be in to the lan nic..  And out of the opt nic.. These are the 2 places where you could put rules..  To do the inbound rule just place the traffic on the lan interface.  If you want to stop traffic from going out the opt, then you would need to put that on floating tab picking out of the opt interface..

    Its always best to block the traffic before it actually enters the firewall.. Why process the packet all the way through pfsense just to stop it from leaving the opt interface.. Just drop it as it tries to enter at the lan..

    if you wanted to save yourself time in creating rules.. And you knew say that you didn't want lan, opt, optX, optY etc.. going out to internet on 53... Then you could put a floating rule on wan interface outbound direction blocking 53.. Now if lan or lop sent traffic it be dropped on the oubound direction of the wan interface.




  • @johnpoz:

    "Remember, the floating rule didn't even specify the OPT at all - thus "OUT[bound]" couldn't possibly be "OUT[bound]"."

    Its not outbound of the opt interface its OUTBOUND of the LAN interface…  Why does this concept cause people so much pain??

    Pfsense has interface - lan.. If the traffic is from the lan toward pfsense then its INBOUND or ingress to that interface... If its into the LAN from direction of pfsense then its outbound from the interface, or egress..  This is how every single interface works on any device be it router or switch..  Please look up ingress and egress in networking terms..

    Attached is simple drawing...

    The red arrows are inbound traffic to the specific interface (ingress), green  is outbound (egress)..

    So if you have PC on lan that sends SYN to IP on OPT..  That syn would be in to the lan nic..  And out of the opt nic.. These are the 2 places where you could put rules..  To do the inbound rule just place the traffic on the lan interface.  If you want to stop traffic from going out the opt, then you would need to put that on floating tab picking out of the opt interface..

    Its always best to block the traffic before it actually enters the firewall.. Why process the packet all the way through pfsense just to stop it from leaving the opt interface.. Just drop it as it tries to enter at the lan..

    if you wanted to save yourself time in creating rules.. And you knew say that you didn't want lan, opt, optX, optY etc.. going out to internet on 53... Then you could put a floating rule on wan interface outbound direction blocking 53.. Now if lan or lop sent traffic it be dropped on the oubound direction of the wan interface.

    This is my lab hanging on my wall in which I conducted these tests. The raspberry pi sits on the OPT1 (circled) and a laptop (unseen in the photo) is the LAN. When packets from the PI (OPT1) were sent to the laptop (LAN), they dropped per the rules I described because the direction they traveled were OUTside relative the LAN (remember the LAN is the only interface selected in the Floating rule for these tests). "OUT" cannot possibly be "OUTbound"




  • @johnpoz:

    The red arrows are inbound traffic to the specific interface (ingress), green  is outbound (egress)..

    Per the drawing you posted, the green arrow on the LAN supports my tests and it's conclusion that packets are treated per the direction of their movement relative to the interface selected. This is the exact opposite of citing the packets as "egressing" per your own drawing. I agree 100% with the drawing you posted, but realize that the green arrow you have for the LAN is NOT "egress" at all - its OUTside (i.e. relative to the interface).

    Specifically, citing "green  is outbound (egress).." is a total contradiction to the drawing you posted. That green arrow is not "OUTbound", it is OUTside relative the the LAN.


  • LAYER 8 Netgate

    When testing the OUT direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they are OUT[side] relative the the LAN.

    When testing the ANY direction on the floating rule, packets originating from the OPT to the LAN station DROP - because they include packets that are OUT[side] relative the the LAN.

    When testing the IN direction on the floating rule, packets originating from the OPT to the LAN station PASS - because they are not OUT[side] relative the the LAN (i.e. the rule simply doesn't apply).

    You are simply not getting it. The reason the packets are dropped in the first two examples is because your floating rule catches the traffic (actually the state creation) as it leaves the LAN interface OUTBOUND.

    The reason the packets are not dropped in the third case is because the state creation is inbound on OPT1 (passed by the any/any rule there) and outbound on LAN. Your floating rule here is for LAN inbound. That would catch states created BY LAN hosts (LAN in), not TO LAN hosts (LAN out).



  • @Derelict:

    The reason the packets are dropped in the first two examples is because your floating rule catches the traffic (actually the state creation) as it leaves the LAN interface OUTBOUND.

    Cannot possibly be correct because the packet didnt "leave" the LAN at all - that was the whole point of the test. They "left" the OPT1 and matched the floating rule because it was OUTside relative to the interface.


  • LAYER 8 Netgate

    Of course they didn't leave LAN. They were blocked by the firewall so the state was never created.

    Let's get some terminology clear:

    inside/outside

    LAN/Trusted –- inside --- FIREWALL --- outside --- Internet/Untrusted

    inbound (ingress) / outbound (egress)

    inbound  --->|
                | Interface
    outbound <--



  • @Derelict:

    Of course they didn't leave LAN. They were blocked by the firewall so the state was never created.

    Let's get some terminology clear:

    inside/outside

    LAN/Trusted –- inside --- FIREWALL --- outside --- Internet/Untrusted

    inbound (ingress) / outbound (egress)

    inbound  --->|
                | Interface
    outbound <--

    The drawing posted by johnpoz is spot on. However, it seems you guys both believe that "OUT" means "OUTbound" (or egress). "OUT" is the direction of packets relative to the interface, its is not "egress" at all.. So in my case, packets sent from the raspberry pi were OUTside relative to the LAN.


  • LAYER 8 Netgate

    OUTSIDE is a location
    OUTBOUND is a direction



  • @Derelict:

    OUTSIDE is a location
    OUTBOUND is a direction

    Whatever the case, "OUT" is neither egress traffic nor is it "OUTbound" traffic. It is traffic that is relative to the interface thats been selected.  I agree with johnpoz's drawing 100%.



  • LAYER 8 Netgate

    Exactly. You seem to confuse ingress/egress with inside/outside. That is only true when you are talking about the WAN interfaces.

    Traffic from LAN hosts INGRESSES the firewall on its way INBOUND into the LAN interface. Reply traffic for those connections EGRESSES the LAN interface on its way back OUTBOUND to the LAN hosts - relative to the LAN interface.

    The only problem here is your failure to properly comprehend these terms in English as they relate to common usage when describing firewall behavior.

    There is nothing at all curious about the floating rule behavior you have described.



  • @Derelict:

    Exactly. You seem to confuse ingress/egress with inside/outside. That is only true when you are talking about the WAN interfaces.

    The drawings and tests I performed are 100% on. The confusion regarding IN,OUT,ANY direction is because of people citing it OUT as OUTbound/egress traffic when it is not.

    @Derelict:

    Traffic from LAN hosts INGRESSES the firewall on its way INBOUND into the LAN interface.

    Agree 100%

    @Derelict:

    Reply traffic for those connections EGRESSES the LAN interface on its way back OUTBOUND to the LAN hosts - relative to the LAN interface.

    Test # 2 was the complete opposite and while I agree with your use of terms, the 'EGRESS" traffic you're referring to has nothing to do with the OUT direction.

    @Derelict:

    The only problem here is your failure to properly comprehend these terms in English as they relate to common usage when describing firewall behavior.

    This is as backwards as the terms being discussed.  EGRESS has nothing to do with OUT under floating rules.

    @Derelict:

    There is nothing at all curious about the floating rule behavior you have described.

    I didn't…


  • LAYER 8 Netgate

    The drawings and tests I performed are 100% on. The confusion regarding IN,OUT,ANY direction is because of people citing it OUT as OUTbound/egress traffic when it is not.

    Yes it is, relative to the interface. That is why you select an INTERFACE and a DIRECTION RELATIVE TO THAT INTERFACE.



  • @Derelict:

    The drawings and tests I performed are 100% on. The confusion regarding IN,OUT,ANY direction is because of people citing it OUT as OUTbound/egress traffic when it is not.

    Yes it is, relative to the interface. That is why you select an INTERFACE and a DIRECTION RELATIVE TO THAT INTERFACE.

    OUT is not "egress/outbound" traffic. Think about what you're saying above. If you agree with me that that OUT,IN,ANY are DIRECTION(s) RELATIVE TO an INTERFACE selected, then you cannot possibly say that "OUT" is egress or outbound without contradicting yourself. "OUT" is not outbound traffic


  • LAYER 8 Netgate

    I am done. Someone else's turn.



  • @Derelict:

    I am done. Someone else's turn.

    I think you have finally seen the difference and might be too proud to admit it. Don't beat yourself up because I confused "OUT" as being associated with OUTbound/egress for years until I finally sat down and went through those tests I posted. I see you and johnpoz have many postings in these forums and its great to have people actively helping one another. Don't get flustered. And dont be too proud to admit you might have learned something new in this discussion.


  • LAYER 8 Netgate

    Sigh.



  • @Derelict:

    Sigh.

    ditto



  • @Derelict:

    Sigh.

    Lots of people have holes in their firewall configs for the very reasons being discussed here."OUT" is the direction of packets relative to the interface.


  • LAYER 8 Netgate

    Now you seem to be equating "egress" with "traveling from the inside to the outside. From the trusted to the untrusted. From the LAN to the WAN/Internet."

    That is not it at all. "ingress" is "INto an interface." WAN or LAN, inside or outside, doesn't matter. It is traffic received by an interface coming into (aka ingressing) the firewall.

    "egress" is "OUT of an interface." WAN or LAN, inside or outside, doesn't matter. It is traffic transmitted by an interface going out of (aka eggressing) the firewall.

    Look at this again - Really, honestly look at it:

    inside/outside

    LAN/Trusted –- inside --- FIREWALL --- outside --- Internet/Untrusted

    inbound (ingress) / outbound (egress)

    inbound  --->|
                | Interface
    outbound <--

    You insist on using nonstandard terms. I have been trying to get on the same terminology for several posts.

    No, I don't need your money. I know I am correct. Use it to buy a dictionary.



  • @Derelict:

    "egress" is "OUT of an interface." WAN or LAN, inside or outside, doesn't matter. It is traffic transmitted by an interface going out of (aka eggressing) the firewall.

    So long as you don't associate the above as having anything to do with the "OUT" direction under floating rules, then I agree. If you're saying that it does and that "OUT" is OUTbound/egress traffic then it's simply not correct. And this confusion keeps perpetuating on these forums because people are posting stuff they dont really understand.


  • LAYER 8 Netgate

    That is exactly what it means in floating rules. That is exactly what your tests showed.

    Maybe I will take that $500. Let's make that 0.0625 bitcoin before you try to pay in zim dollars or something.



  • @Derelict:

    That is exactly what it means in floating rules. That is exactly what your tests showed.

    Maybe I will take that $500.

    So just to be absolutely clear, if you're saying then that "OUT" in floating rules applies to egress/outbound traffic, then I'm all for doing a conference with you. My test #2 showed that OUT is NOT egress outbound traffic - it is the direction of traffic relative to the interface. So if you'r still up fo it citing "Maybe I will take that $500." lets arrange for a time this weekend or next.


  • LAYER 8 Netgate

    See there you go again. Can you not read? We are both saying that OUT is RELATIVE TO THE INTERFACE. You keep bandying about this nonsense about "egress outbound traffic."

    Please define "egress outbound traffic" so everyone knows wtf you are talking about.

    There is a diagram in my signature. Use that as a reference when you describe it.

    It is per interface, nobody has ever said any different.



  • @Derelict:

    See there you go again. Can you not read? We are both saying that OUT is RELATIVE TO THE INTERFACE. You keep bandying about this nonsense about "egress outbound traffic."

    Please define "egress outbound traffic" so everyone knows wtf you are talking about.

    There is a diagram in my signature. Use that as a reference when you describe it.

    It is per interface, nobody has ever said any different.

    Gotta go to work, but Ill respond tonight/tomorrow. I feel like Bill Nye debating Ken Ham with comments like "Can you not read?".


  • LAYER 8 Global Moderator

    "Specifically, citing "green  is outbound (egress).." is a total contradiction to the drawing you posted. That green arrow is not "OUTbound", it is OUTside relative the the LAN. "

    I suggest you research ingress and egress.  These are standard networking 101 terms that if you do not grasp your going to have a hard time of it.

    edit:  Lets try it another way.  A nic or even a port on a switch is like putting a door on a house..  With pfsense or the switch being the house.  If the traffic is going into the interface from the network it is ingress - your going into the house..  if the traffic exiting the nic towards the network then its egress (leaving the house)..


  • LAYER 8 Netgate

    Good God, man. If there was confusion before you have only compounded it.


  • LAYER 8 Global Moderator

    Here maybe this video will help ;)

    Youtube Video



  • LAYER 8 Netgate

    Even that video confuses outside/inside.

    Unless you're talking about outside/inside the firewall.


  • LAYER 8 Global Moderator

    Outside is always the wire..  Connected to the nic/port..  When your talking ingress/egress to a port/nic


Log in to reply