Firewall fails to match traffic with ACK flag
-
I have setup my firewall to pass whitelisted LAN traffic and with a final rule to reject everything else with logging. My expectation was that this would result in result in no log entries for the protocols I have passed, but my experience has been that there often are log entries that show the final rule is blocking traffic that I know I have a rule to pass.
Dec 29 10:22:20 LAN What is this? (1461874052) 192.168.10.123:43400 54.230.5.224:80 TCP:FA
Dec 29 10:22:20 LAN What is this? (1461874052) 192.168.10.123:43400 54.230.5.224:80 TCP:PA
Dec 29 09:31:37 LAN What is this? (1461874052) 192.168.10.124:38009 54.201.136.14:80 TCP:FAThere is a rule to pass all traffic matching a LAN Net source address and Port 80 destination on my LAN interface. My expectation is to NOT see these entries, but the only way to rid myself of them is to duplicate the rule and go to the advanced settings to check for the ACK flag out of just the ACK flag. This workaround rids me of these false alarm log entries, and I have done it for multiple TCP protocols now.
Why does the firewall fail to match the traffic if it has an ACK flag?
-
https://doc.pfsense.org/index.php/Why_do_my_logs_show_%22blocked%22_for_traffic_from_a_legitimate_connection
-
That does not match.
It is not the "Default deny rule IPv4" (i.e. rules 5 and 6 as shown in pfTop) that is triggering this log entry, it is my final user rule (rules 392 and 393). Also, as I said in the post, duplicating the rule and adding the ACK flag matching does match the packet and therefore prevent the final rule for logging the message.
p.s. I inhibit the implicit default block rules from logging on my system, as I trust pfsense.
Manage Firewall Log Log firewall default blocks
Log packets matched from the default block rules in the ruleset
Packets that are blocked by the implicit default block rule will not be logged if this option is unchecked. Per-rule logging options are still respected. -
Yeah, it's your completely redundant and useless final user rule.The reason is exactly the same.
-
The firewall rules only apply to packets that use a proper handshake. The PA packet is associated with a state that no longer exists and is not a syn packet to start a new state. Therefore none of your rules will catch it because it will NEVER be a useful packet. The firewall blocks it with the default block rule that blocks all unknown states and invalid traffic. This falls under invalid traffic.
-
If your seeing out of state traffic you quite possibility have a asymmetrical setup. Or you have devices that are mutihomed, say wifi and wired or multiple wifi and switching connections and not creating a new session and trying to use the old session they had. Say with a cell connection, then switch over to wifi and try and reuse that session..
So to pfsense that is out of state, and yes would be blocked..
Draw up your network, what devices are creating the traffic - what are they using for gateway, do they have multiple connections? Typically out of state traffic is a syn of an asymmetrical configuration.
Looks to be something talking to amazon
Amazon Technologies Inc. AMAZON-2011L (NET-54-224-0-0-1) 54.224.0.0 - 54.239.255.255 -
Respectfully, Doctornotor, no, the reason is not the same.
Rules are processed in order until there is a match, then processing stops. Both my last rule plus the modified rule that includes the TCP Ack flag are processed after the rule cited in the other post. So, whatever the "Default deny rule IPv4" does, it did not match the packet and therefore stop the rule processing.
I can see nothing in the rule definition that L3 criteria should fail to match because of a state table inconsistency. Or, conversely why a modified rule that uses the same criteria and also processes the TCP flags to check for an ACK should then match the packet.
Calling this final rule "redundant and useless" is your opinion. My opinion is that using explicit rules are preferable, especially when there is undocumented behavior that is not fully understood.
I am happy to believe packets inconsistent with the connection state tables are blocked, preferably before any of my rules are processed. But, I have chosen not to be notified of this basic behavior. So unless you have something that you can point to, please stop making arguments that sound reasonable, but are not verifiable.
I realize that there is some magic that occurs to change data put into the pfsense rule editor into something that the nginx pfp-fpm process uses as an input for packet filtering. I think if I could read the input file, it may shed some light on what is happening here, but I don't where this file is located in the filesystem.
Has anyone else looked behind the curtain, so to speak?
-
If your seeing out of state traffic you quite possibility have a asymmetrical setup. Or you have devices that are mutihomed, say wifi and wired or multiple wifi and switching connections and not creating a new session and trying to use the old session they had. Say with a cell connection, then switch over to wifi and try and reuse that session..
So to pfsense that is out of state, and yes would be blocked..
Draw up your network, what devices are creating the traffic - what are they using for gateway, do they have multiple connections? Typically out of state traffic is a syn of an asymmetrical configuration.
Looks to be something talking to amazon
Amazon Technologies Inc. AMAZON-2011L (NET-54-224-0-0-1) 54.224.0.0 - 54.239.255.255Pretty basic network; the LAN is two interconnected TP-Link smart switches, each with a UniFi Access Point. The nodes generating the log entries above were wired, so no switching access points. I had intended to VLAN this configuration, but currently everything is just on the default VLAN.
-
If you want to allow such out of state traffic..
https://doc.pfsense.org/index.php/Asymmetric_Routing_and_Firewall_Rules
I personally would look to see why your seeing it vs trying to allow it, and fix the reason its happening at the root of the problem vs masking an issue by allowing or not logging it.
Out of state could also be caused by no traffic on session for extended amount of time, and the firewall times out and closes the session, but the client just decides after the timeout to send more traffic. Possible device was in standby or something woke up and tried to use its old session that had timed out.
if your not being flooded by it, you could chalk it up to bad client and just ignore the log spam.. Client should be smart enough to create a new session when it doesn't hear back from the old session.
Its possible the client didn't get the fin or rst and left the session open, and still wants to talk after firewall has closed session, etc.
-
If you want to just remove out-of-state log spam, just disable default block logging and create your own "default block" that logs. Then you will only log potentially valid state traffic.
-
I personally would look to see why your seeing it vs trying to allow it, and fix the reason its happening at the root of the problem vs masking an issue by allowing or not logging it.
I do not want anyone to think I favor forwarding this traffic, assuming that it is out-of-order per the connection state. (although I do not see a proof that this is happening, just a theory that might be true)
I have, to the best of my knowledge, setup my firewall to whitelist trusted traffic through it, and then block the rest with logging. The log packets then become part of an investigation process for me. Mostly I see traffic on ports that I have not chosen to trust yet. But these entries are just inconsistent with my rules, so now I have one reason to not trust the firewall; inaccurate rule matching.
-
If you want to just remove out-of-state log spam, just disable default block logging and create your own "default block" that logs. Then you will only log potentially valid state traffic.
Or, as I have explained, create a slightly modified second rule to check the TCP flag for ACK with no logging. On the plus side, the second rule does not permit anything that the first rule should have permitted. But I hate to implement this as it is a workaround to something that the firewall software or the pfsense rule editor should just fix.
-
The default timeout for an ESTABLISHED:ESTABLISHED TCP session is 24 hours of zero activity so something else is probably killing the state.
tcp.first 120s
tcp.opening 30s
tcp.established 86400s
tcp.closing 900s
tcp.finwait 45s
tcp.closed 90sDiagnostics > Command prompt
Most human-readable version of the rules:
cat /tmp/rules.debug
What they are expanded to:
pfctl -vvsr
What you are seeing is just a stateful firewall doing what stateful firewalls do.
-
"But these entries are just inconsistent with my rules,"
Huh?? Who says the rule you put in is even triggering?
Lets state this again, the firewall will and does block out of state traffic! Did you look at the link about allowing (or logging) that sort of traffic.. Post up the settings of your rule that you feel should match this traffic.
Unless you specifically setup a rule not too - pfsense as with any stateful firewall block out of state traffic.. Some of them are nice enough to log it, others just drop it without a log completely, etc.
What I would suggest if you really want to get to the bottom of it. Remove any rules that would allow or not log such traffic any any special block rules you have at the end of your interface tab.. Log your default deny rules.. Prob easy thing to do would be to log all your rules, even the default allow if that is what your doing. And send these logs to syslog, or deal with the log files on pfsense and match up your traffic to where you see the syn, if you ever do?? And then when that traffic gets blocked because of out of state..
-
"But these entries are just inconsistent with my rules,"
Huh?? Who says the rule you put in is even triggering?
Proof of triggering rules is simple; either check for the rule under the "label" pfTop monitor, or if you are really anal, make the rule log matches. I can see that the "IPv4 TCP HTTP" rule has matched 22,393,818 packets and "IPv4 TCP HTTP ack" rule has matched 77 packets.
Lets state this again, the firewall will and does block out of state traffic! Did you look at the link about allowing (or logging) that sort of traffic.. Post up the settings of your rule that you feel should match this traffic.
And I am glad that it does! However, just like the fact that I do not log the 22M packets of the "IPv4 TCP HTTP" rule, I do not want to log invalid state packets. I am guessing that those invalid states packets fall in one of the two "Default deny rule IPv4" rules. It might be one of the other 6 blocking rules with some matching packets, but, not a reasonable guess if the label is to be considered. It also might be blocked even before or after any of these rules are processed. Below is the pfTop of the builtin pfsense rules, filtered of rules with zero packets and rules that pass traffic.
pfTop: Up Rule 1-397/397, View: label
RULE LABEL PKTS BYTES STATES MAX ACTION DIR LOG Q IF PR K
3 Block IPv4 link-local 8 1156 3303748K Block In Log Q
4 Block IPv4 link-local 4 896 3303748K Block In Log Q
5 Default deny rule IPv4 10273 622770 3303748K Block In Log
6 Default deny rule IPv4 2356 177977 3303748K Block Out Log 7 Default deny rule IPv6 84633 14927083 3292120K Block In Log
8 Default deny rule IPv6 83 34220 3292120K Block Out Log
46 Block snort2c hosts 371 32591 3307852K Block Any Log Q
47 Block snort2c hosts 439 47980 3307760K Block Any Log QWhat I would suggest if you really want to get to the bottom of it.
Remove any rules that would allow or not log such traffic any any special block rules you have at the end of your interface tab..
Log your default deny rules..
Prob easy thing to do would be to log all your rules, even the default allow if that is what your doing.
And send these logs to syslog, or deal with the log files on pfsense and match up your traffic to where you see the syn, if you ever do??
And then when that traffic gets blocked because of out of state..I appreciate the suggestion. I already do log the packets that do not match my pass rules, which is how I detected this issue in the first place. Perhaps I am misunderstanding, but are you in favor of logging traffic for every rule?
-
Most human-readable version of the rules:
cat /tmp/rules.debug
What they are expanded to:
pfctl -vvsr
Eureka! Thank you Derelict!
This is what I was looking for to see "behind the curtains" of the pfsense rule editor.
-
"but are you in favor of logging traffic for every rule?"
To find the SYN that opened the state sure.. In troubleshooting why your seeing out of state traffic.. You need to know when/if the connection was even started through pfsense. Without seeing when the session was created, ie the syn you can not tell if its just timeout or the session was closed by client/server and pfsense saw the fin/rst and closed the session and then client continued to send traffic on that now closed session.
Without seeing the full conversation.. your not going to know WHY your seeing the out of state..
A full sniff would be better for sure.. But you can atleast try and get an idea of when the session was started if you log when it was allowed via the syn, etc.
-
To find the SYN that opened the state sure.. In troubleshooting why your seeing out of state traffic.. You need to know when/if the connection was even started through pfsense. Without seeing when the session was created, ie the syn you can not tell if its just timeout or the session was closed by client/server and pfsense saw the fin/rst and closed the session and then client continued to send traffic on that now closed session.
Without seeing the full conversation.. your not going to know WHY your seeing the out of state..
What in the log describes connection states the firewall is tracking, or, how packets match or not those states?
-
Appreciate all the response as it has helped me find at least some knowledge backed with evidence.
First, I have a number of rules entered in pfSense rule, including one that allows IPv4 TCP https traffic to enter the firewall from the LAN (em1) interface. I then duplicated the rule, clicked on Display Advanced, and the checked just the ACK flag out of only the Ack flag.
To zero counters, I executed "pfctl -z" from the Command Prompt found on the Diagnostics menu.
After waiting a few minutes, I executed this command "pfctl -vvsr | grep -A 2 HTTPS"; the output is below.
@91(1482887221) pass in quick on em1 inet proto tcp from 192.168.10.0/24 to any port = https flags S/SA keep state label "USER_RULE: IPv4 TCP HTTPS"
[ Evaluations: 24 Packets: 1006 Bytes: 326304 States: 13 ]
[ Inserted: pid 21873 State Creations: 23 ]
@92(1473019611) pass in quick on em1 inet proto tcp from 192.168.10.0/24 to any port = https flags A/A keep state label "USER_RULE: IPv4 TCP HTTPS Ack"
[ Evaluations: 1 Packets: 2 Bytes: 141 States: 0 ]
[ Inserted: pid 21873 State Creations: 1 ]As you can see from this, there is a difference in the flags.
Where the modified rule shows "flags A/A", the unmodified rule shows "flags S/SA". So, the first rule checks for Syn flag set, and the Ack flag cleared. In the few minutes I waited, there were 1004 packets matched and passed per this rule.
Consequently, any traffic with SYN cleared and ACK set or cleared, or, SYN and the ACK flag set fail to match. Consequently, the second rule matches the traffic with just the ACK flag set will match ACK set and SYN set or cleared, but not HTTPs traffic with both SYN and ACK cleared. In the few minutes I waited, there were 2 packets matched and passed per this rule, or about 0.2% of the HTTPS TCP packets.
Not much, but it adds up as "IPv4 TCP HTTPS" is second only to "IPv4 TCP HTTP" in passing traffic in my configuration. Not sure why, but some TCP protocols generate more of traffic that doesn't have SYN set and ACK cleared.
Incidently, I did notice that there also was an advanced setting for State Type, which defaults to Keep, but also has types of Sloppy, Synproxy, and None. I tried the setting of None on the first rule to see if I could create a single rule for HTTPS that handled everything, but, as you can see, the second rule still matched 2 packets, so the "no state" keeping setting HTTPS was not a solution.
@91(1482887221) pass in quick on em1 inet proto tcp from 192.168.10.0/24 to any port = https flags S/SA no state label "USER_RULE: IPv4 TCP HTTPS"
[ Evaluations: 530 Packets: 529 Bytes: 33456 States: 0 ]
[ Inserted: pid 52915 State Creations: 0 ]
@92(1473019611) pass in quick on em1 inet proto tcp from 192.168.10.0/24 to any port = https flags A/A keep state label "USER_RULE: IPv4 TCP HTTPS Ack"
[ Evaluations: 1 Packets: 2 Bytes: 117 States: 1 ]
[ Inserted: pid 52915 State Creations: 1 ]I take it that from this default, that these other three TCP flag combinations are not valid for normal TCP traffic. So, I blocked with no logging those packets with the following rules. Why bother? So I can monitor how often packets like this show up. Should be less than 1%, but this way I can verify this through either pfTop or the pfctl command.
@154(1483314486) block drop in quick on em1 inet proto tcp from 192.168.10.0/24 to any flags A/A label "USER_RULE: IPv4 TCP ACK flag set"
[ Evaluations: 1815 Packets: 1363 Bytes: 85473 States: 0 ]
[ Inserted: pid 51695 State Creations: 0 ]
@155(1483366880) block drop in quick on em1 inet6 proto tcp from 2001:xxxx:xxxx:xxxx:xxxx::/80 to any flags A/A label "USER_RULE: IPv6 TCP ACK flag set"
[ Evaluations: 12 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: pid 51695 State Creations: 0 ]
@156(1483314555) block drop in quick on em1 inet proto tcp from 192.168.10.0/24 to any flags /SA label "USER_RULE: IPv4 TCP SYN+ACK flags cleared"
[ Evaluations: 7 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: pid 51695 State Creations: 0 ]
@157(1483366970) block drop in quick on em1 inet6 proto tcp from 2001:xxxx:xxxx:xxxx:xxxx::/80 to any flags /SA label "USER_RULE: IPv6 TCP SYN+ACK flags cleared"
[ Evaluations: 0 Packets: 0 Bytes: 0 States: 0 ]
[ Inserted: pid 51695 State Creations: 0 ]
