Firewall rules ignored or overridden?
-
Did you read that doc link?
Then filter on the information you're looking for.
-
I just read that doc link. I have no clustering or load balancing and no multi-WAN. Also, there are no gateways set on the LAN interface settings. If Asymmetric Routing is occurring then it must be on Comcast's backend network since my computer only has a single routable path to the Internet.
As per the manual fix in that document, I have reenabled "State Type" = "sloppy state" and this time added "TCP flags" = "Any flags". Also, I added the same rule on the LAN in the outbound direction using a floating rule. It will take a couple of hours to collect enough traffic to know if this solution works or not. I will report back once I know for sure.
Thanks for the helpful suggestions.
-
The top section. If the other sections are N/A, they're N/A.
-
Not sure what your doing but if this is what you have
ISP -> Modem -> Pfsense -> WorkstationI really don't see how its possible you could have asymmetrical routing. Unless you have wifi and wired connected on your workstation at the same time. you for sure should not have to set any sort of sloppy states on any rules.
Did you read the link about legit traffic blocked in the logs Derelict provided? Can you post up a sample of what your seeing via a screenshot not some ascii art.
You know you can increase the number of logs displayed in the log in the gui, you can also go to the actual files. What exactly are you seeing blocked. Badly written code that doesn't send keep alives and doesn't talk for hours and then tries to continue the conversation could cause out of state blocks. Because the firewall will after a specific time not seeing any traffic on a specific state will close that state. If it then sees traffic it would be blocked for being out of state.
-
I understand that legitimate traffic will be blocked and usually I see that in the logs as a few orphan packets coming in when a session has ended or expired. What I don't expect to see is over 50 blocked packets every 11min completely filling up my logs and making logging useless. Either these packets should be dropped silently or better yet the appropriate rule should be identified to pass the traffic through. I have included a screenshot of the System Log Firewall page. Due to the length of the scrollable page it took three images. Also, I have redacted my public IP for privacy.
-
Lots of blocks outbound on WAN? What did you do in floating rules?
-
According to the logs, the rules causing the blocks are as follows:
The rule that triggered this action is:
@8(1000000106) block drop out log inet6 all label "Default deny rule IPv6"AND
The rule that triggered this action is:
@5(1000000103) block drop in log inet all label "Default deny rule IPv4"There are no floating rules except for some VOIP QoS rules created by the Traffic Shaper wizard which are completely unrelated. I removed the previous floating rule because I think we can all agree that this is not an Asymmetric Routing issue. The previous rule I created was the one listed in the document that was linked to which said to create a floating rule on the LAN for outbound traffic for IPv4 TCP 993 traffic with ANY flags and Sloppy State. I have also reverted the pass rule on the LAN for TCP 993 so that the advanced option ANY flags is removed and Sloppy State is back to keep.
-
If you think those are log spam, go to Status > System Logs, Settings tab and uncheck Log firewall default blocks.
-
While I agree that disabling logging of the default rule would remove these packets from the log, that doesn't really help. I would like to be able to see what packets are being blocked by the firewall and hiding all blocked packets just to get rid of the ones from port 993 is overkill. If the solution is to hide these packets from the log instead of passing them through the firewall, then I would want to create a more specific block rule that only matched these packets with no logging so that I could continue to see packets blocked by the default rule.
-
Your IMAP client is trying to use a connection after the state has been deleted for inactivity or it is otherwise misbehaving.
What isn't working other than the logs you are seeing?
The default timeout for an established TCP state is:
tcp.established 86400s
One whole day. With zero activity.
Not sure how long you think the states should be kept around but feel free to adjust that.
-
I'm fine with 1day but I suspect that sessions are closed after 1hr or less.
Where do you see that setting? I do not see an entry for tcp.established in /tmp/rules.debug. Also, is that the default setting when "Firewall Optimization Options" = Normal? Or is that for a different setting like conservative?
-
That is the setting for Normal.
This is an existing IMAP state for my mail client:
igb0_vlan223 tcp 192.0.2.96:143 <- 192.168.223.6:52466 ESTABLISHED:ESTABLISHED
[3857059653 + 131008] wscale 1 [3105376859 + 66608] wscale 5
age 94:39:32, expires in 23:59:54, 64046:44625 pkts, 11286281:32760196 bytes, rule 283
id: 00000000585fc42a creatorid: 5297d028
igb1 tcp 192.51.100.226:13887 (192.168.223.6:52466) -> 192.0.2.96:143 ESTABLISHED:ESTABLISHED
[3105376859 + 66608] wscale 5 [3857059653 + 131008] wscale 1
age 94:39:32, expires in 23:59:54, 64046:44625 pkts, 11286281:32760196 bytes, rule 211You can get the same thing with:
pfctl -vvss | grep -a2 mailserverip:993
Existing state timeouts:
pfctl -st
-
Immediately after seeing blocked packets in the log, I search for the server that was blocked using the command you provided.
pfctl -vvss | grep -a2 207.46.10.10:993igb1 tcp 207.46.10.10:993 <- 192.168.1.40:65410 TIME_WAIT:TIME_WAIT
[1970684899 + 130688] wscale 0 [2546679215 + 5346] wscale 5
age 00:03:58, expires in 00:00:10, 24:19 pkts, 2348:5296 bytes, rule 143
id: 010000005822dc19 creatorid: fe76c013
igb0 tcp X.X.X.X:46700 (192.168.1.40:65410) -> 207.46.10.10:993 TIME_WAIT:TIME_WAIT
[2546679215 + 5346] wscale 5 [1970684899 + 130688] wscale 0
age 00:03:58, expires in 00:00:10, 24:19 pkts, 2348:5296 bytes, rule 88
id: 010000005822dc1a creatorid: fe76c013
igb1 tcp 207.46.10.10:993 <- 192.168.1.40:65412 TIME_WAIT:TIME_WAIT
[3526826672 + 130731] wscale 0 [4281023330 + 5256] wscale 5
age 00:03:58, expires in 00:00:08, 21:16 pkts, 2102:4714 bytes, rule 143
id: 010000005822dc1b creatorid: fe76c013
igb0 tcp X.X.X.X:39701 (192.168.1.40:65412) -> 207.46.10.10:993 TIME_WAIT:TIME_WAIT
[4281023330 + 5256] wscale 5 [3526826672 + 130731] wscale 0
age 00:03:58, expires in 00:00:08, 21:16 pkts, 2102:4714 bytes, rule 88
–
age 00:07:20, expires in 23:52:41, 9:9 pkts, 1607:1079 bytes, rule 87
id: 0000000058236060 creatorid: fe76c013
igb1 tcp 207.46.10.10:993 <- 192.168.1.40:65411 TIME_WAIT:TIME_WAIT
[3710722489 + 129846] wscale 0 [766408712 + 5526] wscale 5
age 00:03:58, expires in 00:00:07, 27:22 pkts, 2684:6934 bytes, rule 143
id: 00000000582360ae creatorid: fe76c013
igb0 tcp X.X.X.X:13185 (192.168.1.40:65411) -> 207.46.10.10:993 TIME_WAIT:TIME_WAIT
[766408712 + 5526] wscale 5 [3710722489 + 129846] wscale 0
age 00:03:58, expires in 00:00:07, 27:22 pkts, 2684:6934 bytes, rule 88
id: 00000000582360af creatorid: fe76c013
igb1 tcp 207.46.10.10:993 <- 192.168.1.40:65413 TIME_WAIT:TIME_WAIT
[2772107354 + 130731] wscale 0 [1219956310 + 5256] wscale 5
age 00:03:58, expires in 00:00:11, 21:16 pkts, 2102:4714 bytes, rule 143
id: 00000000582360b0 creatorid: fe76c013
igb0 tcp X.X.X.X:49628 (192.168.1.40:65413) -> 207.46.10.10:993 TIME_WAIT:TIME_WAIT
[1219956310 + 5256] wscale 5 [2772107354 + 130731] wscale 0
age 00:03:58, expires in 00:00:11, 21:16 pkts, 2102:4714 bytes, rule 88
–
age 00:02:10, expires in 00:00:07, 11:10 pkts, 1193:1059 bytes, rule 88
id: 00000000582360d3 creatorid: fe76c013
igb1 tcp 207.46.10.10:993 <- 192.168.1.40:65417 ESTABLISHED:ESTABLISHED
[3804055570 + 130731] wscale 0 [4282722200 + 5256] wscale 5
age 00:02:02, expires in 23:58:01, 21:16 pkts, 2102:4726 bytes, rule 143
id: 00000000582360d6 creatorid: fe76c013
igb0 tcp X.X.X.X:31046 (192.168.1.40:65417) -> 207.46.10.10:993 ESTABLISHED:ESTABLISHED
[4282722200 + 5256] wscale 5 [3804055570 + 130731] wscale 0
age 00:02:02, expires in 23:58:01, 21:16 pkts, 2102:4726 bytes, rule 88The established connections have the expected 24hr session time. The waiting connections have about 4min. Several minutes later if I check the state again with the command, all of the sessions are closed except for a few sessions about to time out.
igb1 tcp 207.46.10.10:993 <- 192.168.1.40:65417 TIME_WAIT:TIME_WAIT
[3804055570 + 130731] wscale 0 [4282722200 + 5256] wscale 5
age 00:03:59, expires in 00:00:05, 21:17 pkts, 2102:4766 bytes, rule 143
id: 00000000582360d6 creatorid: fe76c013
igb0 tcp X.X.X.X:31046 (192.168.1.40:65417) -> 207.46.10.10:993 TIME_WAIT:TIME_WAIT
[4282722200 + 5256] wscale 5 [3804055570 + 130731] wscale 0
age 00:03:59, expires in 00:00:05, 21:17 pkts, 2102:4766 bytes, rule 88 -
There is no explanation but that your client or the server is closing them.
You will have to packet capture to see what's actually happening.
-
There only two reason states get closed short of a reboot or someone killing them via PFSense management or custom package. Time out or a packet that tells the connection to end, RST or FIN. It seems the timeout is not the case. I could guess at what the issue is, but I wouldn't want to be wrong.
-
After pouring over the logs I can now confirm that it is not just one MUA client or host that is exhibiting this issue.
Device 1: OS X 10.12.1 using Thunderbird 45.4.0 for the mail client connecting to several IMAP servers including gmail.
Device 2: iOS 10.1.1 using the native Mail application for the client connecting to several IMAP servers including gmail.The host OS's are different, the mail clients are different, and the mail servers are different. The only common denominator for these blocked packets are the pfsense firewall.
I'm not sure how the packet capture is providing insight into why pfsense is blocking these packets. I see TCP 993 connections being opened and closed. But only pfsense knows why it decided to drop some of the packets. Unfortunately pfsense only logs that the "Default deny rule IPv6" (or IPv4) rule was applied. That doesn't make it clear why the explicit pass rules for TCP 993 aren't being applied and I'm not sure what I'm looking for in the packet captures to tell me that. Perhaps there is more aggressive logging that can be enabled?
As for blaming something outside of pfsense, it seems unlikely now that I have two different devices with the same issue. If we are saying the issue is not pfsense, then that means that two separate mail servers such as google's gmail and microsoft's office365 both have the same protocol flaw showing up in their TCP sessions. Or that both Thunderbird and iOS Mail with two different code bases have the same flaw. Similarly that could mean both OS X and iOS having the same networking flaw, however I will grant that this last theory might be possible since OS X and iOS do share some code. It is far more likely though that the one firewall filtering all of the traffic for these two stacks has some aggressive setting or edge case bug that is dropping packets.
Any suggestions are appreciated.
-
No, only the client and server know why the connection is closed. All the firewall cares about is that the two parties on either side decided to close the connection.
pfSense is only logging the firewall exceptions because the client and server decided to close the connection then one of them decided to try to pass more traffic over the closed state.
Packet capture. Who is closing the connection?
-
" I see TCP 993 connections being opened and closed."
Post up this sniff.. As Derelict stated if client or server is sending fin?? Did the other ack with a fin as well.. Fin does not mean the connection has to be closed, it is just stating hey I am done, are you done too?
You really need to see the sniff of the conversation to figure out what is going on. If you have conversation being closed and then still trying to talk on it that is a problem. Because yes when the firewall see's that devices closed their conversation he will remove the state, and until his sees a new conversation with syn, those packets would be out of state and dropped.
-
Okay, I've made some progress but the issue is still not resolved.
From the pfsense terminal:
telnet imap.gmail.com 993; //Successful connection
telnet imap-mail.outlook.com 993; //Successful connection
telnet imap.mail.yahoo.com 993; //Successful connection
telent imap.comcast.net 993; //Successful connection
telnet imap.aol.com 993; //Successful connectionFrom any workstation:
telnet imap.gmail.com 993; //Successful connection
telnet imap-mail.outlook.com 993; //Connection fails
telnet imap.mail.yahoo.com 993; //Connection fails
telent imap.comcast.net 993; //Successful connection
telnet imap.aol.com 993; //Connection failsFor IMAP servers that fail, a packet capture on the LAN shows the following:
1. Initial Syn packet from the workstation
2. Followed by a tcp retransmission to the same IMAP server 1sec later
3. Every 1sec another retransmission occursIt looks like pfsense silently drops a packet so the connection never succeeds. The strange thing is that this doesn't occur for all IMAP servers.
-
What magical packages like pfblocker, snort, suricata, squid, and squidguard are you running?
-
So your saying the packet never goes out the wan? But you see it on the lan of pfsense..
-
No relevant packages installed. Just Status_Traffic_Totals for bandwidth stats.
I just performed a packet capture from the WAN interface and I am seeing tcp retransmissions from the IMAP server to m public IP. It looks like only the SYN packet makes it out. After that the SYN/ACK gets dropped so both sides keep retransmitting. The workstation sends SYNs and the IMAP server sends SYN/ACKs.
Also, pfsense logs the blocked packet sometimes. When it does, it is listed as:
The rule that triggered this action is:
@5(1000000103) block drop in log inet all label "Default deny rule IPv4"My LAN Firewall rule to pass IPv4+6 TCP out to port 993 should implicitly let this traffic back in. It is letting the SYN out but the implied pass for the SYN/ACK back in is not being respected.
-
Note that I did have packages installed in the past however I have since removed them and they reported that the removal was successful. So unless package removal is totally broken, then there should be no package filtering. This should be purely a pfsense filtering issue.
-
Get the IP address of the server and while all that is going on, what states do you have in Diagnostics > States?. You can filter on the IMAP server IP address there. That will catch both WAN and LAN states.
No, packages removed should be OK.
-
See image.
-
If pfsense does not see the syn,ack back the state would never go into established and would time out in by default 30 seconds. Then packets after that would be dropped by the default rule because there would be no state.
It could be less than 30 sec if you messed with the setting on the firewall..
"After that the SYN/ACK gets dropped so both sides keep retransmitting."
Gets dropped where? Your saying you see this syn,ack on your wan sniff??
-
Well, there are your firewall states.
All looks fine.I didn't look far enough over - you guys and your large screen captures… So what are the exact blocked packets you are seeing on WAN?This all just works. Not sure what you did to break it.
If you feel like it do a Diagnostics > Command Prompt and execute cat /tmp/rules.debug and paste the results in a PM.
-
I'm looking through rules.debug and near the top I see entries that concern me. Specifically the following:
#Snort tables
table <snort2c>table <virusprot>As I mentioned previously, I initially installed some ports but subsequently removed them and success was reported. Would successful removal of the relevant ports also remove these artifacts or are these inactive entries in the ruleset which have accumulated from the previous ports but which now have no effect?</virusprot></snort2c> -
Those are always there. I don't think that's it.
From the PM'd pcap it looks like IPv6 is working and IPv4 is broken. That is probably the difference between the different IMAP servers.
-
Yeah. the presence of AAAA records corresponds to the servers you report as working. The ones that fail do not have AAAA records.
So un-do whatever you did for IPv4 and make it look like IPv6 and you should be GTG. :)
-
Not so easy.
Floating Rules:
All rules here are pass or match and unrelated.WAN Rules:
All rules here are pass except for pfsense's block RFC1918 and block bogons.LAN Rules:
All rules here are pass except for NETBIOS on my network.
Also, the rule allowing traffic bound for 993 is an IPv4+6 rule so if it is passing v6 traffic then presumably it is passing v4 traffic as well.Unless there are some other hidden rules then that is everything.
-
Yeah looks like you have ipv6 working, but don't see the start of that conversation. But looks like all your ipv4 is borked..
Where exactly did you sniff this?? Looks like syn,ack is sent from public to a private.. But no answer??
-
Not sure what to tell you, man. You screwed the pooch somewhere along the way.
Send your /tmp/rules.debug
-
That is a microsoft IP.. Clearly its sending a syn,ack to private IP address. So that was on the LAN of pfsense?? Where was this sniff taken?
-
You don't have like LAN and WAN on the same dumb switch or something do you? Something's not right there. It's like that capture contains both pre-nat and post-nat replies.
-
So yeah this is odd.. Looks like wan sniff showing your public IP sending syn to the public IP. And you see the syn,ack back. And then you see the syn,ack sent on to the rfc1918 address?? But then you see a retrans of the syn back to the server from your public..
So the client never saw the syn,ack so its resending syn.. But if we are seeing the lan side of this - where was the syn from the client?
Need to understand where you sniff this at.. Something is not right here.. If we were seeing both wan and lan sides of the sniff.. Then we should of seen the incoming syn to to the pfsense lan, and then it going out the public, then the syn,ack coming back, and then it going out the lan, etc.. But where is the original syn from the 192.168.1.40 addess if that is the case??
You got some sort of asymmetrical issue going on?? Where the syn from the client is coming into pfsense, but its sending the traffic back to the client on wrong interface? But where exactly are you sniffing that seeing both wan and lan traffic?
-
Sorry for the delay.
The issue where I could not telnet to port 993 was caused by a Floating Pass rule allowing outbound traffic to TCP 993 in an attempt to solve the original issue. It is counter intuitive that a pass rule would cause traffic to get blocked however it looks like the floating rule prevented the TCP session which was originally allowed by the LAN rule passing TCP 993.
Although I can now telnet successfully to mailservers on port 993, the original issue still persists. See screenshot. I am trying to come up with an appropriate packet capture that I can provide to show this however I am having trouble sanitizing the capture to not reveal private information.
-
What is you want to pull out of the capture, or what do you want to change in like IP?
Normally best thing is to capture only want you want to share.. Changing the IP of is quiet easy with say https://www.tracewrangler.com/ or bit-twist.
As to your original issue.. All of those blocks you list are out of state.. FA, RA… To what your rules are doing? I don't see how a floating allow would prevent a lan allow? If you have floating and lan rules that do the same thing?? Its possible doing such a thing could lead you to a state of something being closed and then being seen on as out of state??
Really would need to fully understand your flow of traffic, making sure you don't have loops, and your firewall rules. If your PM to me was to say your sniff on the lan.. Then how were you seeing packets with wan address?? You got something clearly borked that is for sure..
A lan sniff should not show you wan traffic.. But in that sniff see SYN from public IP to your public dest.. Its like maybe you have some sort of issue where your running your wan over your lan layer 2? But yeah if pfsense sense is seeing a FIN to close the state, and then sees another FIN to close the state it already closed then it would be out of state and listed as a block..
This seems like what is happening. Your sniff if that was on the LAN of pfsense or some other device or port on your LAN.. It should be impossible to see your public IPs.. But they were in there along with rfc1918 address of 192.168.1.40??