Making missing state rule reject instead of block

obroni

Hi,

I have an application with a long tcp timeout (15 mins). I'm seeing a problem where if for any reason (ie FW restart) the TCP state is missing in the PF state table, the application will hang for the 15 minute period at which point it establishes a new fresh connection and everything continues working.

I believe what I want to achieve is for PFsense to return a RST packet if the state is missing, rather than silently drop traffic. Looking through the firewall logs I can see these silent drops appear to be done by a hidden rule. Is there anyway of overriding it or modifying the behaviour to reject instead of block?

Thanks

Nullity

@obroni:

Hi,

I have an application with a long tcp timeout (15 mins). I'm seeing a problem where if for any reason (ie FW restart) the TCP state is missing in the PF state table, the application will hang for the 15 minute period at which point it establishes a new fresh connection and everything continues working.

I believe what I want to achieve is for PFsense to return a RST packet if the state is missing, rather than silently drop traffic. Looking through the firewall logs I can see these silent drops appear to be done by a hidden rule. Is there anyway of overriding it or modifying the behaviour to reject instead of block?

Thanks

Have you tried adding adding an explicit reject rule?

obroni

I have a rule which would do a reject if the rule question fails to match, but I don't believe that this rule is picking up these invalid state packets. Is there some sort of TCP flag I need to set on the rule to pick them up?

NOYB

@obroni:

Looking through the firewall logs I can see these silent drops appear to be done by a hidden rule. Is there anyway of overriding it or modifying the behaviour to reject instead of block?

Don't get hung up on what the logs report. "Rejects are always logged as blocked."

For more details search for "Reject Log", and also see the following.

https://forum.pfsense.org/index.php?topic=50655.msg269967#msg269967
http://doc.pfsense.org/index.php/Firewall_Logs

Derelict

The real solution is to fix the application.

obroni

I'm not sure if the application is at fault here, if it did receive a RST, it would try and re-establish a connection and continue on its way. The application is Ceph, a software defined storage solution, being a LAN based solution, it isn't really expecting a networking device to be silently dropping packets, as it will keep trying client IO to prevent data loss. But perhaps it should time out the TCP session quicker?? I will have a look if there are any options I can tune.

I will have another look at the PFsense rules, but I'm pretty sure the stateless packets were shown as being blocked by some sort of internal rule and not my "catch all" rule down the bottom of the list. If everyone is sure there are no internal/hidden rules that deal with stateless packets like this, then I will have a deeper dig into whats happening.

Harvy66

The firewall rules you add apply only to new states. Out-of-state packets just get blocked. PFSense is meant to only be stateful. Out of state packets are never useful. They should be the exception, not the rule, caused by strange abnormalities. If they happen a lot, that's because something is trying to do something incorrectly.

obroni

Thanks that makes sense. I guess what I need to do is for PFsense to send RST instead of silently blocking, it would sure be nice if this behaviour was able to be configured per interface. ie WAN you probably just want to block, but other interfaces reject.

Otherwise how does the application differentiate between a loss of connectivity, where it should keep trying with the existing session and in the case where the state is lost and it should re-establish a connection from scratch.

Harvy66

@obroni:

Otherwise how does the application differentiate between a loss of connectivity, where it should keep trying with the existing session and in the case where the state is lost and it should re-establish a connection from scratch.

PFSense has a 24 hour timeout by default for established connections. It will only close a state if this idle timeout is reached or it sees a proper FIN or RST packet to indicate that it needs to close the state.

More than likely it's not PFSense killing the connection early, but possibly some of these packets are getting lost and one of the devices closes the connection and the other device doesn't get the packets notifying the connection was killed.

johnpoz

If you create your own reject rule at the bottom vs the default deny. If the rule is out of state that block/reject would trigger before the default deny so you should be able to send a RST.

The problem is that is a bad idea on wan for sure. Because every single packet that did not match an allow rule would be causing firewall to send RST.

You could I guess make the reject rule on the bottom only trigger on your specific IP or set of flags, etc.

But sending RST for the default deny would be a really bad idea..

kpa

@johnpoz:

But sending RST for the default deny would be a really bad idea..

Umm, this is certainly a valid and widely used configuration and it does exactly that for TCP connections and an unreachable ICMP message for UDP on a block rule including the default deny:


...
set block-policy return
...

From pf.conf(5):


set block-policy
           The block-policy option sets the default behaviour for the packet
           block action:

           drop      Packet is silently dropped.
           return    A TCP RST is returned for blocked TCP packets, an ICMP
                     UNREACHABLE is returned for blocked UDP packets, and all
                     other packets are silently dropped.

           For example:

                 set block-policy return

Care to elaborate what you had in mind?

jimp

Valid, perhaps, but I'm not sure about "widely used". Certainly not something I'd ever suggest anyone set.

Any option that causes you to send out data when you receive a request is a potential DDoS reflection abuse target.

Someone could spoof the source and you'd end up sending data toward an attacker's target.

Even if it's not more data than was sent to you, it still hides the source of the traffic.

johnpoz

^ yeah with jimp here, it for sure is not Widely Used.. Most firewalls drop.. Now if internal firewall, you might want to RST stuff. but on the public internet no not a good idea.

Now I have setup reject for a couple of things. I like traceroute to work ;) So linux traceroute uses specific udp ports, for those I have setup reject. So When I do a traceroute through my pfsense on ipv6 I see the pfsense hop. And if you trace to my public ipv4 those ports are also set for reject so I see an an answer.

But rejecting every port that I don't have open would not be a good idea..

Harvy66

@jimp:

Someone could spoof the source and you'd end up sending data toward an ~~attacker~~ attackee.

Reflection attack

johnpoz

While yes reflection attack is possible when you send a RST.. Unless there is amplification in the attack its not very attractive method of attack.

Its more of performance and bandwidth thing for your own firewall. Do you want it doing the extra work of sending RST for every single packet it sees that it does not allow in? There can be quite a bit of noise on the public internet.

So your bandwidth was already used once in seeing the noise, if your not letting it in - its NOISE.. So why increase the volume of the noise by sending RST.. Which is extra work for your firewall and extra bandwidth (how ever small) that you really do not need to send. Now it can be handy depending on the use case - for example my traceroute example. If you just drop those packets then with a udp traceroute you wont see that hop. And maybe you want to see that hop or end target so the extra work and packets you might send with noise to those ports might be worth it to you.

You can argue that hey if someone sends me traffic to port that is closed, vs them sending retrans of those packets and having to wait for their application to say hey nobody answered, might be nice to just let them know on the first packet that port is closed, etc. Internet is not really all candy canes and suger plums ;) While in the perfect world hey lets all be nice and tell the sender hey buddy port is closed RST. But why should you make the asshat looking for open smtp servers by probing random IPs on 25 for smtp.. Let him freaking timeout vs telling him right away you don't run one, etc.

Which is why you might want to turn that on internally.. On my internal networks that have limited outbound ports open, I send them rejects on ports they are trying to go out that I block. Because hey its my devices doing it, etc. So sure let them know vs letting them retrans, etc.

jimp

@Harvy66:

@jimp:

Someone could spoof the source and you'd end up sending data toward an ~~attacker~~ attackee.

Reflection attack

Fixed, thx.

Nullity

@johnpoz:

Which is why you might want to turn that on internally.. On my internal networks that have limited outbound ports open, I send them rejects on ports they are trying to go out that I block. Because hey its my devices doing it, etc. So sure let them know vs letting them retrans, etc.

I use REJECT on my LAN interface as well and it seems to have no problems, but I also have a whitelist firewalling setup like you.

obroni

Hi, Thanks for all the replies, this is on an internal firewall between a DMZ and LAN so I believe RST's should be a safe option here.

From what I understand from some of the posts here, is that a normal reject rule should indeed pickup out of state packets. When I get a chance to take everything down again, I will take some tcpdumps and try and see if I can see any RST's being generated.