PF States limit reached.
-
I've seen state tables that size only in instances when malware is in play, or in one case, when an intern for a well-known antimalware company wrote a naive script querying their entire list of malicious domains with no limits on queries per second.
You may also want to try setting firewall optimization to "aggressive" but the preferable option is to limit max src states.
-
@beatvjiking said in PF States limit reached.:
I've seen state tables that size only in instances when malware is in play, or in one case, when an intern for a well-known antimalware company wrote a naive script querying their entire list of malicious domains with no limits on queries per second.
You may also want to try setting firewall optimization to "aggressive" but the preferable option is to limit max src states.
My thinking exactly. This is a student network, so god knows what are they trying to do.
Interestingly enough some time ago I was doing tests with hping and packet generator(pktgen I think) and I have managed to fully load up the device (full CPU, full state table etc, interfaces at capacity), however normally after I stop the test the device always recovered. It crashed only once, from many test, but I could not reproduce it. This full lockup I've never managed to reproduce.
This is exactly why the states numbers have been raised. I've tested it up to 5M states with no issues.
-
I read the doco the same way, why doesn't the firewall start just nuking sessions straight away? It shouldn't be possible to hit 5M with your config.
I agree that's not the right solution for your problem, but regardless, shouldn't this be working for you?
-
That was my thinking exactly. I've just added the "max src states" to all the firewall rules(which are pass).
-
Also the firewall had a kernel panic on reboot.(decide to reboot it because the graphs were not working).
I checked in /var/crashes and there was no dump.
-
So this problem happens every 200 days or so? Uptime in screenshot.....
-
No it happens once a day for the last 3 days. I don't reboot the firewall, I just flush the states. "pfctl -F states all"
-
Just wanted to report it has not happened since I put the limits on.
-
You can probably find in your logs what device(s) are attempting to open so many sessions and address whatever is happening - i.e. malware or what have you.
-
Just to report it happened again. In my eyes, there are two options: Option 1 is adaptive timeouts are not working. Option 2 is the device somehow running out of memory. I can see in the monitoring graph that when the states reach roughly 900k the device becomes un-resposive. I've set much lower adaptive timeouts now and put the max states to 5mil(8G RAM). max src states is at 8096 on each firewall rule.
If anybody has a suggestion on how to simulate a lot of connection states from multiple IP, I would love to hear it.
-
You're not alone: https://www.google.com/search?client=firefox-b-1-d&channel=cus&q=pfsense+pf+states+limit+reached
this one mentions a Spiceworks scan of a large subnet:
https://forum.netgate.com/topic/81059/zone-pf-states-pf-states-limit-reached-how-to-find-the-offender/10
Given that post (simultaneous scan), how often are the adaptive timeouts processed/changed/updated by pfSense? (instantly, every 5 minutes, etc.) -
When I recommended 8k states, that was a very high ceiling. It works well in my environment but in most environments that can be far far lower with no negative impacts on user experience. 512 is a reasonable limit to impose on your allow rules. You may want to try that as an alternative to more RAM :)
-
Is this running on Hyper-V?
-
@Derelict said in PF States limit reached.:
Is this running on Hyper-V?
Appreciate your reply. It is on a physical box. Supemicro Atom C2750.
