[zone: pf states] PF states limit reached - how to find the offender?

nikolaii

Hello,

this morning I couldn't connect to my infrastructure anymore. On the remote console of the firewall I could see the following message:

[zone: pf states] PF states limit reached

So I restarted the firewall and voila, everything is back to normal. But I'd like to know certain things:

how to find out what fulled the states tables up?
how to make the firewall accept new states even it the states tables are full? (maybe by discarding old ones?).

Currently I have this setting (I think this is the default one):


pfctl -sm
states        hard limit   201000
src-nodes     hard limit   201000
frags         hard limit     5000
table-entries hard limit   200000

Thank you.
Nicolas

Harvy66

There is an option under advanced settings that allows you to increase the rate at which states are expired as your table fills up.

nikolaii

I'll have a look on this, thank you.

Meanwhile, I was looking on the RRD graphs, and I don't see the limit (201000) being reached when the firewall has become unresponsive. So I guess there must be something else, right?

Any clue where to look for?

Thank you.
Nicolas

status_rrd_graph_img.php.png
status_rrd_graph_img.php.png_thumb

Shellite

I know I am bumping an old thread here, sorry!

I am suffering this issue also, happens after the box has run for ~1 month and the state table in RRD at the time showed ~0.4k states (middle of the night) so it was very quiet, even on the busiest of days would be lucky to hit 1.6k states.

Did you ever find a reason Nicolas?

@nikolaii:

I'll have a look on this, thank you.

Meanwhile, I was looking on the RRD graphs, and I don't see the limit (201000) being reached when the firewall has become unresponsive. So I guess there must be something else, right?

Any clue where to look for?

Thank you.
Nicolas

nikolaii

@Shellite:

Did you ever find a reason Nicolas?

Well unfortunately not, but fortunately this issue didn't occur again since…

Good luck!

stephenminta

I am having the same issue. Did anyone ever find a solution?

kswtch

Same issue here. Firewalls hangs in the middle of the night (just before midnight) with "PF states limit reached" showing up in the console.
0.7k states at the time. Normally the FW tops at 5k states. FW had been running for 48 hrs. We never saw that in the years before.
A reboot solved the issue. No clue what happend though.

edit:
We let pfSense forward all messages to our syslog server. At the time pfSense stopped responding with "states limit reached" it dumped many many many thousand lines at once to our syslog server. Most certanly, if every line is one state, thats what caused it. All the lines have the same timestamp but are clearly showing logs over a longer period.

JuantonJohn

If you have an open port (web interface) and it was DenialofService attacked; wouldn't that possibly cause the freeze / hang / crash? Maybe something trying to buffer overrun and exploit an open port?

…guessing...

mcdiesel

We had the same issue today. States typically running at 10k during work day.

Something suddenly goes nuts, fills all 201,000.

Culprit turned out to be Spiceworks, scanning 10.0.0.0/16 It does it by brute force, initiating an nmap scan to tens of thousands of (nonexistant) hosts concurrently.

Event spike (but not the source) shows up nicely in Status/Monitoring/Spanner left axis:System/States

Harvy66

You can configure pfSense to shorten the timeouts of the states as it gets fuller.

AdaptiveScaling.png_thumb