[zone: pf states] PF states limit reached - how to find the offender?

[nikolaii]

Hello,

this morning I couldn't connect to my infrastructure anymore. On the remote console of the firewall I could see the following message:

[zone: pf states] PF states limit reached

So I restarted the firewall and voila, everything is back to normal. But I'd like to know certain things:

• how to find out what fulled the states tables up?
• how to make the firewall accept new states even it the states tables are full? (maybe by discarding old ones?).

Currently I have this setting (I think this is the default one):

pfctl -sm
states        hard limit   201000
src-nodes     hard limit   201000
frags         hard limit     5000
table-entries hard limit   200000

Thank you.
Nicolas

[Harvy66]

There is an option under advanced settings that allows you to increase the rate at which states are expired as your table fills up.

[nikolaii]

I'll have a look on this, thank you.

Meanwhile, I was looking on the RRD graphs, and I don't see the limit (201000) being reached when the firewall has become unresponsive. So I guess there must be something else, right?

Any clue where to look for?

Thank you.
Nicolas

[Shellite]

I know I am bumping an old thread here, sorry!

I am suffering this issue also, happens after the box has run for ~1 month and the state table in RRD at the time showed ~0.4k states (middle of the night) so it was very quiet, even on the busiest of days would be lucky to hit 1.6k states.

Did you ever find a reason Nicolas?

@nikolaii:

I'll have a look on this, thank you.

Meanwhile, I was looking on the RRD graphs, and I don't see the limit (201000) being reached when the firewall has become unresponsive. So I guess there must be something else, right?

Any clue where to look for?

Thank you.
Nicolas

[nikolaii]

@Shellite:

Did you ever find a reason Nicolas?

Well unfortunately not, but fortunately this issue didn't occur again since…

Good luck!

[stephenminta]

I am having the same issue. Did anyone ever find a solution?

[kswtch]

Same issue here. Firewalls hangs in the middle of the night (just before midnight) with "PF states limit reached" showing up in the console.
0.7k states at the time. Normally the FW tops at 5k states. FW had been running for 48 hrs. We never saw that in the years before.
A reboot solved the issue. No clue what happend though.

edit:
We let pfSense forward all messages to our syslog server. At the time pfSense stopped responding with "states limit reached" it dumped many many many thousand lines at once to our syslog server. Most certanly, if every line is one state, thats what caused it. All the lines have the same timestamp but are clearly showing logs over a longer period.

[JuantonJohn]

If you have an open port (web interface) and it was DenialofService attacked; wouldn't that possibly cause the freeze / hang / crash?  Maybe something trying to buffer overrun and exploit an open port?

…guessing...

[mcdiesel]

We had the same issue today.  States typically running at 10k during work day.

Something suddenly goes nuts, fills all 201,000.

Culprit turned out to be Spiceworks, scanning 10.0.0.0/16    It does it by brute force, initiating an nmap scan to tens of thousands of (nonexistant) hosts concurrently.

Event spike (but not the source) shows up nicely in Status/Monitoring/Spanner  left axis:System/States

[Harvy66]

You can configure pfSense to shorten the timeouts of the states as it gets fuller.