Growing states table, some leak?

[w0w]

after some time it becomes full and firewall starts spamming console with “pf states limit reached” as a result no internet and no GUI accessible.

[johnpoz]

@w0w that is a lot of states.. How many clients? Do you have a bunch of devices doing p2p or something?

You typically run at 50% cpu? how much traffic is being moved.. What sort of traffic?

Do you have something or somethings creating lots of states faster then they expire? What makes up the majority of states - is it normal clients talking to some website(s).. Typical internet sort of traffic - or do you have something like p2p running where client might be talking to 1000's of devices in the torrent cloud, etc.

[w0w]

@johnpoz
Nothing changed, only 23.09 is installed. No changes in clients or traffic. CPU usage was random I think. May be some pfBlocker update or whatever.

I don't think it is real states, it looks like something went wrong. Because what you see on the first screenshot I did right after states are cleared using

[jimp]

I'm not seeing anything out of the ordinary here in my lab or on my edge running 23.09. My edge is hovering around ~1100 states and the states graph shows it going up/down as expected/when expected.

If you look under Status > Monitoring at the System / States graph it should show you a bit about what is happening.

If you look at the dump of states yourself (e.g. pfctl -ss or pfctl -vvss) you might also see a pattern going on that explains it, like some sort of traffic looping or a local device making lots of connections.

With a table that large the States Summary would probably not load since it wouldn't have enough RAM, but it might be worth trying at least.

[w0w]

@jimp

What does this button do then?
I am asking because it has no effect on the table size. I can even say that it was not reset until I rebooted firewall. In fact, reroot does not helpm only reboot.
I will try suggested :

@jimp said in Growing states table, some leak?:

pfctl -ss or pfctl -vvss

Just switched again to this firewall, will see ...

[Raul Ramos]

Hi,

This happens to me on a Backup pfSense Box. I though was something wrong with HA Settings.
My normal state table is 20 000, and the thing was already in 200 000.

[Dobby_]

You should may be change the "State Table size" to another amount, a higher
I mean here. If you have enough RAM installed that should be not that problem.

Something is doing there a "mass scan" in your network? You are a gamer
and scan much for good game server connections?

If you high up the state table size it will be using much more RAM, but if you
own or inserted enough of it, you may be better of to march this way here.

System > Advanced Settings > Firewall & NAT RAM

You should have a look inside and what amount is written there in.
You may be double the amount and monitor the RAM usage again.

If you are sorted only with small RAM option, 2 GB or 4 GB soldered onboard
you may be able to set up the "swap" partition to 4 GB or better 8 GB if you
will be able to realize it.

[w0w]

@Dobby_
There is going nothing. No ip/port scan, no p2p, spambots ddos, whatever. Nothing can generate so much entries in the table in my network. This is definitely some software bug.
Btw this is also secondary fw in CARP. Twice it growed when main firewall was in maintenance mode, so it was master.
When this happened for the second time, I looked to the Diagnostic -States and did not find those over million states showed on the main page, I saw less then several thousand entries. Then I did reset to states in the GUI, see screenshot above. Nothing changed. This is not the question of table size. I can put there billion state size, this just takes more time to fill it until it full again 🤭

[w0w]

at the same time

pfctl -ss>/root/dump.txt produces the file with 3441 lines not 1000213 as it theoretically should be. So what are those “ghost” states, @jimp?

[jimp]

Interesting that it doesn't line up. I checked two HA pairs in my lab (one Plus, one CE) and neither of them show that kind of behavior on either node.

What does pfctl -si show in its data for states? What about other info there?

That's the kind of weirdness I might expect to see if somehow the world and kernel don't match up, like either the kernel isn't being updated and it's running an older kernel, or the kernel updated but the base system didn't.

Check the output of pkg -x pfSense and uname -a and see what the output shows on there.

[w0w]

@jimp said in Growing states table, some leak?:

pfctl -si

@jimp said in Growing states table, some leak?:

pkg -x pfSense

pkg info? -x does not recognized as command

@jimp said in Growing states table, some leak?:

uname -a

14.0-ALPHA2 FreeBSD 14.0-ALPHA2 amd64 1400094 #1 plus-devel-main-n256131-b9588f9fb62: Sat Aug 26 18:08:03 UTC 2023     root@freebsd:/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/obj/amd64/iYn5SZ87/var/jenkins/workspace/pfSense-Plus-snapshots-master-main/sources/FreeBSD-src-plus-devel-main/amd64.amd64/sys/pfSense amd64

@jimp said in Growing states table, some leak?:

Interesting that it doesn't line up.

I switched from primary to secondary by putting primary into maintenance mode. May be I did it several times. At the same time I have fatal traps sometimes, it's on the other topic. And there is PPPoE on the WAN1, WAN2 is DHCP. MultiWAN gateway is used

[w0w]

This post is deleted!

[jimp]

Hmm, a lot of the info in pfctl -si looks suspiciously large.

Maintenance mode shouldn't have had any effect on pfsync / state data.

The base and kernel seem to match up, though.

All that said, PPPoE isn't supported with HA, nor is DHCP, so we don't do any testing in that regard. It's designed for, and only suitable for, static WANs, so who knows what kind of unpredictable results you might have.

Doing state sync is worthless in your case without a proper static WAN setup so you may as well disable and see if things stabilize. Without static WANs and proper CARP VIPs on all WANs there is no hope of seamless connection failover so pfsync is just complicating matters.

[w0w]

@jimp said in Growing states table, some leak?:

All that said, PPPoE isn't supported with HA, nor is DHCP, so we don't do any testing in that regard. It's designed for, and only suitable for, static WANs, so who knows what kind of unpredictable results you might have.

Yes, I know that PPPoE is not supported. I don't use it in CARP anyway. It is controlled by script, when firewall is not primary it just put it down and vise versa, my ISP also bans if I use more than 1 session for a long time.
WAN2 is used in CARP. Actually, those DHCPs means static lease on upstream router, so it actually the same all the way. So yes, I use WAN2 CARP IP, this configuration works fine since 2.6 alpha… So what next? Try to disable WAN2 CARP?

[jimp]

Not sure what to suggest since even if they are static in DHCP that's still not a supported configuration. No dynamic WANs are supported.

If it worked, consider yourself lucky as it worked by pure luck.

The top suspect would be any custom scripts and PPPoE since those are very far from standard.

Ideally if anyone else could reproduce it you could find something in common and track down how to reproduce it from a bare minimum configuration. Without more leads, it's difficult to speculate about what might be happening.

[w0w]

@jimp
Ok. Already changed to static. Will try to stop the script also.

[w0w]

@Raul-Ramos
What HA configuration are you using? Do you have some custom scripts enabled?

[Raul Ramos]

@w0w I do not have any custom scripts. Some packages: HAproxy, FreeRadius, acme, zabbix, wiregard, more two or three, nothing fancy.

My systems are not standard: Two virtualized instances of Proxmox, diferente boxes, similar hardware, each one have dedicated raw device networks (one em(0/1) other igb(0/1)) with 4 or five Vlans.
HA is configured to use Multicast on a VLANs created for SYNC, I tested using a IP but states continue to grow until nothin is usable.

At this moment the main instance is on the 23.05.1, backup is on the 23.09, all good, don't know if the stats are synced correctly, backup instance is 7000 down in stats count.

I change WAN setting to static. I'll test it later.

[w0w]

CARP is configured to use static IPs, custom script is disabled. States are keep growing and NOT cleaned by

it does not log anything also.
If I do

pfctl -F states

Then it clears a few thousands of states ex 4300, but GUI shows that overall states is 48000 currently. This is happening on both firewalls. Even if it is lower than critical limit.

[marcosm]

Have you tried to reproduce this with a clean install / minimal config? pfctl not clearing all of the states is certainly unexpected.