High rate of state table searches
-
Hello all-
Been using pfSense for around a year now but just stumbled across something that is somewhat alarming.
Under pfInfo I see my state table searches value is consistently at 2500-2600/second regardless of throughput. Is this normal?
If this is normal, that's great, but in contrast my inserts and removals are about 4-5/second which makes me feel this is abnormal.
It would be helpful if anyone reading this could post what they see on their unit.. Diagnostics > pfInfo > State Table > Searches Rate
Thanks in advance for your feedback!
-
I also have about the same search rate, it seems very stable.
-
Under pfInfo I see my state table searches value is consistently at 2500-2600/second regardless of throughput. Is this normal?
If this is normal, that's great, but in contrast my inserts and removals are about 4-5/second which makes me feel this is abnormal.
Searches is generally equal to your pps rate. Insertions and removals are your rate of connection establishment and tear down. A single long-lived or high throughput connection can result in thousands of searches over its lifetime.
Though 2500-2600 searches a second on a system that's under very low load is a bit off from normal. My home system runs about 3/sec insert and removal, and ~250/sec searches. Our office firewall does about 10 times the inserts and removals as yours, but slightly less than half the search rate. Our primary colo has about 10 times the search rate as yours, with about 60 times the inserts and removals. Those are all more in line with what is typical, from what I've seen on many, many systems over the years.
I'm guessing you're blocking a significant number of packets from something, maybe broadcast and multicast crud on LAN and/or WAN, or similar. Much more than the typical norm.
-
Coincidence then. I restarted recently to apply mbuf setting changes and stuff and now that I look at it, my searches per second is down at 1.7k/s and it's pretty much the same as the averaged PPS between my WAN and LAN interface.
-
Thanks for the replies.
There are some details that I should probably add to give an overall view of the setup.. it may or may not be relevant.
This is a two host setup with ~19 CARP VIPs in a mix of WAN and ~9 VLAN'd interfaces on LAN. pfSync is running as a VLAN interface within the LAN physical interface (I'd like to change this, I know it's not best practice).
WAN and LAN ports on each unit (Dell R200s) each go to separate switches in a stacked config. WAN is an LACP group, again two ports on separate switches, that go to the colo.
Snort is running on WAN in a base config.
Search rate only seems to increase by 100-200/s between rates of <1Mbps and >800Mbps consistent traffic. Standby unit has search rate of 400-500/s.
I've looked at RRD data for all interfaces and none of them have any blocked traffic counts out of the ordinary.
Does any of this info lend any insight into the high searches/second at rest?
-
Check the Packets RRD graphs, it's likely more or less in line with the search rate.
There isn't a "you must have X rate of searches given Y rate of insertions/deletions", an average setup will be roughly along the lines of the numbers I posted previously, but that's dependent on what's happening on your network and can vary widely depending on the typical load your system is under. It sounds like you have many packets going through a small number of connections.