No state tables!?!?!?

rcfa

So here's a thing that totally puzzles me…
I have two similarly configured pfSense boxes, one local, the other at a remote location (since today). The remote unit used to be some sort of ZyWall until it was replaced.
The pfSense-ZyWALL setup has been up for several years, and I noticed that I never have any state tables active, and I had brought that question up at some point before, but it was suggested that since I use the two devices to bridge two nets with IPSec, IPSec might gobble up everything at a level too low for it to show up in state tables. So I rolled with that.

Turns out now that I have two pfSense boxes, the new one does show states in the state tables as expected, and the other unit shows none, like before.

Needless to say, that's not right.

There are a few optional packages installed on the older box that are not on the new/remote box, so my question number one: does anyone know of any package that interferes with the (display of) state tables?

Are there any other known issues that would show an empty state table?

What else could be the cause of this?

Even more to the point, wiping the system and setting it up from scratch would be a major PITA, and wiping it and restoring saved settings would likely result in the same behavior. So if anyone has a clue on how to get to core of this issue, it would be much appreciated.

What is likely related: In the past I tried all sorts of bandwidth limiting, which also never worked. Since the setup went through quite a few upgrades over the years, I wonder if there might be any issues related up upgrading systems that could have resulted in this situation.

Before anyone asks: I just removed all extra packages that the one system had but the other didn't ~~except for bind which refused to deinstall itself (?).~~ Bind was deinstallable after I had removed to other extraneous packages, must have been blocked by something, which is strange, cause packages are supposed to be independent from each other…

Also, the hardware is identical, these are twin units, except for the configuration.

cmb

If states show 0, it's because the filter is disabled under System>Advanced. There is no alternative.

rcfa

@cmb:

If states show 0, it's because the filter is disabled under System>Advanced. There is no alternative.

Thanks for the suggestion, but I hate to break the news: unfortunately there must be an alternative, because under System>Advanced>Firewall/NAT the "disable all packet filtering" is NOT checked.
I wish it had been such a simple oversight by me, because then I could fix it just as easily :(

Maybe something has the same effect as this setting, etc. So what would be set where if that checkmark were set? Maybe something causes the same effect as if that check were set, so if I can look for that, I'll at least know where to start investigating this…

rcfa

Let me put it another way: assuming I did have the System>Advanced>Firewall/NAT the "disable all packet filtering" is checked (which I do not), what actual system configuration file would that setting end up in? What processes would or wouldn't be running? etc.
I want to check if despite what the GUI shows me, the system is in a state consistent with that having been checked or not.

If the system acts as if that setting were set, even though it's not, then we have found some sort of GUI bug (and need to figure out what triggers it). e.g. maybe a setting is duplicated in some config file, and the UI always only acts on the first instance, and thus the second instance which is never changed permanently overrides whatever line the UI keeps changing.

If the system is set up the way it should be, and there are still no states, then the question is, what else is going on.

charliem

@rcfa:

Let me put it another way: assuming I did have the System>Advanced>Firewall/NAT the "disable all packet filtering" is checked (which I do not), what actual system configuration file would that setting end up in? What processes would or wouldn't be running? etc.
I want to check if despite what the GUI shows me, the system is in a state consistent with that having been checked or not.

Look around for the 'disablefilter' variable in /etc/inc/filter.inc.
The PHP code for that page in the gui is /usr/local/www/system_advanced_firewall.php
Look through the saved config xml code in /cf/conf/config.xml
PHP code for displaying states detail in the gui is /usr/local/www/diag_dump_states.php

Be careful.

cmb

That was over-simplified to some extent. The other two options are you've created a slew of rules that don't keep state at all, including outbound floating rules, or you had exactly 0 active connections going to or through the system (which is impossible unless you're viewing the states at the console).

stephenw10

This will show you the state table from the console (states doesn't appear to work):

pfctl -s info

Steve

Edit: Ah 'states' in in 10, 'state' is in 8.3. 'info' is probably more useful here.

cmb

and "pfctl -ss" to dump the contents of the state table.

rcfa

So here's some good news and some more questions:
The good news is, I have states again.

What's questionable from my POV is what fixed the issue:
I had previously played around with the traffic shaper, but never got it to work properly.
So all rules referring to any of the queues were either disabled or deleted (actually in the process of troubleshooting this, I deleted all of the disabled rules, and any other rule that still had a reference to traffic shaping.) This did NOT fix the issue.

What did fix the issue, was deleting the queues that weren't referenced by anything.
Does that make sense, and why?
Also, the queues were set up by the traffic shaper wizard way back when.

I would expect that queues not referenced are not used and shouldn't interfere with anything, guess I'm wrong, either because I don't the workings well enough, or because there is/was a bug or messed up configuration somewhere.

Anyone have a comment regarding the issue if this is a matter of my stupidity or some sort of bug/corrupted config file?

stephenw10

It would be easy to miss a floating rule.

Check /tmp/rules.debug.

Steve

jimp

Also try:

pfctl -f /tmp/rules.debug

It may print an error, and if so, correct that and try again. Could be that the bogons list is corrupt or something similar. Usually that would produce an alert, but you can check it that way also.

rcfa

Thanks all for the various suggestions, I'll keep these handy for future reference.

However, right now, things are working again. As I said, the strange thing was, that even after trimming down the rules to next to nothing I still had no states, but the moment when I deleted the various queues that were originally set up by the traffic shaper wizard (but which I didn't use because all the relevant rules were first disabled, and then deleted), then things worked again.

So either the queues configuration was corrupt, or there's another bug that lets queues that are set up but not used in rules, affect things. I should have saved a before and after set of configuration settings, but I forgot. That would have made it easy to diff and try to figure out the issue. For the time being, as far as I'm concerned, the case is closed, but in case this point to a bug in the traffic shaper wizard or elsewhere, it's something to keep in mind that it seems that queues that according to the web GUI are unused can affect things.

rudger_wolvram

I know this post is old, but for my particular problem (which I'll post as a separate topic) this is the top result from google and thought I would add what fixed it for me for the next person that googles lost state table functionality.

So, In 2.3.2 I had the same exact problem, 0/200k states. After someone above mentioned that you should look for errors in /tmp/rules.debug, it hit me.

pfBlockerNG on my little Intel atom 2gb RAM firewall was throwing errors in that file (but never caused a problem) because of memory allocation limits. I finally rebooted it after making a change to the memory allocation for x86 installs (it was mentioned in a thread about x64 memory allocation problems) and lost the ability to have any states in my state table.

I uninstalled pfBlockerNG and the states came back right away, so, long story short, if you suddenly lose your state tables, look at the package or sub-system causing errors in /tmp/rules.debug (in at least 2.3+ it shows in the top right as a notification) and either see if you can correct the error or uninstall the package and do without.

stephenw10

If you run the manual ruleset reload command at the CLI (or from Diag > Command Prompt) it will report the error that is causing that. Almost certainly an unpopulated alias that pfBlocker created in that case.

pfctl -f /tmp/rules.debug

https://doc.pfsense.org/index.php/Firewall_Rule_Troubleshooting#Ruleset_Loading

Steve