Rules didn't apply to the host on reboot

nazar-pc

I rebooted pfSense a day ago and noticed just now that one of the rules didn't actually apply the way it is supposed to.
The role is using hostname of the device that has static DHCP mapping.

The rule is supposed to prevent the device from reaching Internet (Reject action) and worked for many years, but now looking at it not applied properly, I guess it didn't always work?

I can see in states table that connections are established, but I didn't find how to show what rules are used by pfSense on https://docs.netgate.com/pfsense/en/latest/troubleshooting/firewall.html

SteveITS

@nazar-pc see https://docs.netgate.com/pfsense/en/latest/troubleshooting/firewall.html#new-rules-are-not-applied

Can pfSense resolve the hostname?

Check Diagnostics >Tables for the alias.

nazar-pc

pfSense can resolve the hostname and correct IP addresses are present in Diagnostic -> Tables.

Existing connection should not be a problem when the whole pfSense reboots (at least that is my understanding, it would be very strange if it was).

The only thing I didn't do yet is pressing "Filter reload" because I want to figure out what causes the issue and if I press it, firewall might start working again, but I would have no way to find out why.

SteveITS

@nazar-pc is IPv6 in use?

johnpoz

@nazar-pc My guess would be host resolution, ipv6 could be another - but assume if you saw state you were seeing states related to IP you would expect to be blocked - ipv4 or ipv6.

If you do see the proper IP in the table. Then most likely related to your rule order, or a floating rule that was not there before that allows it.

Or maybe your rules are not fully loading - you might need to adjust your table size.

nazar-pc

@nazar-pc is IPv6 in use?

@nazar-pc My guess would be host resolution, ipv6 could be another - but assume if you saw state you were seeing states related to IP you would expect to be blocked - ipv4 or ipv6.

I have IPv6 disabled everywhere possible, so that is not it.

But since the table shows correct entries, I'm not sure where else to look.
The order of the rules is fine, I didn't change it and it certainly worked for years in its current state. And I do not use floating rules either.

If you do see the proper IP in the table. Then most likely related to your rule order, or a floating rule that was not there before that allows it.

I forgot to mention that the hosts are coming from IP aliases. So it looks as if the aliases were read by the firewall before DHCP server configuration was able to resolve them or something like that.

Or maybe your rules are not fully loading - you might need to adjust your table size.

I'm quite sure it is not the table size, but how do I check that? State table is at 2% right now according to the dashboard, but I suspect it is not the one you're talking about.

Also I'm wondering if there is a way to see the effective rendered rules that are actually applied. For example, I have a rule with ID X and I would like to see what its rendered output looks like.

SteveITS

@nazar-pc
There’s https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries but it’s rarely a problem unless you’re using large aliases like pfBlocker countries.

If the table is correct the alias is populated.

To view rules:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/firewall.html#ruleset-failing-to-load

nazar-pc

I do have a large list with IP addresses of countries, but the aliases are rendered correctly on /diag_tables.php and I have plenty of free RAM, so should not the the root cause.

I tried pfctl -f and the output looks like this:

table <smart_devices> persist
smart_devices = "<smart_devices>"

...

block return  in  quick  on $GUEST inet from $smart_devices to any ridentifier 123 label "USER_RULE: ..." label "id:123"

But I do not see the exact IP addresses themselves. Are they supposed to be present in the output explicitly? And if not, where can I find them them?

Patch

@nazar-pc in a similar situation for me after a long investigation this was the cause https://redmine.pfsense.org/issues/15708#change-78602

nazar-pc

@Patch Hm... not entirely sure what filterdns is and what it has to do with my issue.
filterdns process is running on my system.

Patch

@nazar-pc said in Rules didn't apply to the host on reboot:

not entirely sure what filterdns is

It is the program Netgate wrote to convert alias definitions to alias tables and update the code which actually implements the firewall filtering in pfsense.

As the above bug describes, if the total number of entires in all aliases (with at least 1 FQDN) exceeds 750 then; filterdns locks up and no further alias table updates or changes to firewall filtering will occur. If pfsense is restarted then a different set of aliases will not be populated and different filters will fail to perform the filtering specified in your configuration file.

The fix is easy to fry when you suspect there maybe an issue https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html ie in GUI go to

System > Advanced on the System Tunables tab.
Create an entry for kern.threads.max_threads_per_proc and set it to 4096
Status > Filter Reload and clicking Reload Filter.

SteveITS

@Patch OP says the alias/table is correct, though.

@nazar-pc The max table entries are a set limit, based on RAM by default but not RAM itself. If you are using pfBlocker the guideline I've always heard is to start at 2 million and raise if necessary. But again if your table is correct then it's not running out of space to populate the table.

Sorry the link I sent is incorrect. "pfctl -f /tmp/rules.debug" should show errors or blank output if correct. So, the same as filter reload.

This page is what I should have sent:
https://docs.netgate.com/pfsense/en/latest/firewall/pf-ruleset.html#interpreted-rules

nazar-pc

@SteveITS I have 4G of RAM assigned to pfSense VM, and I do not use pfBlocker. I have just a few small aliases and one from a file with ~16k CIDRs. I think I'm fine there.

pfctl -sr shows the same line I saw in the file generated by pfctl -f, but it doesn't have smart_devices definition. If that definition is the same as shown under /diag_tables.php then everything should work as expected.

I've done a bunch of experiments and looks like IP address works, but hostname doesn't. I do not use FQDN, just the just the base hostname assigned in static DHCP mapping.

Another issue turned out to be pfSense bug that was open for almost a year now, but frustratingly closed as "I can't reproduce this here. I tested this extensively before the last release." despite multiple people hitting the same issue over and over again: https://redmine.pfsense.org/issues/15956

Obviously, with dynamic mapping, the table still contained static mapping, so for that particular device even if the rules were working correctly, the IP address used was still wrong.

Feeling like I'm walking through the minefield. Did they break things in pfSense 2.8.0/2.8.1? I'm certain this used to work just fine in the past.

johnpoz

@nazar-pc said in Rules didn't apply to the host on reboot:

I do not use FQDN, just the just the base hostname assigned in static DHCP mapping.

I doubt that would ever work - So the table was not populated like you said then?

For the alias to work, what you put in it, hostname, fqdn would have to resolve to the correct IP you want the rule to apply too.. The IP(s) would be in the table of the alias, if they are not - then no the rule would never work.

nazar-pc

@johnpoz The table was definitely populated with correct IP addresses after reboot when I wrote original post. But now hostnames are indeed not resolved anymore

I have restarted DHCP server and did multiple filter reloads since though. What is even going on here?

johnpoz

@nazar-pc use a fqdn.. I don't see how just a hostname would ever work.. maybe way back many versions ago when unbound would just resolve based on hostname - but that should never happen in dns.. For dns to resolve it has to be a fqdn. They fixed that many versions ago.

it can work from a client just using hostname in two fashions - the client does a broadcast for the name with some discovery protocol like mdns or wsd, llmnr, etc.

Or if using dns it auto adds a search suffix for you, even though the user didn't type it in. Pfsense is not going to do that.

Use a fqdn for you devices you want to resolve - now does the table populate?

if you are having some problem with resolution - since you say your setting dhcp reservations for them, etc. just put in the IP in the alias so no resolution has to happen

but if the alias your wanting does not have IPs the rule is not going to work.

nazar-pc

@johnpoz This is a hostname of the device on my LAN/guest network. When I do nslookup abc it successfully resolves to abc.localdomain, so I totally expect the same exact behavior inside aliases.

I just tried FQDN and the table is indeed populated correctly.

Just to test it further, I replaced FQDN back with just base hostname and rebooted pfSense. The tables are correct even without FQDN!

This makes no sense...

johnpoz

@nazar-pc said in Rules didn't apply to the host on reboot:

it successfully resolves to abc.localdomain

and set a debug in your nslookup - it actually asked for the fqdn.

unbound is not going to answer a non fqdn query.

see what the client has set

filterdns is not going to do that.

nazar-pc

@johnpoz My point is that search localdomain in /etc/resolv.conf exists for a reason and since it is part of the pfSense configuration, I see no reason why it can't pick it up automatically.

And it does in fact do that, but only sometimes. I guess I'm hitting multiple bugs at once here, very confusing

johnpoz

@nazar-pc set a fqdn or IP.

The thing should pop up a warning if what you put in is not a fqdn. That is the bug I see, it shouldn't accept non fqdn entry as a host here. But I just tested and it does. Which is wrong.