Filterdns has stopped resolving hostnames in firewall aliases

SteveITS

Happened again. Two hostnames that resolve to the same valid/correct IP are not in the table in pfSense. The log lists both:

Adding Action: pf table: AliasName host: host.example.com
Adding Action: pf table: AliasName host: host2.example.net

...but they're not in the table until I do the killall and then a filter reload.

SteveITS

Just us? :( Any idea of what to look for in logs? Since I can't seem to find an error...

Gertjan

@SteveITS said in Filterdns has stopped resolving hostnames in firewall aliases:

the "Is" state:

Is = Interrupted, and sleeping - so it's waiting for 'something'.
So, just guessing : the main job is hammering the DNS subsystem, normally the Resolver, with DNS requests.
What if unbound, the resolver was restarted / stopped ? and filterdns missed that / doesn't time out, and is waiting (sleeping) forever ?

My questions boils down to : what happens with your unbound ? Does it restart a lot ? Look at the resolver log to find out.

I can't recall if there is a command that can be used to see what a process is waiting for. Some one knows ?

SteveITS

@Gertjan Unbound's been running since May 1 on this router. Not using DHCP registration, or even DHCP on this router.

unbound 19499   0.0  2.3 124144  92208  -  Ss    1May25     14:45.04 /usr/local/sbin/unbound -c /var/unbound/unbound.conf

One of Jim's comments in 8758 was, "The I state indicates it's sleeping for over 20 seconds and per-se is not the problem because filterdns threads sleep for 1 minute so it will stay as S in the first 20 seconds and then move to I." So that may just be a red herring.

I didn't write it above but the missing IP in question this time was my home, and I log in every single day. Also AFAICT the IP didn't change (no notification in pfSense). So the IP just disappeared from the table one day.

SteveITS

Happened again.

/var/etc/filterdns.conf contains hostnames and table names as expected.

: ps aux | grep dns
root    14880   0.0  0.2  20348   9672  -  S    Fri20       0:11.89 /usr/local/sbin/lighttpd_pfb -f /var/unbound/pfb_dnsbl_lighty.conf
root    29469   0.0  0.1  21872   3552  -  Is   Fri20       0:02.73 /usr/local/sbin/filterdns -p /var/run/filterdns-ipsec.pid -i 60 -c /var/etc/ipsec/filterdns-ipsec.hosts -d 1
root    64743   0.0  0.4  88956  17488  -  Is   Fri20       0:08.51 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
root    14206   0.0  0.1  13040   2656  0  S+   17:39       0:00.00 grep dns

The table in question has only one IP in it, not two.

"grep filterd resolver.log" shows "Adding Action: pf table:" for the missing hostname.

As above, I had to "killall filterdns" and then Status>Filter Reload to recover.

SteveITS

Still just us huh? :(

Today I was unable to connect because the table was missing the IP again. At 4 am pfSense logged several "failed to resolve host" errors (from various hostnames)...based on the time I expect a ISP outage which is not uncommon as they do overnight restarts.

To be a bit different I "edited" the alias to save without changes, and applied...logs do show the "Adding Action" entry for the hostname in question...but it is not added and the table still contains the 29 entries it did before that.

As above, I had to "killall filterdns" to recover and add the 30th IP to the table.

edit:
Also of note this IP has not changed recently...the point being it was removed from the pf table at some point.

slu

@SteveITS said in Filterdns has stopped resolving hostnames in firewall aliases:

As above, I had to "killall filterdns" and then Status>Filter Reload to recover.

Tried this, but doesn't work for me.
But your issue look similar to mine...

slu

@SteveITS do you use pfBlockerNG?

SteveITS

@slu said in Filterdns has stopped resolving hostnames in firewall aliases:

do you use pfBlockerNG?

Yes and no...we use it for GeoIP aliases and DNSBL is enabled for "DoH/DoT/DoQ Blocking" but don't have other DNS lists enabled.

slu

@SteveITS I have no idea why this stop working, nothing abnormal in my logs.
Have two 2.8.0 with the same issue, try 25.07 later...

slu

@SteveITS said in Filterdns has stopped resolving hostnames in firewall aliases:

"DoH/DoT/DoQ Blocking"

Maybe this is the issue, because ACME doesn't work if all lists are enabled/blocked.

@stephenw10 ~~do you know aliases with FQDN use one of this lists?~~

Edit: Checked this on my 25.07 appliance without pfBlockerNG, there is no issue and aliases working.

SteveITS

@slu I don't think so because it works normally for us. It just stops and/or doesn't work for certain FQDNs at some point, and doesn't recover when it requeries the FQDNs every 5 minutes.

The DoH blocking is to not resolve those specific names, e.g. "cloudflare-dns.com" even though connecting by IP for a regular DNS lookup works fine.

Haven't tried 25.07 yet...but this issue takes a month or two to show up for us. Or at least, to be noticed...it's quite obvious when I can't connect from home each day but other locations are much less used.

slu

@SteveITS did you test/saw this:
https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html

Maybe we hit a limit because of pfBlockerNG?

SteveITS

@slu That's the "Unable to create monitoring thread" error; not seeing that here.

slu

@SteveITS not seeing this error, but I increase the value anyway and for the moment both systems working again. Monitoring that for the next days...

Gertjan

@slu said in Filterdns has stopped resolving hostnames in firewall aliases:

Maybe this is the issue, because ACME doesn't work if all lists are enabled/blocked.

I've checked them all, and activated, for years now :

I also use the ACME pfSense package for a long time now.
No issues what so ever.

the acme.sh shell script uses the available DNS to find the Letsencrypt server (or alternative) for the renewal request. As pfSense resolves by default, it doesn't care and won't use any 'DoH' DNS servers.

If you set up pfSense, the resolver, as a forwarder, and you forward to a 'DoH/DoT/DoQ' listed server, then unlist that DNS server.

slu

@Gertjan said in Filterdns has stopped resolving hostnames in firewall aliases:

I also use the ACME pfSense package for a long time now.
No issues what so ever.

Off topic:
Thanks for the feedback, interesting this works in your setup. For some reason the ACME script try different DNS server and get a timeout because pfBlockerNG reply (for example) for one.one.one.one NXDOMAIN. Maybe its relevant how ACME is configured.

Since we use the DNS servers from our ISP this can't be the issue here.

Gertjan

@slu said in Filterdns has stopped resolving hostnames in firewall aliases:

aybe its relevant how ACME is configured.

Nice catch !
This :

implies that when you set DNS Sleep to '0', it's the script itself that starts polling every 'x' seconds the domain name servers.
If its using one of the Doh etc, (which you've blocked with pfBlockerng) then yeah, that fails ...

Set DNS Sleep to "200" or so and solved ^^