Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs)

bmeeks

The aliases are updated by a custom binary piece called filterdns. The entire source code for that binary is here: https://github.com/pfsense/FreeBSD-ports/tree/devel/net/filterdns.

That binary has a long and checkered past of not working correctly at random intervals. The last major rework was done in 2022 as best I recall. It is a somewhat complicated multithreaded application. It launches individual threads for each alias (and may even launch a separate thread for each FQDN defined in each alias). Been a while since I looked into the C code, so I don't recall the exact implementation.

I know this does not directly help you solve your issue, but if you have some C coding expertise you might take a look at the binary's code and spot something.

Bob.Dig

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

I've had many issues with this in the past

Your a member for 11 hours... It is running ok here, so good luck.

bmeeks

@Bob-Dig does make a good point. The "I have a problem" posts related to filterdns have decreased quite a bit since the big update in 2022.

You don't state the pfSense version you are running in your post. Are you on the latest 2.7.2 CE version or do you have pfSense Plus 24.11? Judging from the decrease in posts about FQDN resolution, the newest pfSense versions with the updated filterdns daemon seem to be working better.

dark.baritone

@Bob-Dig I might have been a member for 11 hours, but I've been running pfSense since 2015 :P

dark.baritone

@bmeeks yes, I'm on the latest version. I apologize I should have mentioned that. Unfortunately the problems persist. Is there anything I can do to help debug? I was hoping for some kind of logging. Looking at the C code isn't that helpful since a) I don't understand C and b) I'm more curious about what it's doing in my particular case.

dark.baritone

@bmeeks Thanks for the reply! It looks like there's a reference to a log file in the variable LOG_INFO that is outputting some good information. Any idea where this is set or what it's set to for a default installation?

SteveITS

Errors should log to Status/System Logs/System/DNS Resolver. It should attempt every 5 minutes unless the default timer was changed.

dark.baritone

@SteveITS I found the logging. It's indeed happening in /var/log/resolver.log, but I'm not seeing any USEFUL log information.

Given a line line this: https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c#L502 can you give me insight into how I would increase the log level so I can get better debugging output to help me figure out the issue?

Gertjan

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

Given a line line this: https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c#L502 can you give me insight into how I would increase the log level so I can get better debugging output to help me figure out the issue?

Ah, you want the debug level to be higher, like 9 ?

You've found line 502.
Look for the place where the global variable 'debug' is set :

https://github.com/pfsense/FreeBSD-ports/blob/b1e731874a62b97181b588c4dfbf1a48119dd110/net/filterdns/files/filterdns.c#L895

So, find where 'filterdns' is executed by pfSEnse, and set the 'd' command line option yourself to a higher value :
It's here :
https://github.com/pfsense/pfsense/blob/6bf3e080f56facab1f00e29acd24dff62d5bd707/src/etc/inc/system.inc#L1649

mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 1");

You'll know what to do now

dark.baritone

@Gertjan thank you! Ok so I tried that. I opened system.inc and updated the line to be:

mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 9");

and I'm still only getting two lines in my log files. I even rebooted the system but examples below are the only lines I'm seeing in /var/log/resolver.log:

filterdns[88435]:       Adding Action: pf table: hosts_application_containers host: myapp.mydomain.com
filterdns[88435]:               Adding host myapp.mydomain.com

Even when I know that domain's IP is not being added to the table (because the table is empty).

Konstanti

@dark-baritone

Hi !
what is the output in the console after executing command

ps -ax | grep filterdns

If you see only two entries in the log and nothing else , it is possible that the program crashes at the time of startup.

Gertjan

@Konstanti said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

ps-ax | grep filterdns

lol - that's a syntax error.
You mean :

[24.11-RELEASE][root@pfSense.bhf.tld]/root: ps -ax | grep filterdns
70495  -  Is       0:14.75 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
98260  -  Is       0:06.38 /usr/local/sbin/filterdns -p /var/run/filterdns-cpzone1-cpah.pid -i 300 -c /var/etc/filterdns-cpzone1-captiveportal.conf -d 1

I've changed the 'd 1' to 'd 9' myself, and, to be sure, I've rebooted.

[24.11-RELEASE][root@pfSense.bhf.tld]/root: ps -ax | grep filterdns
81666  -  Is   0:00.01 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1
88517  -  Is   0:00.00 /usr/local/sbin/filterdns -p /var/run/filterdns-cpzone1-cpah.pid -i 300 -c /var/etc/filterdns-cpzone1-captiveportal.conf -d 1

Still 'd 1' ?!

Let's fact check :

[24.11-RELEASE][root@pfSense.bhf.tld]/root: grep 'd 9' /etc/inc/system.inc
                                mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 9");

Hummmmm.

Again - search better, more precise :

[24.11-RELEASE][root@pfSense.bhf.tld]/root: grep "-d 9" /etc/inc/*
/etc/inc/system.inc: mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 9");

Now - are their any other places where filterdns is started ? :

[24.11-RELEASE][root@pfSense.brit-hotel-fumel.net]/root: grep "\-d 1" /etc/inc/*
/etc/inc/captiveportal.inc:                         " -i 300 -c {$cp_filterdns_filename} -d 1");
/etc/inc/filter.inc:                            mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns.pid -i {$resolve_interval} -c {$g['varetc_path']}/filterdns.conf -d 1");
/etc/inc/ipsec.inc:                             mwexec_bg("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-ipsec.pid -i {$interval} -c {$g['varetc_path']}/ipsec/filterdns-ipsec.hosts -d 1");
/etc/inc/ipsec.inc:                     mwexec_bg("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-ipsec.pid -i {$interval} -c {$g['varetc_path']}/ipsec/filterdns-ipsec.hosts -d 1");

... yes !
4 more places where filterdns is started !
If you don't use the portal neither ipsec, editing /etc/inc/filter.inc will do the trick.

Konstanti

@Gertjan

I would have chosen a slightly different path (for testing only)
For example
From console

kill 81666
/usr/local/sbin/filterdns -f -i 300 -c /var/etc/filterdns.conf -d 9

in this case, I use the option -f to debug the program.
at the same time, without making changes to the PF code

or
/etc/inc/filter.inc: mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns.pid -i {$resolve_interval} -c {$g['varetc_path']}/filterdns.conf -d 9");

Gertjan

@Konstanti

Way easier and better, I upvote.

SteveITS

@dark-baritone Just to ask what is https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries set to? If using pfBlocker or anything that generates a lot of entries the advice I'd heard long ago was to set it to 2 million and increase as necessary.

dark.baritone

ps -ax | grep filterdns

@Konstanti returns nothing! Now, I know that it runs at SOME point it runs because there are SOME logs, but very telling that it's not CURRENTLY running.

dark.baritone

@SteveITS Yeah I have it set extremely high. Like 80,000,000 (I have plenty of free memory). Just because I wanted to make sure that wasn't causing a problem.

dark.baritone

@Konstanti @Gertjan doing it this way works perfectly for logging.

Ok so for one, when I run it manually, I'm seeing:

filterdns: could not start host thread for test.domain-not-being-added.org

Sent to stderr in the shell.

So that seems to be happening either here or here. Either of those seem to error when it gets to here or here. But I might be wrong about that.

In the log I do see:

filterdns[65219]: [105139] ("filterdns.c":675    check_hostname_create()): Creating a new thread for host test.domain-not-being-added.org

For the domains that are actually adding to the tables, I see log lines like:

filterdns[65219]: [978477] ("filterdns.c":507                host_dns()):               found address 123.123.123.123 for host test2.domain-being-added-correctly.org
filterdns[65219]: [978477] ("filterdns.c":434                addr_add()):                       adding address 123.123.123.123 for test2.domain-being-added-correctly.org

For full disclosure just in case it matters, the table with the domain where the thread is failing is fairly large but doesn't seem prohibitively so and is currently sitting at around 7,500 entries. The alias with the domain where the thread is failing has about 260 domains listed in it. I haven't seen any documentation anywhere on limits to sizes other than "1k per table entry" and "all tables must fit within about half of the max table entries size". I have about 14GB of RAM free which should be plenty.

Thank again to everyone for the help. If there's something specific you all need out of the debug logs or anything else, I'm happy to provide.

bmeeks

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

@Konstanti @Gertjan doing it this way works perfectly for logging.

Ok so for one, when I run it manually, I'm seeing:
filterdns: could not start host thread for test.somedomain.org
Sent to stderr in the shell.

So either here or here both of which seem to error when it gets to here or here? I might be wrong about that.

In the log I do see:
filterdns[65219]: [105139] ("filterdns.c":675    check_hostname_create()): Creating a new thread for host test.domain-not-being-added.org
For the domains that are actually adding to the tables, I see log lines like:
filterdns[65219]: [978477] ("filterdns.c":507                host_dns()):               found address 123.123.123.123 for host test2.domain-being-added-correctly.org
filterdns[65219]: [978477] ("filterdns.c":434                addr_add()):                       adding address 123.123.123.123 for test2.domain-being-added-correctly.org
But the other hosts that aren't showing up only show up as the two lines that I pasted in my previous comment.

For full disclosure just in case it matters, the table where the thread is failing is fairly large but doesn't seem prohibitively so and is currently sitting at around 7,500 entries. The alias containing the domain that is erroring with the thread error has about 260 domains listed in it.

Thank again to everyone for the help. If there's something specific you all need out of the debug logs or anything else, I'm happy to provide.

Somewhere along the way when I briefly researched the history of filterdns problems before making my initial post in this thread, I encountered a link that said out-of-the-box FreeBSD has a built-in limit on the number of threads a process can spawn. Perhaps you are hitting that limit?

Here is one old post about increasing the limit: https://serverfault.com/questions/134616/increasing-freebsd-threads.

Here is an old post from 2009 to the FreeBSD mailing list: https://lists.freebsd.org/pipermail/freebsd-threads/2009-April/004554.html.

Maybe you are hitting a limit with a large alias ?? However, you would expect a more meaningful error message like "... can't create additional threads.." or "... exceeded thread limit ..." or something similar.

I see a lot of Google hits on "Linux thread limits", but very few results that address FreeBSD. I suspect FreeBSD certainly has its own internal limits of both max threads for the whole system and then max threads launched per process.

SteveITS

@dark-baritone This is sort of a side note but each hostname will create a filterdns process for each domain so that would be 260 processes for 260 names. It's not terribly efficient but they are not that large. Possibly some sort of process limit?

I would lightly question though why you need to resolve 260 names every few minutes. Normally that's used for dynamic DNS or similar.

Any chance one is invalid, like a wildcard or something that doesn't resolve? Can you try adding them to another alias and see what happens? I realize that would take time but...

The other 7240 entries in the alias are just IPs?

There is this but at 80m you're pretty large anyway.
https://docs.netgate.com/pfsense/en/latest/firewall/aliases.html#alias-sizing-concerns