Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs)
-
It's hard to believe that this far into pfSense development that alias resolution is still a problem.
I've searched and searched to get either debug output or a log of any kind of the aliases being processed so that I can understand why they're not being refreshed.
I can ping a host and get a result from the resolver that is correct, however an alias with that same hostname does not show up in the alias table.
I've had many issues with this in the past but never as bad as it's happening right now. I have one alias that has a dozen FQDN in it (and has worked in the past) but currently has ZERO entries in the table. This is an alias that is being used in a firewall rule.
Can anyone help me with how I should begin debugging this issue?
-
The aliases are updated by a custom binary piece called
filterdns. The entire source code for that binary is here: https://github.com/pfsense/FreeBSD-ports/tree/devel/net/filterdns.That binary has a long and checkered past of not working correctly at random intervals. The last major rework was done in 2022 as best I recall. It is a somewhat complicated multithreaded application. It launches individual threads for each alias (and may even launch a separate thread for each FQDN defined in each alias). Been a while since I looked into the C code, so I don't recall the exact implementation.
I know this does not directly help you solve your issue, but if you have some C coding expertise you might take a look at the binary's code and spot something.
-
@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
I've had many issues with this in the past
Your a member for 11 hours... It is running ok here, so good luck.
-
@Bob-Dig does make a good point. The "I have a problem" posts related to
filterdnshave decreased quite a bit since the big update in 2022.You don't state the pfSense version you are running in your post. Are you on the latest 2.7.2 CE version or do you have pfSense Plus 24.11? Judging from the decrease in posts about FQDN resolution, the newest pfSense versions with the updated
filterdnsdaemon seem to be working better. -
@Bob-Dig I might have been a member for 11 hours, but I've been running pfSense since 2015 :P
-
@bmeeks yes, I'm on the latest version. I apologize I should have mentioned that. Unfortunately the problems persist. Is there anything I can do to help debug? I was hoping for some kind of logging. Looking at the C code isn't that helpful since a) I don't understand C and b) I'm more curious about what it's doing in my particular case.
-
@bmeeks Thanks for the reply! It looks like there's a reference to a log file in the variable
LOG_INFOthat is outputting some good information. Any idea where this is set or what it's set to for a default installation? -
Errors should log to Status/System Logs/System/DNS Resolver. It should attempt every 5 minutes unless the default timer was changed.
-
@SteveITS I found the logging. It's indeed happening in /var/log/resolver.log, but I'm not seeing any USEFUL log information.
Given a line line this: https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c#L502 can you give me insight into how I would increase the log level so I can get better debugging output to help me figure out the issue?
-
@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
Given a line line this: https://github.com/pfsense/FreeBSD-ports/blob/devel/net/filterdns/files/filterdns.c#L502 can you give me insight into how I would increase the log level so I can get better debugging output to help me figure out the issue?
Ah, you want the debug level to be higher, like 9 ?
You've found line 502.
Look for the place where the global variable 'debug' is set :So, find where 'filterdns' is executed by pfSEnse, and set the 'd' command line option yourself to a higher value :
It's here :
https://github.com/pfsense/pfsense/blob/6bf3e080f56facab1f00e29acd24dff62d5bd707/src/etc/inc/system.inc#L1649mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 1");You'll know what to do now
-
@Gertjan thank you! Ok so I tried that. I opened
system.incand updated the line to be:mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 9");and I'm still only getting two lines in my log files. I even rebooted the system but examples below are the only lines I'm seeing in
/var/log/resolver.log:filterdns[88435]: Adding Action: pf table: hosts_application_containers host: myapp.mydomain.com filterdns[88435]: Adding host myapp.mydomain.comEven when I know that domain's IP is not being added to the table (because the table is empty).
-
Hi !
what is the output in the console after executing commandps -ax | grep filterdns
If you see only two entries in the log and nothing else , it is possible that the program crashes at the time of startup.
-
@Konstanti said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
ps-ax | grep filterdns
lol - that's a syntax error.
You mean :[24.11-RELEASE][root@pfSense.bhf.tld]/root: ps -ax | grep filterdns 70495 - Is 0:14.75 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 98260 - Is 0:06.38 /usr/local/sbin/filterdns -p /var/run/filterdns-cpzone1-cpah.pid -i 300 -c /var/etc/filterdns-cpzone1-captiveportal.conf -d 1I've changed the 'd 1' to 'd 9' myself, and, to be sure, I've rebooted.
[24.11-RELEASE][root@pfSense.bhf.tld]/root: ps -ax | grep filterdns 81666 - Is 0:00.01 /usr/local/sbin/filterdns -p /var/run/filterdns.pid -i 300 -c /var/etc/filterdns.conf -d 1 88517 - Is 0:00.00 /usr/local/sbin/filterdns -p /var/run/filterdns-cpzone1-cpah.pid -i 300 -c /var/etc/filterdns-cpzone1-captiveportal.conf -d 1
Still 'd 1' ?!
Let's fact check :
[24.11-RELEASE][root@pfSense.bhf.tld]/root: grep 'd 9' /etc/inc/system.inc mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 9");Hummmmm.
Again - search better, more precise :
[24.11-RELEASE][root@pfSense.bhf.tld]/root: grep "-d 9" /etc/inc/*
/etc/inc/system.inc: mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-route.pid -i {$interval} -c {$g['varetc_path']}/filterdns-route.hosts -d 9");Now - are their any other places where filterdns is started ? :
[24.11-RELEASE][root@pfSense.brit-hotel-fumel.net]/root: grep "\-d 1" /etc/inc/* /etc/inc/captiveportal.inc: " -i 300 -c {$cp_filterdns_filename} -d 1"); /etc/inc/filter.inc: mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns.pid -i {$resolve_interval} -c {$g['varetc_path']}/filterdns.conf -d 1"); /etc/inc/ipsec.inc: mwexec_bg("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-ipsec.pid -i {$interval} -c {$g['varetc_path']}/ipsec/filterdns-ipsec.hosts -d 1"); /etc/inc/ipsec.inc: mwexec_bg("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns-ipsec.pid -i {$interval} -c {$g['varetc_path']}/ipsec/filterdns-ipsec.hosts -d 1");... yes !
4 more places where filterdns is started !
If you don't use the portal neither ipsec, editing /etc/inc/filter.inc will do the trick. -
I would have chosen a slightly different path (for testing only)
For example
From console- kill 81666
- /usr/local/sbin/filterdns -f -i 300 -c /var/etc/filterdns.conf -d 9
in this case, I use the option -f to debug the program.
at the same time, without making changes to the PF codeor
/etc/inc/filter.inc: mwexec("/usr/local/sbin/filterdns -p {$g['varrun_path']}/filterdns.pid -i {$resolve_interval} -c {$g['varetc_path']}/filterdns.conf -d 9"); -
@Konstanti
Way easier and better, I upvote. -
@dark-baritone Just to ask what is https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries set to? If using pfBlocker or anything that generates a lot of entries the advice I'd heard long ago was to set it to 2 million and increase as necessary.
-
ps -ax | grep filterdns@Konstanti returns nothing!
Now, I know that it runs at SOME point it runs because there are SOME logs, but very telling that it's not CURRENTLY running. 
-
@SteveITS Yeah I have it set extremely high. Like 80,000,000 (I have plenty of free memory). Just because I wanted to make sure that wasn't causing a problem.
-
@Konstanti @Gertjan doing it this way works perfectly for logging.
Ok so for one, when I run it manually, I'm seeing:
filterdns: could not start host thread for test.domain-not-being-added.orgSent to stderr in the shell.
So that seems to be happening either here or here. Either of those seem to error when it gets to here or here. But I might be wrong about that.
In the log I do see:
filterdns[65219]: [105139] ("filterdns.c":675 check_hostname_create()): Creating a new thread for host test.domain-not-being-added.orgFor the domains that are actually adding to the tables, I see log lines like:
filterdns[65219]: [978477] ("filterdns.c":507 host_dns()): found address 123.123.123.123 for host test2.domain-being-added-correctly.org filterdns[65219]: [978477] ("filterdns.c":434 addr_add()): adding address 123.123.123.123 for test2.domain-being-added-correctly.orgFor full disclosure just in case it matters, the table with the domain where the thread is failing is fairly large but doesn't seem prohibitively so and is currently sitting at around 7,500 entries. The alias with the domain where the thread is failing has about 260 domains listed in it. I haven't seen any documentation anywhere on limits to sizes other than "1k per table entry" and "all tables must fit within about half of the max table entries size". I have about 14GB of RAM free which should be plenty.
Thank again to everyone for the help. If there's something specific you all need out of the debug logs or anything else, I'm happy to provide.
-
@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
@Konstanti @Gertjan doing it this way works perfectly for logging.
Ok so for one, when I run it manually, I'm seeing:
filterdns: could not start host thread for test.somedomain.orgSent to stderr in the shell.
So either here or here both of which seem to error when it gets to here or here? I might be wrong about that.
In the log I do see:
filterdns[65219]: [105139] ("filterdns.c":675 check_hostname_create()): Creating a new thread for host test.domain-not-being-added.orgFor the domains that are actually adding to the tables, I see log lines like:
filterdns[65219]: [978477] ("filterdns.c":507 host_dns()): found address 123.123.123.123 for host test2.domain-being-added-correctly.org filterdns[65219]: [978477] ("filterdns.c":434 addr_add()): adding address 123.123.123.123 for test2.domain-being-added-correctly.orgBut the other hosts that aren't showing up only show up as the two lines that I pasted in my previous comment.
For full disclosure just in case it matters, the table where the thread is failing is fairly large but doesn't seem prohibitively so and is currently sitting at around 7,500 entries. The alias containing the domain that is erroring with the thread error has about 260 domains listed in it.
Thank again to everyone for the help. If there's something specific you all need out of the debug logs or anything else, I'm happy to provide.
Somewhere along the way when I briefly researched the history of
filterdnsproblems before making my initial post in this thread, I encountered a link that said out-of-the-box FreeBSD has a built-in limit on the number of threads a process can spawn. Perhaps you are hitting that limit?Here is one old post about increasing the limit: https://serverfault.com/questions/134616/increasing-freebsd-threads.
Here is an old post from 2009 to the FreeBSD mailing list: https://lists.freebsd.org/pipermail/freebsd-threads/2009-April/004554.html.
Maybe you are hitting a limit with a large alias ?? However, you would expect a more meaningful error message like "... can't create additional threads.." or "... exceeded thread limit ..." or something similar.
I see a lot of Google hits on "Linux thread limits", but very few results that address FreeBSD. I suspect FreeBSD certainly has its own internal limits of both max threads for the whole system and then max threads launched per process.
