Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs)

SteveITS

@dark-baritone Just to ask what is https://docs.netgate.com/pfsense/en/latest/config/advanced-firewall-nat.html#firewall-maximum-table-entries set to? If using pfBlocker or anything that generates a lot of entries the advice I'd heard long ago was to set it to 2 million and increase as necessary.

dark.baritone

ps -ax | grep filterdns

@Konstanti returns nothing! Now, I know that it runs at SOME point it runs because there are SOME logs, but very telling that it's not CURRENTLY running.

dark.baritone

@SteveITS Yeah I have it set extremely high. Like 80,000,000 (I have plenty of free memory). Just because I wanted to make sure that wasn't causing a problem.

dark.baritone

@Konstanti @Gertjan doing it this way works perfectly for logging.

Ok so for one, when I run it manually, I'm seeing:

filterdns: could not start host thread for test.domain-not-being-added.org

Sent to stderr in the shell.

So that seems to be happening either here or here. Either of those seem to error when it gets to here or here. But I might be wrong about that.

In the log I do see:

filterdns[65219]: [105139] ("filterdns.c":675    check_hostname_create()): Creating a new thread for host test.domain-not-being-added.org

For the domains that are actually adding to the tables, I see log lines like:

filterdns[65219]: [978477] ("filterdns.c":507                host_dns()):               found address 123.123.123.123 for host test2.domain-being-added-correctly.org
filterdns[65219]: [978477] ("filterdns.c":434                addr_add()):                       adding address 123.123.123.123 for test2.domain-being-added-correctly.org

For full disclosure just in case it matters, the table with the domain where the thread is failing is fairly large but doesn't seem prohibitively so and is currently sitting at around 7,500 entries. The alias with the domain where the thread is failing has about 260 domains listed in it. I haven't seen any documentation anywhere on limits to sizes other than "1k per table entry" and "all tables must fit within about half of the max table entries size". I have about 14GB of RAM free which should be plenty.

Thank again to everyone for the help. If there's something specific you all need out of the debug logs or anything else, I'm happy to provide.

bmeeks

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

@Konstanti @Gertjan doing it this way works perfectly for logging.

Ok so for one, when I run it manually, I'm seeing:
filterdns: could not start host thread for test.somedomain.org
Sent to stderr in the shell.

So either here or here both of which seem to error when it gets to here or here? I might be wrong about that.

In the log I do see:
filterdns[65219]: [105139] ("filterdns.c":675    check_hostname_create()): Creating a new thread for host test.domain-not-being-added.org
For the domains that are actually adding to the tables, I see log lines like:
filterdns[65219]: [978477] ("filterdns.c":507                host_dns()):               found address 123.123.123.123 for host test2.domain-being-added-correctly.org
filterdns[65219]: [978477] ("filterdns.c":434                addr_add()):                       adding address 123.123.123.123 for test2.domain-being-added-correctly.org
But the other hosts that aren't showing up only show up as the two lines that I pasted in my previous comment.

For full disclosure just in case it matters, the table where the thread is failing is fairly large but doesn't seem prohibitively so and is currently sitting at around 7,500 entries. The alias containing the domain that is erroring with the thread error has about 260 domains listed in it.

Thank again to everyone for the help. If there's something specific you all need out of the debug logs or anything else, I'm happy to provide.

Somewhere along the way when I briefly researched the history of filterdns problems before making my initial post in this thread, I encountered a link that said out-of-the-box FreeBSD has a built-in limit on the number of threads a process can spawn. Perhaps you are hitting that limit?

Here is one old post about increasing the limit: https://serverfault.com/questions/134616/increasing-freebsd-threads.

Here is an old post from 2009 to the FreeBSD mailing list: https://lists.freebsd.org/pipermail/freebsd-threads/2009-April/004554.html.

Maybe you are hitting a limit with a large alias ?? However, you would expect a more meaningful error message like "... can't create additional threads.." or "... exceeded thread limit ..." or something similar.

I see a lot of Google hits on "Linux thread limits", but very few results that address FreeBSD. I suspect FreeBSD certainly has its own internal limits of both max threads for the whole system and then max threads launched per process.

SteveITS

@dark-baritone This is sort of a side note but each hostname will create a filterdns process for each domain so that would be 260 processes for 260 names. It's not terribly efficient but they are not that large. Possibly some sort of process limit?

I would lightly question though why you need to resolve 260 names every few minutes. Normally that's used for dynamic DNS or similar.

Any chance one is invalid, like a wildcard or something that doesn't resolve? Can you try adding them to another alias and see what happens? I realize that would take time but...

The other 7240 entries in the alias are just IPs?

There is this but at 80m you're pretty large anyway.
https://docs.netgate.com/pfsense/en/latest/firewall/aliases.html#alias-sizing-concerns

dark.baritone

@SteveITS Between IPv4 and IPv6 and the fact that some of these aliases are occasionally returning different IPs, those 260 domains are responsible for all of those IPs. I know having aliases that return different IPs is not the best use of aliases, but up until this point it's been working great for me. The resolution of the domain name hasn't been off so much that I've been being blocked when I shouldn't be and makes maintenance of my firewall rules manageable.

dark.baritone

@bmeeks WTF man. Increasing the max threads did it!

I had to update to increase by about 2,000 threads to make it work, but eventually the entire thing ran with no problems!

I'd love to get everyone's input on whether 7,500 values in a table is just like absolutely insane or just "unadvised" and/or anything else I'm doing that could be done a "better" way. Presumably most people don't run into this issue

Although it seems like there should be a better way to understand that this is happening other than going through what I just went through hahaha

THANKS AGAIN TO EVERYONE WHO JUMPED INTO THIS THREAD!! I know your all's time is valuable and I appreciate it more than you could know.

bmeeks

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

Increasing the max threads did it!

Glad that worked. It was an educated guess .

To answer your other question about the alias size, I think it is fair to say the designers of the filterdns logic probably did not anticipate more than 100 entries in a FQDN alias.

Sounds like either the pfSense code needs to check the entries count in an alias assigned to filterdns and squawk if a limit is exceeded, or else restructure the code a bit so multiple FQDN hosts are resolved per thread (instead of launching a separate thread for each host).

SteveITS

@dark-baritone pfBlocker (I think just -devel currently due to the underlying provider changes) can set up aliases using ASN (IP subnets registered to a company).

You could make a redmine.pfsense.org report that the "filterdns: could not start host thread" message is not logged (which is what I gather from your posts).

If I had to guess, I'd guess there is a performance crossover between "start 100 processes that trigger every 5 minutes on their own" and "have one thread spawn 100 processes every 5 minutes."

bmeeks

@SteveITS said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

I'd guess there is a performance crossover between "start 100 processes that trigger every 5 minutes on their own" and "have one thread spawn 100 processes every 5 minutes."

I was thinking something more like --

Start a thread that resolves 4 hostnames every 5 minutes instead of a single hostname every 5 minutes. The started threads do their work, then sleep for 5 minutes to wake again and resolve the host. Right now I believe it is one thread per host. It would cut down on total threads if each thread resolved a few more hosts. And that still should not overwork that thread too much.

dark.baritone

This post is deleted!

dark.baritone

I searched and it looks like it's already being tracked: https://redmine.pfsense.org/issues/15708

SteveITS

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

I searched and it looks like it's already being tracked: https://redmine.pfsense.org/issues/15708

...which links to https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html, though with a slightly different error message.

bmeeks

@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):

I searched and it looks like it's already being tracked: https://redmine.pfsense.org/issues/15708

Ha! That's probably where I ran across the mention of a FreeBSD limit . I didn't recall where I had seen that, but it probably was that Redmine ticket.

Getting old and so easily forgetting stuff is such a pain in the rear --