Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs)
-
@dark-baritone This is sort of a side note but each hostname will create a filterdns process for each domain so that would be 260 processes for 260 names. It's not terribly efficient but they are not that large. Possibly some sort of process limit?
I would lightly question though why you need to resolve 260 names every few minutes. Normally that's used for dynamic DNS or similar.
Any chance one is invalid, like a wildcard or something that doesn't resolve? Can you try adding them to another alias and see what happens? I realize that would take time but...
The other 7240 entries in the alias are just IPs?
There is this but at 80m you're pretty large anyway.
https://docs.netgate.com/pfsense/en/latest/firewall/aliases.html#alias-sizing-concerns -
@SteveITS Between IPv4 and IPv6 and the fact that some of these aliases are occasionally returning different IPs, those 260 domains are responsible for all of those IPs. I know having aliases that return different IPs is not the best use of aliases, but up until this point it's been working great for me. The resolution of the domain name hasn't been off so much that I've been being blocked when I shouldn't be and makes maintenance of my firewall rules manageable.
-
@bmeeks WTF man. Increasing the max threads did it!
I had to update to increase by about 2,000 threads to make it work, but eventually the entire thing ran with no problems!
I'd love to get everyone's input on whether 7,500 values in a table is just like absolutely insane or just "unadvised" and/or anything else I'm doing that could be done a "better" way. Presumably most people don't run into this issue
Although it seems like there should be a better way to understand that this is happening other than going through what I just went through hahaha
THANKS AGAIN TO EVERYONE WHO JUMPED INTO THIS THREAD!! I know your all's time is valuable and I appreciate it more than you could know.
-
@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
Increasing the max threads did it!
Glad that worked. It was an educated guess
.To answer your other question about the alias size, I think it is fair to say the designers of the
filterdnslogic probably did not anticipate more than 100 entries in a FQDN alias.Sounds like either the pfSense code needs to check the entries count in an alias assigned to
filterdnsand squawk if a limit is exceeded, or else restructure the code a bit so multiple FQDN hosts are resolved per thread (instead of launching a separate thread for each host). -
@dark-baritone pfBlocker (I think just -devel currently due to the underlying provider changes) can set up aliases using ASN (IP subnets registered to a company).
You could make a redmine.pfsense.org report that the "filterdns: could not start host thread" message is not logged (which is what I gather from your posts).
If I had to guess, I'd guess there is a performance crossover between "start 100 processes that trigger every 5 minutes on their own" and "have one thread spawn 100 processes every 5 minutes."
-
@SteveITS said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
I'd guess there is a performance crossover between "start 100 processes that trigger every 5 minutes on their own" and "have one thread spawn 100 processes every 5 minutes."
I was thinking something more like --
Start a thread that resolves 4 hostnames every 5 minutes instead of a single hostname every 5 minutes. The started threads do their work, then sleep for 5 minutes to wake again and resolve the host. Right now I believe it is one thread per host. It would cut down on total threads if each thread resolved a few more hosts. And that still should not overwork that thread too much.
-
This post is deleted! -
I searched and it looks like it's already being tracked: https://redmine.pfsense.org/issues/15708
-
@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
I searched and it looks like it's already being tracked: https://redmine.pfsense.org/issues/15708
...which links to https://docs.netgate.com/pfsense/en/latest/troubleshooting/filterdns-thread-errors.html, though with a slightly different error message.
-
@dark-baritone said in Alias Entries Are Not Being Added To The Tables (Even Hardcoded IPs):
I searched and it looks like it's already being tracked: https://redmine.pfsense.org/issues/15708
Ha! That's probably where I ran across the mention of a FreeBSD limit
. I didn't recall where I had seen that, but it probably was that Redmine ticket.Getting old and so easily forgetting stuff is such a pain in the rear --
