TLD question
-
Hey folks,
i enabled tld in pfblockerng-dev last night and was wondering about the performance.
looked around at the posts of others and some say their system crashed or whatever.. i didn't face that issue, although my ram temporarily went up to over 75% (of a 16gig system, which is kind've incredible considering how lean and efficient bsd is), and my /var folder temporarily filled to 98% (and i devoted 1Gig to it and /tmp).
i guess i understand how it has to process all the subdomains, etc.. and i understand how it increases security, so here's my question:
once all the subdomains have been processed, doesn't it effectively 'shrink' the list? put another way, if it's blocking an entire domain instead of filtering out the subdomains, wouldn't that increase performance?
put yet another way,www.crap.com/this_weeks_crap/todays_special_crap, which would normally take multiple ip addresses, has now been whittled down to just just blocking www.crap.com altogether..
i guess i have to study more on how ip ranges work.. it seems to me that just blocking the top domain saves all the trouble of the particulars of the subdomains, which (in many cases) result in faster performance.
thanks for the education and patience
-
@jc1976 While I am no expert on how PFBlocker works, I know what I see in terms of memory use and 75% of 16 gigs sounds like you are using many more lists than I am using; I've never seen 75% of my 8 gigs get used. I'm using 11 DNSBL lists including the large OISD. If you are using all or almost all the lists that would be the cause. And yes I have the TLD function ON, with a whitelist of maybe 130 domains and a TLD exclusion of about 40 domains. No need to overload it with all the lists or anywhere close to it, I am sure there is lots of duplication between them and all that has to get deduped with each update, which takes processor and memory. I've seen lists get deduped down to where there were only 3 or 4 domains or IPs being used on some of the lists because of all the duplication, so I pick and choose what gives good coverage for my use-case without having lots of overlap. Takes some time to get a good spread of coverage without the same thing five times over, needing to be weeded out.
-
@jc1976 Male sure you ust the Python mode in pfBlocker. It uses a lot less memory. My setup has 16GB memory with a lot of DNSBL and I rarely see the memory usage above 20%.
-
it turned out to be just an initial increase.. once all had been processed and idled for a while, memory and /var had reduced down to normal levels (like ~40% on both fronts).
i can't use python mode.. for whatever reason, unbound locks up. all works fine, although whenever i run a dnsbl reload, it takes a good amount of time (about 5 minutes). so i have it set to auto reload around 1:15am when i know i'll be asleep..
-
@jc1976 said in TLD question:
i can't use python mode.. for whatever reason, unbound locks up.
I'm glad things settled down for you. Python mode is known to lock up if DHCP Registrations is enabled. Uncheck this setting in Services/DNS Resolver/General Settings and you shouldn't have issues.
If you have particular clients that you need to find by their host name then assign them a static IP in their DHCP server and enable this setting in Services/DNS Resolver/General Settings
-
i'll check again, but i'm pretty sure i don't have either of those checked.
the only dhcp used is on the WAN side because that's what the isp needs.
as for lan, the dhcp server isn't enabled. I use the built in dhcp server of my wifi router for that. it has an 8 port gig switch built into it. it hands out ips and tells clients to look to 192.168.1.1 (the ip of my pfense box) for dns. the pfsense box handles internet security only (content blocking, malwhatever, firewalling, etc...) -
@jc1976 yeah, checked last night. i don't have any of those things enabled that would cause python mode to crash..
does python mode refer to python the language? or the mode to a mathematical function which is used to help processing?
-
@jc1976 said in TLD question:
@jc1976 yeah, checked last night. i don't have any of those things enabled that would cause python mode to crash..
does python mode refer to python the language? or the mode to a mathematical function which is used to help processing?
I'm in no way a programmer so really wouldn't know how to answer your question but there's detailed info on unbound python here: (https://unbound.docs.nlnetlabs.nl/en/latest/developer/python-modules.html)
What versions of pfSense and pfblocker are you using? I seem to recall that there were some issues in earlier versions.
-
@dma_pf i'm running the latest version of both.