PfBlocker table count usage not increasing
-
@owner-of-a_BAKERY What was the error? Note PHP has a memory limit also, default 512 MB.
If a feed doesn’t load I’d expect something in the pfBlocker log.
-
@SteveITS Sorry, I havent saved it, I can only remember some error code like 23. I also cannot replicate it anymore. About the default PHP memory: Does it mean that I don't have to allocate more than 512MB so it won't make a difference?
-
@SteveITS if its not the RAM that is causing not all table counts to be loaded, then what exactly may be stopping pfsense to load all the available blocking lists accordingly?
-
If a feed doesn’t load I’d expect something in the pfBlocker log.
Aside from that... If I reload / update I have the minor feeling / look-over that some of my custom and some feeded lists are not listed. Which "pfBlocker log" do you mean exactly? There are many...
-
@owner-of-a_BAKERY NEVERMIND I think I know which one your asking for @SteveITS.
Here are SOME of the blocklists named right on the homepage of my pfsense that seem to just randomly vanish out of the system, my custom ones aren't listed, but proofable also not in use...:
[ pfB_VPN_6_v6 - Ejrv_VPNv6_v6 ] Download FAIL [ 09/1/24 00:40:08 ] [ pfB_VPN_4_v4 - Ejrv_VPNv4_v4 ] Download FAIL [ 09/1/24 00:40:03 ] [ pfB_MAIL_v4 - LB_BL_v4 ] Download FAIL [ 09/1/24 00:40:02 ] [ pfB_PRI4_v4 - CoinBlocker_v4 ] Download FAIL [ 09/1/24 00:39:00 ] [ DNSBL_Compilation - OISD ] Download FAIL [ 09/1/24 00:36:30 ] [ DNSBL_Malicious2 - Ponmocup ] Download FAIL [ 09/1/24 00:36:15 ] [ DNSBL_Malicious2 - Malc0de ] Download FAIL [ 09/1/24 00:35:13 ] [ DNSBL_Compilation - OISD ] Download FAIL [ 09/1/24 00:23:07 ] [ DNSBL_Malicious2 - Ponmocup ] Download FAIL [ 09/1/24 00:22:22 ] [ pfB_VPN_6_v6 - Ejrv_VPNv6_v6 ] Download FAIL [ 09/1/24 00:02:47 ] [ pfB_TOR_v4 - DMe_TOR_EN_v4 ] Download FAIL [ 09/1/24 00:02:44 ] [ pfB_VPN_4_v4 - Ejrv_VPNv4_v4 ] Download FAIL [ 09/1/24 00:02:43 ] [ pfB_MAIL_v4 - LB_BL_v4 ] Download FAIL [ 09/1/24 00:02:43 ] [ pfB_PRI4_v4 - CoinBlocker_v4 ] Download FAIL [ 09/1/24 00:01:42 ] [ DNSBL_Compilation - OISD ] Download FAIL [ 09/1/24 00:01:27 ] [ DNSBL_Malicious2 - Ponmocup ] Download FAIL [ 09/1/24 00:01:10 ] -
@owner-of-a_BAKERY The PHP limit is the memory used by PHP while a page is loading. There's a setting under System > Misc I think, pretty sure it made it into 2.7.2.
For the failed downloads, do the lists exist? Can you download the URL yourself? "nearly every offered Feed" seems like...a lot. If you've selected the UT1 adult feed that one in particular is over 1 GB of disk space to extract. (not sure how big it is, I was testing something)
-
@SteveITS said in PfBlocker table count usage not increasing:
For the failed downloads, do the lists exist? Can you download the URL yourself?
The domains seem to be down so no, even though the URL's are given, I cannot install them manually.
"nearly every offered Feed" seems like...a lot.
I know
Only now I realise why at some point SWAP usage popped up. About 30 min. before I was writing this pfsense did crash. When I looked at the console I saw this:Sep 2 00:02:00 kernel swap_pager: out of swap space Sep 2 00:02:00 kernel swp_pager_getswapspace(1): failed Sep 2 00:02:34 kernel pid 79687 (unbound), jid 0, uid 59, was killed: failed to reclaim memory Sep 2 00:08:31 kernel swap_pager: out of swap space Sep 2 00:08:31 kernel swp_pager_getswapspace(2): failed Sep 2 00:08:31 kernel swp_pager_getswapspace(1): failed Sep 2 00:08:39 kernel pid 71610 (unbound), jid 0, uid 59, was killed: failed to reclaim memory Sep 2 00:08:40 kernel pid 71973 (unbound-control), jid 0, uid 59, was killed: failed to reclaim memoryBack then when I was allocating 8 GB (I remember now) I didn't see a SWAP usage counter, I presume this is because there is already enough RAM (, but still if so, then why arent more table counts used as there is no more SWAP usage needed? As of for now I could claim that all this time the table count didn't increase because SWAP usage was active and thereafter preventing more load by more table counts, but this cannot apply based on no table count increase with 8GB RAM allocated and SWAP usage deactivated...?!) Maybe you can make more out of those error codes, clear up some of my misunderstanding / questions and solve this mysterious issue.
-
@owner-of-a_BAKERY I would narrow your problem down as far as possible. Which list do you think is not being counted properly, and why?
-
@SteveITS as said, the following are marked as not installed (because of failed downloads, I'm presuming those addresses are just unreachable/down):
#ALL DNSBL https://malc0de.com/bl/BOOT http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-shadowserver.csv https://dbl.oisd.nl/Aside from that, I wouldn't consider those 3 blocklists to be the issue why not all tables are used... As said, I'm still not 100% sure, whether "table counts" and blocked-ip's/DNS are considered to be the same, but I'm guessing it is. Thereafter my issue is that as shown here, there is only a fraction of the available table counts blocked and I don't know why is that. That is why I'm here to get a clearer picture or even be able to block all available table counts and not just 437.713 from 4.952.721.
-
@owner-of-a_BAKERY Do you have deduplication enabled in pfB? It works but there can be side effects.
What I was trying to say was, start with a low number and see if the counts match up. If they do, add a few more until they do not match.
Not sure about the memory but I would expect it takes more memory to read in and process a list, than to store the IPs in a table.