PfBlocker table count usage not increasing

owner-of.a_BAKERY

I don't know whether I understand something wrong, but im currently using nearly every offered Feed and some custom ones. These have a shown count of about 5-7m in total. I've cranked up the max. table count of the firewall to 20m and restarted pfsense with instead of 2 cores and 2 gb of ram with 4 cores and 8gb of ram (because I also got a notification indicating that some lists are not being loaded because of not enough memory, error code 23 or smth...). To my suprise, this time it ate up 90,5% of my RAM (with 2gb it was about 70%) so I thought that all the tables should now be loaded, but no... I still got the following results:

pfSense Table Stats
-------------------
table-entries hard limit 20000000
Table Usage Count         437713

What am I doing or understanding wrong? Either I'm misunderstanding the term table block for ip/domain-blocking or smth. is off here, please let me know what is the case here. Some (especially most of my custom added lists) are not loaded/listed at all. As far as I understand it, not a fraction of all the available / downloaded tables are being used as their count is listed below:

Alias table IP Counts
-----------------------------
  295374 total
  141587 /var/db/aliastables/pfB_SFS_v4.txt
   61905 /var/db/aliastables/pfB_DNS_4_v4.txt
   18029 /var/db/aliastables/pfB_PRI1_v4.txt
   15844 /var/db/aliastables/pfB_PRI3_v4.txt
   10610 /var/db/aliastables/pfB_MAIL_v4.txt
    9702 /var/db/aliastables/pfB_TOR_v4.txt
    9485 /var/db/aliastables/pfB_SCANNERS_v4.txt
    7173 /var/db/aliastables/pfB_uBlock_v4.txt
    5871 /var/db/aliastables/pfB_AWS_4_v4.txt
    4191 /var/db/aliastables/pfB_PRI4_v4.txt
    3703 /var/db/aliastables/pfB_SFS_6_v6.txt
    2759 /var/db/aliastables/pfB_AWS_6_v6.txt
    2758 /var/db/aliastables/pfB_PRI5_v4.txt
     608 /var/db/aliastables/pfB_PRI2_v4.txt
     377 /var/db/aliastables/pfB_Proxy_IP_v4.txt
     183 /var/db/aliastables/pfB_DNS_6_v6.txt
     171 /var/db/aliastables/pfB_BlockListDE_v4.txt
     135 /var/db/aliastables/pfB_DoH_IP_v4.txt
     102 /var/db/aliastables/pfB_PRI1_6_v6.txt
     100 /var/db/aliastables/pfB_DoH_6_v6.txt
      55 /var/db/aliastables/pfB_Torrent_IP_v4.txt
      13 /var/db/aliastables/pfB_Internic_6_v6.txt
      13 /var/db/aliastables/pfB_Internic_4_v4.txt

===[ DNSBL Domain/IP Counts ] ===================================

 4952721 total
 1904633 /var/db/pfblockerng/dnsbl/Chad_Mayfield.txt
  653697 /var/db/pfblockerng/dnsbl/Turkey_High_Risk.txt
  402319 /var/db/pfblockerng/dnsbl/Maltrail_BD.txt
  395354 /var/db/pfblockerng/dnsbl/Lightswitch05.txt
  377454 /var/db/pfblockerng/dnsbl/hageziGAMBLING.txt
  298309 /var/db/pfblockerng/dnsbl/CoinBlocker_All.txt
  174482 /var/db/pfblockerng/dnsbl/StevenBlack_ADs.txt
  151658 /var/db/pfblockerng/dnsbl/PhishingArmy.txt
   77552 /var/db/pfblockerng/dnsbl/AntiSocial_UK_BD.txt
   60960 /var/db/pfblockerng/dnsbl/AdguardDNS.txt
   59267 /var/db/pfblockerng/dnsbl/Joewein_base.txt
   52518 /var/db/pfblockerng/dnsbl/PhishTank.txt
   49773 /var/db/pfblockerng/dnsbl/SFS_Toxic_BD.txt
   45034 /var/db/pfblockerng/dnsbl/SNAFU_List.txt
   35550 /var/db/pfblockerng/dnsbl/Easyprivacy.txt
   34493 /var/db/pfblockerng/dnsbl/Anudeep_BL.txt
   28410 /var/db/pfblockerng/dnsbl/Prigent_Malware.txt
   17623 /var/db/pfblockerng/dnsbl/DandelionSprouts.txt
   15178 /var/db/pfblockerng/dnsbl/MS_2.txt
   13227 /var/db/pfblockerng/dnsbl/Quidsup_Trackers.txt
   12353 /var/db/pfblockerng/dnsbl/Frogeye_First.txt
   12031 /var/db/pfblockerng/dnsbl/Maltrail_Blackbook.txt
    9826 /var/db/pfblockerng/dnsbl/Prigent_Crypto.txt
    9112 /var/db/pfblockerng/dnsbl/CCT_BD.txt
    8937 /var/db/pfblockerng/dnsbl/Edwin_Email.txt
    7107 /var/db/pfblockerng/dnsbl/hostsVN.txt
    6951 /var/db/pfblockerng/dnsbl/Spam404.txt
    6666 /var/db/pfblockerng/dnsbl/yHosts.txt
    5893 /var/db/pfblockerng/dnsbl/Chad_Mayfield_1M.txt
    3979 /var/db/pfblockerng/dnsbl/Anudeep_Facebook.txt
    2767 /var/db/pfblockerng/dnsbl/Abuse_ThreatFox.txt
    2239 /var/db/pfblockerng/dnsbl/Matomo_Spam.txt
    1945 /var/db/pfblockerng/dnsbl/Phishing_Army.txt
    1787 /var/db/pfblockerng/dnsbl/neoHosts.txt
    1363 /var/db/pfblockerng/dnsbl/CoinBlocker_Opt.txt
    1219 /var/db/pfblockerng/dnsbl/Frogeye_Multi.txt
     998 /var/db/pfblockerng/dnsbl/EasyPrivacy.txt
     884 /var/db/pfblockerng/dnsbl/ENUMER_STUN.txt
     820 /var/db/pfblockerng/dnsbl/D_Me_ADs.txt
     775 /var/db/pfblockerng/dnsbl/Prigent_Ads.txt
     727 /var/db/pfblockerng/dnsbl/Yhonay_BD.txt
     597 /var/db/pfblockerng/dnsbl/hageziWINDOWS.txt
     506 /var/db/pfblockerng/dnsbl/Ad_Wars.txt
     450 /var/db/pfblockerng/dnsbl/OpenPhish.txt
     442 /var/db/pfblockerng/dnsbl/hageziXIAOMI.txt
     398 /var/db/pfblockerng/dnsbl/hageziTIKTOK.txt
     365 /var/db/pfblockerng/dnsbl/hageziOPPOREALME.txt
     349 /var/db/pfblockerng/dnsbl/LanikSJ.txt
     338 /var/db/pfblockerng/dnsbl/BarbBlock.txt
     307 /var/db/pfblockerng/dnsbl/hageziAPPLE.txt
     280 /var/db/pfblockerng/dnsbl/hageziAMAZON.txt
     267 /var/db/pfblockerng/dnsbl/Easylist_FB.txt
     262 /var/db/pfblockerng/dnsbl/Oneoffdallas_DoH.txt
     248 /var/db/pfblockerng/dnsbl/Max_MS.txt
     246 /var/db/pfblockerng/dnsbl/WaLLy3Ks.txt
     216 /var/db/pfblockerng/dnsbl/AZORult_BD.txt
     215 /var/db/pfblockerng/dnsbl/Kowabit.txt
     182 /var/db/pfblockerng/dnsbl/frellwitsSwedishHostsFile.txt
     165 /var/db/pfblockerng/dnsbl/Perflyst_TV.txt
     158 /var/db/pfblockerng/dnsbl/uBlockSEC.txt
     101 /var/db/pfblockerng/dnsbl/KADhosts.txt
      98 /var/db/pfblockerng/dnsbl/Joewein_new.txt
      92 /var/db/pfblockerng/dnsbl/Quidsup_Mal.txt
      84 /var/db/pfblockerng/dnsbl/hageziHUAWEI.txt
      81 /var/db/pfblockerng/dnsbl/NGOSANG_TORRENT.txt
      77 /var/db/pfblockerng/dnsbl/APT1_Report.txt
      75 /var/db/pfblockerng/dnsbl/hageziVIVO.txt
      58 /var/db/pfblockerng/dnsbl/hageziWEBOS.txt
      32 /var/db/pfblockerng/dnsbl/DigitalSide.txt
      30 /var/db/pfblockerng/dnsbl/EasyList.txt
      25 /var/db/pfblockerng/dnsbl/Botvrij_Dom.txt
      19 /var/db/pfblockerng/dnsbl/TheGreatWall_DoH.txt
      18 /var/db/pfblockerng/dnsbl/Abuse_urlhaus.txt
      15 /var/db/pfblockerng/dnsbl/Bambenek_DoH.txt
      12 /var/db/pfblockerng/dnsbl/Perflyst_Android.txt
      11 /var/db/pfblockerng/dnsbl/Adaway.txt
      10 /var/db/pfblockerng/dnsbl/D_Me_Tracking.txt
       9 /var/db/pfblockerng/dnsbl/VXVault.txt
       3 /var/db/pfblockerng/dnsbl/Perflyst_FireTV.txt
       2 /var/db/pfblockerng/dnsbl/Yoyo.txt
       2 /var/db/pfblockerng/dnsbl/MVPS.txt
       2 /var/db/pfblockerng/dnsbl/Magento.txt
       1 /var/db/pfblockerng/dnsbl/ZeroDot1.txt
       1 /var/db/pfblockerng/dnsbl/NoCoin.txt
       1 /var/db/pfblockerng/dnsbl/MoneroMiner.txt
       1 /var/db/pfblockerng/dnsbl/D_Me_Malv.txt
       0 /var/db/pfblockerng/dnsbl/URLhaus_Mal.txt
       0 /var/db/pfblockerng/dnsbl/uBlock.txt
       0 /var/db/pfblockerng/dnsbl/SWC.txt
       0 /var/db/pfblockerng/dnsbl/StevenBlack_BD.txt
       0 /var/db/pfblockerng/dnsbl/Risky_Hosts.txt
       0 /var/db/pfblockerng/dnsbl/PornOISD.txt
       0 /var/db/pfblockerng/dnsbl/Ponmocup.fail
       0 /var/db/pfblockerng/dnsbl/PL_Adservers.txt
       0 /var/db/pfblockerng/dnsbl/Piwik_Spam.txt
       0 /var/db/pfblockerng/dnsbl/OISD.fail
       0 /var/db/pfblockerng/dnsbl/Malc0de.fail
       0 /var/db/pfblockerng/dnsbl/Krog_BD.txt
       0 /var/db/pfblockerng/dnsbl/KAD_BD.txt
       0 /var/db/pfblockerng/dnsbl/ISC_SDH.txt
       0 /var/db/pfblockerng/dnsbl/H3X_1M.txt
       0 /var/db/pfblockerng/dnsbl/FM_Spam.txt
       0 /var/db/pfblockerng/dnsbl/Fademinds.txt
       0 /var/db/pfblockerng/dnsbl/Fademind_2o7.txt
       0 /var/db/pfblockerng/dnsbl/D_Me_Malw.txt
       0 /var/db/pfblockerng/dnsbl/AdOISD.txt
       0 /var/db/pfblockerng/dnsbl/Adguard_DNS.txt

SteveITS

@owner-of-a_BAKERY What was the error? Note PHP has a memory limit also, default 512 MB.

If a feed doesn’t load I’d expect something in the pfBlocker log.

owner-of.a_BAKERY

@SteveITS Sorry, I havent saved it, I can only remember some error code like 23. I also cannot replicate it anymore. About the default PHP memory: Does it mean that I don't have to allocate more than 512MB so it won't make a difference?

owner-of.a_BAKERY

@SteveITS if its not the RAM that is causing not all table counts to be loaded, then what exactly may be stopping pfsense to load all the available blocking lists accordingly?

owner-of.a_BAKERY

If a feed doesn’t load I’d expect something in the pfBlocker log.

Aside from that... If I reload / update I have the minor feeling / look-over that some of my custom and some feeded lists are not listed. Which "pfBlocker log" do you mean exactly? There are many...

owner-of.a_BAKERY

@owner-of-a_BAKERY NEVERMIND I think I know which one your asking for @SteveITS.

Here are SOME of the blocklists named right on the homepage of my pfsense that seem to just randomly vanish out of the system, my custom ones aren't listed, but proofable also not in use...:

[ pfB_VPN_6_v6 - Ejrv_VPNv6_v6 ] Download FAIL [ 09/1/24 00:40:08 ]
[ pfB_VPN_4_v4 - Ejrv_VPNv4_v4 ] Download FAIL [ 09/1/24 00:40:03 ]
[ pfB_MAIL_v4 - LB_BL_v4 ] Download FAIL [ 09/1/24 00:40:02 ]
[ pfB_PRI4_v4 - CoinBlocker_v4 ] Download FAIL [ 09/1/24 00:39:00 ]
[ DNSBL_Compilation - OISD ] Download FAIL [ 09/1/24 00:36:30 ]
[ DNSBL_Malicious2 - Ponmocup ] Download FAIL [ 09/1/24 00:36:15 ]
[ DNSBL_Malicious2 - Malc0de ] Download FAIL [ 09/1/24 00:35:13 ]
[ DNSBL_Compilation - OISD ] Download FAIL [ 09/1/24 00:23:07 ]
[ DNSBL_Malicious2 - Ponmocup ] Download FAIL [ 09/1/24 00:22:22 ]
[ pfB_VPN_6_v6 - Ejrv_VPNv6_v6 ] Download FAIL [ 09/1/24 00:02:47 ]
[ pfB_TOR_v4 - DMe_TOR_EN_v4 ] Download FAIL [ 09/1/24 00:02:44 ]
[ pfB_VPN_4_v4 - Ejrv_VPNv4_v4 ] Download FAIL [ 09/1/24 00:02:43 ]
[ pfB_MAIL_v4 - LB_BL_v4 ] Download FAIL [ 09/1/24 00:02:43 ]
[ pfB_PRI4_v4 - CoinBlocker_v4 ] Download FAIL [ 09/1/24 00:01:42 ]
[ DNSBL_Compilation - OISD ] Download FAIL [ 09/1/24 00:01:27 ]
[ DNSBL_Malicious2 - Ponmocup ] Download FAIL [ 09/1/24 00:01:10 ]

SteveITS

@owner-of-a_BAKERY The PHP limit is the memory used by PHP while a page is loading. There's a setting under System > Misc I think, pretty sure it made it into 2.7.2.

For the failed downloads, do the lists exist? Can you download the URL yourself? "nearly every offered Feed" seems like...a lot. If you've selected the UT1 adult feed that one in particular is over 1 GB of disk space to extract. (not sure how big it is, I was testing something)

owner-of.a_BAKERY

@SteveITS said in PfBlocker table count usage not increasing:

For the failed downloads, do the lists exist? Can you download the URL yourself?

The domains seem to be down so no, even though the URL's are given, I cannot install them manually.

"nearly every offered Feed" seems like...a lot.

I know Only now I realise why at some point SWAP usage popped up. About 30 min. before I was writing this pfsense did crash. When I looked at the console I saw this:

Sep 2 00:02:00 	kernel 		swap_pager: out of swap space
Sep 2 00:02:00 	kernel 		swp_pager_getswapspace(1): failed
Sep 2 00:02:34 	kernel 		pid 79687 (unbound), jid 0, uid 59, was killed: failed to reclaim memory
Sep 2 00:08:31 	kernel 		swap_pager: out of swap space
Sep 2 00:08:31 	kernel 		swp_pager_getswapspace(2): failed
Sep 2 00:08:31 	kernel 		swp_pager_getswapspace(1): failed
Sep 2 00:08:39 	kernel 		pid 71610 (unbound), jid 0, uid 59, was killed: failed to reclaim memory
Sep 2 00:08:40 	kernel 		pid 71973 (unbound-control), jid 0, uid 59, was killed: failed to reclaim memory

Back then when I was allocating 8 GB (I remember now) I didn't see a SWAP usage counter, I presume this is because there is already enough RAM (, but still if so, then why arent more table counts used as there is no more SWAP usage needed? As of for now I could claim that all this time the table count didn't increase because SWAP usage was active and thereafter preventing more load by more table counts, but this cannot apply based on no table count increase with 8GB RAM allocated and SWAP usage deactivated...?!) Maybe you can make more out of those error codes, clear up some of my misunderstanding / questions and solve this mysterious issue.

SteveITS

@owner-of-a_BAKERY I would narrow your problem down as far as possible. Which list do you think is not being counted properly, and why?

owner-of.a_BAKERY

@SteveITS as said, the following are marked as not installed (because of failed downloads, I'm presuming those addresses are just unreachable/down):

#ALL DNSBL
https://malc0de.com/bl/BOOT
http://security-research.dyndns.org/pub/malware-feeds/ponmocup-infected-domains-shadowserver.csv
https://dbl.oisd.nl/

Aside from that, I wouldn't consider those 3 blocklists to be the issue why not all tables are used... As said, I'm still not 100% sure, whether "table counts" and blocked-ip's/DNS are considered to be the same, but I'm guessing it is. Thereafter my issue is that as shown here, there is only a fraction of the available table counts blocked and I don't know why is that. That is why I'm here to get a clearer picture or even be able to block all available table counts and not just 437.713 from 4.952.721.

SteveITS

@owner-of-a_BAKERY Do you have deduplication enabled in pfB? It works but there can be side effects.

What I was trying to say was, start with a low number and see if the counts match up. If they do, add a few more until they do not match.

Not sure about the memory but I would expect it takes more memory to read in and process a list, than to store the IPs in a table.