pfBlockerNG-devel feedback


  • Moderator

    Its been awhile since pfBlockerNG-devel was released. Just checking to see if anyone had any issues that they would like to report. Working on trying to get it released.

    Devel notes here:
    https://www.reddit.com/r/PFSENSE/comments/8lnugz/pfblockerng_devel_version_released/

    For those still on the fence, please try to install pfBlockerNG-devel as there haven't been many reported issues and there are a lot of features/fixes in devel that users should be migrating towards.

    The only open issue so far is the following SQLite3 issue. I can't get a handle yet on why its only affecting some users and not all others. There is some issue with the journaling mode. There is a quick fix available if you hit that.

    https://forum.netgate.com/topic/130361/pfblockerng-devel-2-1-2/26

    Thanks!



  • Zero issues for me running it on my home prod environment and loving the new features, I think from a code/stability POV I'm very happy with it.

    The only thing I can think of is we need a good KB/reference material on how to do some things with it, I know we have these forums but it can be hard to find your problem or questions sometimes, personally I learn best from Youtube videos



  • Works fine, the only issue so far is a cosmetic issue where the links to the EasyList feeds are shown wrong (it downloads the correct feeds, so it's only the UI), some examples:

    0_1534841139188_UIBug1.png

    0_1534841261338_UIBug2.png



  • @jasonau said in pfBlockerNG-devel feedback:

    Zero issues for me running it on my home prod environment and loving the new features, I think from a code/stability POV I'm very happy with it.

    The only thing I can think of is we need a good KB/reference material on how to do some things with it, I know we have these forums but it can be hard to find your problem or questions sometimes, personally I learn best from Youtube videos

    I concur with Jason's comments. Just upgraded from pfBlockerNG 2.1.2_3 to pfBlockerNG-devel 2.2.1 on a pfSense 2.4.3 Release -P1 (amd64) box with no issues. I'm running multiple openvpn connections and a site-to-site tunnel and everything worked smoothly right after the upgrade.

    I would also like some more concise reference material on how it all works. I know that we have this forum and I have learned a lot from it. But there is certainly a steep learning curve.



  • I have been very satisfied with pfBlockerNG-devel. The only issues i had were cert issues with downloading feed updates which caused my cron jobs to act funny. However, i dug around some posts and found your recommendation to change those feeds to "FLEX" and that seemed to have fixed the problem.

    Otherwise, its been great. Ads dont get blocked get blocked on androids and on the ipad, but they still play. I think this has something to do with the way that apps distribute ads on those platforms.

    The usability of devel has been great. Also recently updated to 2.2.5_9 with no issues that i have seen.



  • Overall very happy, no issues that I could see.

    But I just got this error while looking at log files.

    PHP ERROR: Type: 1, File: /usr/local/www/pfblockerng/pfblockerng_log.php, Line: 192, Message: Allowed memory size of 536870912 bytes exhausted (tried to allocate 129 bytes) @ 2018-08-21 11:23:59


  • Moderator

    @grimson that was fixed early on. You must have been an early tester and created those easylist entries with the old code. You could try a "save" in the Easylist Tab to see if that repairs those entries.


  • Moderator

    @bartkowski that could happen if you attempt to view a very large log file and it exhausted the browser/php memory trying to load that log file.



  • @bbcan177 said in pfBlockerNG-devel feedback:

    @grimson that was fixed early on. You must have been an early tester and created those easylist entries with the old code. You could try a "save" in the Easylist Tab to see if that repairs those entries.

    No, a save does mix them up differently but they still don't match.



  • @bartkowski You could try to increase the "memory_limit" settings in /etc/inc/config.inc. This change will be lost when you upgrade pfsense as it installs the defaut config.inc.

    // Set memory limit to 512M on amd64.
    if ($ARCH == "amd64") {
    	ini_set("memory_limit", "512M");
    } else {
    	ini_set("memory_limit", "128M");
    }
    

    You can also the limit the size of pfblocker log files in pfBlockerNG / General Tab / Log Settings .



  • @ronpfs I'm running the SG-2440 with stock RAM. I'll give this a try.

    Edit: My config.inc file shows the same memory limits.



  • Ok, I did a fresh install (removed previous package and cleaned out any remaining files in /tmp) of pfBlockerNG-devel version 2.2.5_10 on pfSense 2.4.3p1.

    On the first visit of the EasyList feeds page everything is fine. Then I enabled EasyList, EasyPrivacy and Easylist German with all of their content, set Action to Unbound and saved the settings. Now the feed URLs in the UI are mixed up again. I'm using Firefox 61.0.2 but I see the same in IE. They do show fine in the config.xml, so I guess something gets mixed up when reading/interpreting the config for the UI:

    0_1535117945591_easylist_config.txt


  • Moderator

    @grimson

    I'm working on a patch for this... I will shoot you a PM if you don't mind testing that when its completed?



  • @bbcan177 said in pfBlockerNG-devel feedback:

    @grimson

    I'm working on a patch for this... I will shoot you a PM if you don't mind testing that when its completed?

    Happy to help, I have the System_Patches package installed so providing a patch is probably the easiest way to test the fix.


  • Moderator

    @grimson said in pfBlockerNG-devel feedback:

    Happy to help, I have the System_Patches package installed so providing a patch is probably the easiest way to test the fix.

    @Grimson, its just as easy to download the two patched files below. Let me know how it goes.

    fetch -o /usr/local/pkg/pfblockerng/pfblockerng.inc "https://raw.githubusercontent.com/BBcan177/FreeBSD-ports/patch-1/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/pkg/pfblockerng/pfblockerng.inc"
    
    fetch -o /usr/local/www/pfblockerng/pfblockerng_category_edit.php "https://raw.githubusercontent.com/BBcan177/FreeBSD-ports/patch-1/net/pfSense-pkg-pfBlockerNG-devel/files/usr/local/www/pfblockerng/pfblockerng_category_edit.php"
    


  • Looks good, I couldn't reproduce the issue anymore. Thanks.


  • Moderator

    @grimson said in pfBlockerNG-devel feedback:

    Looks good, I couldn't reproduce the issue anymore. Thanks

    Great. Thanks for testing. There are a bunch of changes in the next pull request:

    https://github.com/pfsense/FreeBSD-ports/pull/559



  • I did not notice for a while, but it looks like ever since I upgraded to this version the DNSBL has been crashing and restarting every minute, along with the service watchdog.

    I've also got the out of sync error, but I've force reloaded a bunch of times and the log just says DNSBL is out of sync. Not sure what to look for in it.



  • There is no need to place DNSBL under the System Watchdog.

    You need to post the log of a Force Reload All if you want to get help.

    One thing that can generate Out of Sync warnings is if you have Header/Label that are not unique.



  • @ronpfs ah, well then that should fix that part.

    I've attached the log to this post.
    0_1535239826092_pfblockerng.zip



  • @lordbob75

    *** DNSBL update [ 850567 ] [ 824258 ] ... OUT OF SYNC ! *** [ 08/25/18 16:04:06 ]
    
    

    850567 - 824258 = 26309

    Searching for 26309 show that you load twice Malware_Domains

    
    [ Malware_Domains ]		 Reload [ 08/25/18 15:58:47 ] . completed ..
      ----------------------------------------------------------------------
      Orig.    Unique     # Dups     # White    # TOP1M    Final                
      ----------------------------------------------------------------------
      26446    26446      137        0          0          26309                
      ----------------------------------------------------------------------
    
    
    [ Malware_Domains ]		 Reload [ 08/25/18 15:59:21 ] . completed ..
      ----------------------------------------------------------------------
      Orig.    Unique     # Dups     # White    # TOP1M    Final                
      ----------------------------------------------------------------------
      26446    26446      137        0          0          26309                
      ----------------------------------------------------------------------
    


  • @ronpfs Awesome, thanks. I'll fix it. I'm not sure I understand how the number searched worked though. I'm not sure how to explain what I don't get about it.

       26309 /var/db/pfblockerng/dnsbl/Malware_Domains.txt
    

    In that list, is that number like the start of where that list adds to the master list or something? If that makes sense?



  • @lordbob75 said in pfBlockerNG-devel feedback:

    that number

    That's the number of Domain Names computed after removing Whitelist, TOP1M, Duplicates from other lists, etc.



  • @ronpfs Ok, that's what I figured but wanted to confirm. I appreciate the help!

    Edit: removing the duplicate entry did indeed fix it, awesome.



  • I have tried it and loved it and I can't wait for it to come out :)



  • I just took the plunge and moved to -devel....... It's fantastic. Having all the preset feeds and their organization into groups makes everything so much easier.

    One question though.... I'm confused where to put individual IP addresses and domains that I want to whitelist from ALL the IPV4 feeds.

    For DNSBL, I put domains in the DNSBL Whitelist box and that seems to work.

    For IPV4 on the previous version I had two custom Permit lists, which have got carrried over to the -devel version:

    0_1535738552244_Screen Shot 2018-08-31 at 20.57.32.jpg

    For domains that I want converted to IPs and then whitelisted, I put "Whois" in the source box and the domains in IPv4 Custom_List and this seems to work:

    0_1535738698058_Screen Shot 2018-08-31 at 21.03.44.jpg

    But for IPs that I want whitelisted I put the IPs in IPv4 Custom_List but I don't know what to put for Source and when I leave it blank I get this error:

    0_1535738796693_Screen Shot 2018-08-31 at 20.58.31.jpg

    Am I doing this all wrong or where should I be putting these?



  • @occamsrazor said in pfBlockerNG-devel feedback:

    For domains that I want converted to IPs and then whitelisted, I put "Whois" in the source box and the domains in IPv4 Custom_List and this seems to work:

    You have to change the Format to Whois, then you type a Domain Name in the Source Field.



  • @occamsrazor said in pfBlockerNG-devel feedback:

    But for IPs that I want whitelisted I put the IPs in IPv4 Custom_List but I don't know what to put for Source and when I leave it blank I get this error:

    Change the State to Off



  • @ronpfs said in pfBlockerNG-devel feedback:

    @occamsrazor said in pfBlockerNG-devel feedback:

    But for IPs that I want whitelisted I put the IPs in IPv4 Custom_List but I don't know what to put for Source and when I leave it blank I get this error:

    Change the State to Off

    Ah OK. So if I put State to Off but sill have a list of IPs in the IPv4 Custom_List text entry box they will still get added?

    @ronpfs said in pfBlockerNG-devel feedback:

    @occamsrazor said in pfBlockerNG-devel feedback:

    For domains that I want converted to IPs and then whitelisted, I put "Whois" in the source box and the domains in IPv4 Custom_List and this seems to work:

    You have to change the Format to Whois, then you type a Domain Name in the Source Field.

    If I do that I'd have to create a new "Format, State, Source, Header/Label" for each individual domain. Can I not have a list of domains in the IPv4 Custom_List box and check the "Enable Domain/AS" box", perhaps setting the State to OFF as suggested for the above?

    Thanks....



  • @occamsrazor said in pfBlockerNG-devel feedback:

    Ah OK. So if I put State to Off but sill have a list of IPs in the IPv4 Custom_List text entry box they will still get added?

    Yes.

    @occamsrazor said in pfBlockerNG-devel feedback:

    Can I not have a list of domains in the IPv4 Custom_List box and check the "Enable Domain/AS" box", perhaps setting the State to OFF as suggested for the above?

    Yes you can do that as well.

    You should also inspect the content of the tables in the Logs tab.



  • @ronpfs said in pfBlockerNG-devel feedback:

    Yes you can do that as well.
    You should also inspect the content of the tables in the Logs tab.

    Nice. Thanks a lot for clearing that up. When I go to the top two items in the dropdown seen here I can see all the IPs including the ones converted from domains, so I think that is all working correctly....

    0_1535744394594_Screen Shot 2018-08-31 at 22.38.02.jpg

    One final (I hope) question. Is there a way to keep one single domain whitelist that gets used for both:
    a) Conversion to IPs for IPV4 whitelisting
    b) Use in DNSBL whitelisting
    I get the impression the IPV4 and DNSBL functions operate very separately..... and that you would have to keep domain whitelist in the two places to be sure.



  • @occamsrazor
    IPV4 operates in the IP space. It can take domain names and convert them to IPs before building the tables.

    DNSBL operates in the DNS space, that is only with domain names.

    Instead of using Whitelist, why don't you suppress IPs instead?



  • @ronpfs said in pfBlockerNG-devel feedback:

    Instead of using Whitelist, why don't you suppress IPs instead?

    What would be the advantage of that way vs whitelist? In the IPV4 Suppression box I thought you could only enter ranges not individual IPs. But I guess you can enter them with /32 netmask, right?


  • Moderator

    @BBcan177 just a quick question: I checked on pfBlockerNG devel on a 2.4.4 snapshot system. Still shows php56-5.6.34 as dependency. As 2.4.4 runs on php7.2 I'm wondering, why pfBNG requires usage of the old PHP version (in package manager listing)?


  • Moderator

    @jegr said in pfBlockerNG-devel feedback:

    I checked on pfBlockerNG devel on a 2.4.4 snapshot system. Still shows php56-5.6.34 as dependency. As 2.4.4 runs on php7.2 I'm wondering, why pfBNG requires usage of the old PHP version (in package manager listing)?

    The pfSense devs manage that integration. Here is the commit to the makefile:
    https://github.com/pfsense/FreeBSD-ports/commit/54dd3d529ac6a55cd0c1e05f0c3956fb668d7cbd

    There seem to be some hiccups with this but I believe it to be part of the base pfSense code.


  • Moderator

    @bbcan177 no problem, just wanted to ask as that drew my attention :)

    Edit: My mistake, I set the system to "stable" after updating to 2.4.4-snapshots, to get it to 2.4.4-Release without any further snapshot. That switched Packages back to displaying 2.4.3 info, so the PHP version was old. Switching it back to snaps shows a correct 7.2.9 - my bad!


  • Moderator

    @jegr said in pfBlockerNG-devel feedback:

    @bbcan177 no problem, just wanted to ask as that drew my attention :)

    I did some tests and the only way I could get the PHP version to be out of sync was to set the 2.4.4 machine to use the pfSense 2.3.x branch ?

    EDIT: Haha... yes, I was typing as you made your edit !! :)


  • Moderator

    @bbcan177 said in pfBlockerNG-devel feedback:

    @jegr said in pfBlockerNG-devel feedback:

    @bbcan177 no problem, just wanted to ask as that drew my attention :)

    I did some tests and the only way I could get the PHP version to be out of sync was to set the 2.4.4 machine to use the pfSense 2.3.x branch ?

    EDIT: Haha... yes, I was typing as you made your edit !! :)

    Haha 😄 I was curious, too, as I read through the GIT intel so I backtracked and facepalmed over my own stupidity. Serves me right, better double check my facts before calling bugs 😉



  • No big deal but just to let you know these feeds have been getting download errors for the last few days..... at least for me.

    0_1536825395478_download fails.jpg



  • @BBcan177 Can you see my post https://forum.netgate.com/topic/135362/geoip-policy-based-routing-not-working-with-pfblockerng-devel
    To me it appears as an issue with the new version.