PfBlockerNG v2.0 w/DNSBL

BBcan177

@Merchant:

Hello , I find issue opening somewebsites , ie when i try to open it takes long time when pfblockerNG is enabled

Usually when sites are slow to load with DNSBL, is that you have a multi-segmented LAN, and the desktop doesn't have full access to the DNSBL VIP address… Can you ping the DNSBL VIP address? There is a setting in the DNSBL tab to auto-create a floating permit rule, then select all of the LAN subnets that need access to the DNSBL VIP…

when this happens do i need to manually whitelist all those domains

Goto the Alerts tab, and you will see all of the Domains that are blocked… You can click the "+" icon to suppress this domain... You can also manually add domains to the "Custom Suppression" list in the DNSBL tab at the bottom of that page. When you manually add domains, you need to run a "Force Reload - DNSBL" for those domains to be whitelisted...

note sure if this crash is related to above error but
/usr/local/www/pfblockerng/pfblockerng_update.php:90
/usr/local/www/pfblockerng/pfblockerng_update.php:425
/usr/local/www/pfblockerng/pfblockerng_update.php:419

This error is from the "Update" Tab… Do you notice any issues when viewing that page? When you see this error, the next time... Please review the pfblockerng.log and see if you find any errors around the same time and send those to me...

I also notice this from your post:
pfB_DNSBLIP 37930 0 Apr 20 09:59

Its quite strange for the DNSBLIP alias to have so many IPs… If you have a IP based List, add those to the IPv4 tab instead of DNSBL which is domain based. The DNSBLIP is to collect any IPs that are found in a DNSBL feed.... Its not going to cause any issues, but its better in the IPv4 tab....

Abhishek


This error is from the "Update" Tab... Do you notice any issues when viewing that page? When you see this error, the next time... Please review the pfblockerng.log and see if you find any errors around the same time and send those to me...

sure if i get same crash error again i will will upload the log


Can you ping the DNSBL VIP address?  There is a setting in the DNSBL tab to auto-create a floating permit rule, then select all of the LAN subnets that need access to the DNSBL VIP...

Connection is

WAN –- PFSENSE---LAN ( Single LAN ( No van or multiple network )
i uploaded a pic in above post showing ping , i am able to ping with no delay

Pinging secure.adnxs.com [10.10.10.1] with 32 bytes of data:
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64


Its quite strange for the DNSBLIP alias to have so many IPs... If you have a IP based List, add those to the IPv4 tab instead of DNSBL which is domain based. The DNSBLIP is to collect any IPs that are found in a DNSBL feed.... Its not going to cause any issues, but its better in the IPv4 tab....

yes i will move IP based list from DNSBL to IPV4 tab

BBcan177

@Merchant:

Can you ping the DNSBL VIP address? There is a setting in the DNSBL tab to auto-create a floating permit rule, then select all of the LAN subnets that need access to the DNSBL VIP…

Pinging secure.adnxs.com [10.10.10.1] with 32 bytes of data:
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64

You need to run this command to check for proper connectivity to the DNSBL VIP:
You should also be able to browse to the DNSBL VIP and get the 1x1…

ping 10.10.10.1

Goto the Alerts tab, and see what domains are being blocked… You can also hit F12 in the browser and open Dev Mode... Goto "console" and look for the domains that are blocked... You can try to suppress those domains that are causing you issue... Finally ensure that your desktops only have the pfSense DNS IP defined and no other DNS servers listed...

Abhishek

update

all my DNSBL list are names


http://adaway.org/hosts.txt
http://someonewhocares.org/hosts/
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://winhelp2002.mvps.org/hosts.txt
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&startdate[day]=&startdate[month]=&startdate[year]=&useip=0.0.0.0
http://www.securemecca.com/Downloads/hosts.txt

http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintextoup
http://hosts-file.net/ad_servers.txt
http://adaway.org/hosts.txt
http://sysctl.org/cameleon/hosts
http://adblock.gjtech.net/?format=unix-hosts

https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt
http://hosts-file.net/fsa.txt
http://hosts-file.net/hjk.txt
http://hosts-file.net/pha.txt
https://www.dshield.org/feeds/suspiciousdomains_High.txt
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt
http://www.malwaredomainlist.com/hostslist/hosts.txt
http://mirror1.malwaredomains.com/files/justdomains
https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt
http://hosts-file.net/emd.txt
http://hosts-file.net/exp.txt
http://hosts-file.net/mmt.txt
http://lists.malwarepatrol.net/cgi/getfile?receipt=f1442112770&product=8&list=dansguardian
http://zeustracker.abuse.ch/blocklist.php?download=domainblocklist
https://malc0de.com/bl/BOOT

http://hosts-file.net/grm.txt
http://hosts-file.net/hfs.txt

https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt

http://hosts-file.net/hphosts-partial.txt

http://hosts-file.net/hphosts-partial.txt
http://someonewhocares.org/hosts/hosts
http://winhelp2002.mvps.org/hosts.txt
http://gist.githubusercontent.com/BBcan177/4a8bf37c131be4803cb2/raw

http://osint.bambenekconsulting.com/feeds/dga-feed.gz
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt


Pinging 10.10.10.1 with 32 bytes of data:
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64
Reply from 10.10.10.1: bytes=32 time<1ms TTL=64

Ping statistics for 10.10.10.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 0ms, Average = 0ms

But when i tried to open 10.10.10.1 in webbrowser i am getting pfsense login page instead of 1x1 image , so i guess that might b the problem

so i post my detail settings

SINGLE WAN LINK to NIC 1 PFSEnse ( wan dhcp ) –-----PFsense WAN Port --------PFsense LAN ( SINGLE LAN 192.168.x.x Network )

we dont have Multi WAN or MUlti LAN /Vlan
ie two NIC Cards ( 1 connected ISP LINK ) and other NIC Card ( Connected to Switch )

HTTPS is enabled for Pfsense
Installation is Fresh 2.3 install with xml backup restored

edit


nginx: 2016/04/20 15:09:06 [error] 65374#0: *476 open() "/usr/local/www/track/click/30047178/community.spiceworks.com" failed (2: No such file or directory), client: 10.10.10.1, server: , request: "GET /track/click/30047178/community.spiceworks.com?p=eyJzIjoiYnIzelB6UlBqRVFBSXNvajVBX0k5VVY0SExzIiwidiI6MSwicCI6IntcInVcIjozMDA0NzE3OCxcInZcIjoxLFwidXJsXCI6XCJodHRwczpcXFwvXFxcL2NvbW11bml0eS5zcGljZXdvcmtzLmNvbVxcXC90b3BpY1xcXC8xNTY3NTk1LWNhbi1pLXAydi1hLWRlYWQtcGVyc29uYWwtY29tcHV0ZXI_dXRtX2NhbXBhaWduPWRpZ2VzdCZ1dG1fbWVkaXVtPWVtYWlsJnV0bV9zb3VyY2U9ZGlnZXN0JnV0bWU9Z3JvdXBzK3BvcHVsYXIzXCIsXCJpZFwiOlwiNzFiODNmZjE4MzU3NDhlNzkzZDE3MmYzNDY5MTQ2MzhcIixcInVybF9pZHNcIjpbXCI1Zjg4NGU0OGIyYzY3OTZlMmExYzY3ZjNhNWQ2MDY3NDljZGJhMjFjXCJdfSJ9 HTTP/1.1", host: "track.spiceworks.com"

P1.PNG_thumb

P2.PNG_thumb

P3.PNG_thumb

P4.PNG_thumb

P5.PNG_thumb

P6.PNG_thumb

P7.PNG_thumb

docjay

@BBcan177:

@docjay:

BBCAN,

Hey- thanks for all of your hard work with this package! I'm setting up a 2nd PFsense box for a client and I'm wanting to copy over my 'pfblockerNG' config. Is there an easy way to do it other than doing it all by hand?

thanks

Jamie

Did you try the "Sync" Tab in the package?

LOL, nope. Sorry, I didn't see it or think it would do what I would expect. I'll give it a shot this evening!

Thanks!

BBcan177

@Merchant:

update

An obersevation: why use three different yoyo feeds:
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&startdate[day]=&startdate[month]=&startdate[year]=&useip=0.0.0.0
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintextoup

Is there a reason why you seperated hpHosts, instead of using the single download for them all?
http://hosts-file.net/download/hosts.zip

http://hosts-file.net/fsa.txt
http://hosts-file.net/hjk.txt
http://hosts-file.net/pha.txt
http://hosts-file.net/emd.txt
http://hosts-file.net/exp.txt
http://hosts-file.net/mmt.txt
http://hosts-file.net/grm.txt
http://hosts-file.net/hfs.txt

As a note: Look at the pfblockerng.log, it will show a section "DNSBL Domain/IP Counts" which will show you which DNSBL Feeds have IPs… look for the .ip extension...

But when i tried to open 10.10.10.1 in webbrowser i am getting pfsense login page instead of 1x1 image , so i guess that might b the problem

Yes your problem is due to Squid… I don't use Squid, and can't really help you workaround this issue.... Others have had a similar issue, but haven't seen anyone reply with a solution yet...

docjay

@docjay:

@BBcan177:

@docjay:

BBCAN,

Hey- thanks for all of your hard work with this package! I'm setting up a 2nd PFsense box for a client and I'm wanting to copy over my 'pfblockerNG' config. Is there an easy way to do it other than doing it all by hand?

thanks

Jamie

Did you try the "Sync" Tab in the package?

LOL, nope. Sorry, I didn't see it or think it would do what I would expect. I'll give it a shot this evening!

Thanks!

I tried the sync tab and it worked bad ass! I should have looked at it before, sorry for bothering you & thanks for thinking of this option!

Abhishek

@BBcan177:

@Merchant:

update

An obersevation: why use three different yoyo feeds:
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintext
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&startdate[day]=&startdate[month]=&startdate[year]=&useip=0.0.0.0
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintextoup

Is there a reason why you seperated hpHosts, instead of using the single download for them all?
http://hosts-file.net/download/hosts.zip

http://hosts-file.net/fsa.txt
http://hosts-file.net/hjk.txt
http://hosts-file.net/pha.txt
http://hosts-file.net/emd.txt
http://hosts-file.net/exp.txt
http://hosts-file.net/mmt.txt
http://hosts-file.net/grm.txt
http://hosts-file.net/hfs.txt

As a note: Look at the pfblockerng.log, it will show a section "DNSBL Domain/IP Counts" which will show you which DNSBL Feeds have IPs… look for the .ip extension...

But when i tried to open 10.10.10.1 in webbrowser i am getting pfsense login page instead of 1x1 image , so i guess that might b the problem

Yes your problem is due to Squid… I don't use Squid, and can't really help you workaround this issue.... Others have had a similar issue, but haven't seen anyone reply with a solution yet...

thank you , i will update the host's list
i will try modify my wpad so that host wont use proxy for 10.x.x.x network , will check today , thank you as usual , u rocks :)

update

Updated my wpad and now its working :D , thank you . its loading (1x1)
Now all websites are loading fast

GoldServe

Has anyone noticed the dnsbl alert entries have the wrong time? Seems to be a few hours ahead from all the other firewall entries.

Abhishek

@GoldServe:

Has anyone noticed the dnsbl alert entries have the wrong time? Seems to be a few hours ahead from all the other firewall entries.

mine is showing correct

PFB1.png_thumb

BBcan177

@GoldServe:

Has anyone noticed the dnsbl alert entries have the wrong time? Seems to be a few hours ahead from all the other firewall entries.

Which DNSBL alerts? HTTP or HTTPS. For HTTP, it takes the request time from the remote Lan device. Could the clock be off on one of your desktops?

GoldServe

Seems like HTTP timestamps are wrong. Which remote Lan device? All my laptops and desktops are Macs and their times are all correct. I am running pfsense in ESXI 6…

GoldServe

I don't know if it matters but it seems like my /usr/local/etc/php.ini has this line set

date.timezone="Etc/UTC"

when clearly /etc/rc.php_ini_setup contains the following code:

unset TIMEZONE
# Fetch the timezone from /var/db/zoneinfo if present
if [ -f /var/db/zoneinfo ]; then
	TIMEZONE=$(cat /var/db/zoneinfo)
fi

if [ -z "${TIMEZONE}" ]; then
	# Second option is from config.xml
	TIMEZONE=$(/usr/local/sbin/read_xml_tag.sh string system/timezone)
fi

if [ -z "${TIMEZONE}" ]; then
	# Last option, use default value from $g or Etc/UTC
	TIMEZONE=$(/usr/local/sbin/read_global_var default_timezone "Etc/UTC")
fi

Anyone spot anything wrong? I verified the timezone is correct by running the script /usr/local/sbin/read_xml_tag.sh string system/timezone

Abhishek

Hi,

same situation in my environment. I changes from HTTPS WebUI to HTTP because WPAD is hosted on pfsense itself. But i defined all private IPs to go DIRECT and not using the proxy in the WPAD files. But unfortunately it does not work.

Further I disabled auto proxy configuration and hardcoded the proxy in my browser but DNSBL still does only work with HTTP (pfsense WebUI page) but not with HTTPS. Further it does not log anything :-(

PS:
Anybody here who could suggest a good DNSBL which focuses on command and control server and such stuff?

can u post ur wpad file content , ( Look my wpad file , for me after instructing wpad to give Direct connection to 10.x.x..x now its working okay

@Nachtfalke:

@killmasta93:

FOR PEOPLE WITH GIF ISSUES READ THIS

If your running WPAD and have on your navigator to auto detect proxy you WILL get the webGUI of pfSense instead of the GIF but disable the auto detect proxy on the navigator you then will see the GIF…It seems that WPAD overrides everything even when it comes to OpenVPN also...

Thanks for that information. I am using WPAD with squid3 proxy. But I only get the WebUI with http websites. Is it working for you with httpS websites, too?
Can you describe if you configued something sepcidal to get pfblocker-NG + squid (http + https) working?

For me it looks like it is not working fpr https and it ios not working for http and https to see the bad websites in the "Alerts" tab of pfblopcker-ng.

Regards

@Nachtfalke:

@killmasta93:

Well I have pfBlockerNG working just for IP I had to disable DNSBL because WPAD was overriding everything. Because WPAD files are hosted on pfSense I had to revert to HTTP on the webGUI of pfSense. But by disabling auto detect proxy then works DNSBL but then WPAD wont work BUT transparent proxy will. I guess having WPAD and DNSBL wont work at the same time. UNLESS somehow on the wpad.dat a way to direct or ignore the 10.10.10.1 something that i wish it can do with OpenVPN. You get the HTTPS sites i guess because WPAD overlaps that but the http that would be the transparent proxy which DNSBL does not have an issue. Also depends if your webGUI is HTTPS or HTTP

I also had issue with squid with wpad initially and with BBcan177 help i resolved the issue with my setup

i updated my wpad.pac file to make direct connected to 10.10.10.1


 function FindProxyForURL(url,host)
 {

    if (isInNet( host, "192.168.0.0", "255.255.255.0" )) 
        { return "DIRECT"; }
	 if (isInNet( host, "10.10.10.1", "255.255.255.0" )) 
        { return "DIRECT"; }	
	return "PROXY 192.168.0.1:3128";
 }

now i am able to use proxy with wpad and also get 1x1 gif while blocking

bluepr0

Hello! I just noticed today that pfBlocker is not "working" anymore. It's actually enabled and it was working fine blocking ads, now it seems nothing changed (I'm sure I haven't changed any settings) but I'm not getting ads blocked

Here you can see it's enabled

Any ideas on what's going on? There's no errors in the logs either :(

It seems the latest blocked URLS were from April 19

DNSBL Reject HTTPS,Apr 19 18:15:10,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:10,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:11,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:15:44,ui.skype.com
DNSBL Reject HTTPS,Apr 19 18:16:44,ui.skype.com
DNSBL Reject HTTPS,Apr 19 18:17:44,ui.skype.com
DNSBL Reject HTTPS,Apr 19 18:18:44,ui.skype.com
DNSBL Reject HTTPS,Apr 19 18:19:44,ui.skype.com
DNSBL Reject HTTPS,Apr 19 18:20:44,ui.skype.com
DNSBL Reject HTTPS,Apr 19 18:20:51,ssl.google-analytics.com
DNSBL Reject HTTPS,Apr 19 18:20:51,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:51,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:51,analytics.query.yahoo.com
DNSBL Reject HTTPS,Apr 19 18:20:51,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:51,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:52,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:52,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:52,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:52,e.crashlytics.com
DNSBL Reject HTTPS,Apr 19 18:20:52,e.crashlytics.com

BBcan177

@bluepr0:

Hello! I just noticed today that pfBlocker is not "working" anymore. It's actually enabled and it was working fine blocking ads, now it seems nothing changed (I'm sure I haven't changed any settings) but I'm not getting ads blocked

Ensure that the LAN devices have only pfSense set as their DNS server. Can you ping the DNSBL VIP address? Can you browse to the DNSBL VIP address?

BBcan177

@Merchant:

I also had issue with squid with wpad initially and with BBcan177 help i resolved the issue with my setup

i updated my wpad.pac file to make direct connected to 10.10.10.1
 function FindProxyForURL(url,host)
 {
    if (isInNet( host, "192.168.0.0", "255.255.255.0" )) 
        { return "DIRECT"; }
    if (isInNet( host, "10.10.10.1", "255.255.255.0" )) 
        { return "DIRECT"; }	
    return "PROXY 192.168.0.1:3128";
 }
 
now i am able to use proxy with wpad and also get 1x1 gif while blocking

Thanks Merchant, I hope the other users that had the same issue can get this resolved also… 8)

bluepr0

@BBcan177:

@bluepr0:

Hello! I just noticed today that pfBlocker is not "working" anymore. It's actually enabled and it was working fine blocking ads, now it seems nothing changed (I'm sure I haven't changed any settings) but I'm not getting ads blocked

Ensure that the LAN devices have only pfSense set as their DNS server. Can you ping the DNSBL VIP address? Can you browse to the DNSBL VIP address?

Hello BBcan177,

I can ping the IP without problems, and all my devices are using my pfsense IP as DNS (as before) so it shouldn't be a problem there. However, if by "browse the DNSBL IP" you mean that I should get a reply when trying to open it from a browser then no, I get an error

But as I said, I get ping

BBcan177

@bluepr0:

I can ping the IP without problems, and all my devices are using my pfsense IP as DNS (as before) so it shouldn't be a problem there. However, if by "browse the DNSBL IP" you mean that I should get a reply when trying to open it from a browser then no, I get an error

Do you have a multi-segmented LAN? ie -vlans etc… If so, there is an option in the DNSBL Tab, to create a Floating Permit rule. Select all the Lan segments that need access to the VIP...

When you browse to the VIP, it should return the 1x1... Have you tried a different browser? or other LAN device?

bluepr0

@BBcan177:

@bluepr0:

I can ping the IP without problems, and all my devices are using my pfsense IP as DNS (as before) so it shouldn't be a problem there. However, if by "browse the DNSBL IP" you mean that I should get a reply when trying to open it from a browser then no, I get an error

Do you have a multi-segmented LAN? ie -vlans etc… If so, there is an option in the DNSBL Tab, to create a Floating Permit rule. Select all the Lan segments that need access to the VIP...

When you browse to the VIP, it should return the 1x1... Have you tried a different browser? or other LAN device?

Nop, I don't have VLANs.

Tried with Chrome but I'm getting same result

But I can still ping the IP. It seems a bit weird that I'm getting 265ms though, it's just a small LAN with a 8 port switch

Thanks again!