PfBlockerNG v2.0 w/DNSBL
-
I am seeing an issue where the pages with ads can take up to 20 seconds to timeout prior to loading the page…
Example: landsend.com will sit blank for 20 seconds waiting on assets.adobedtm.com, then loads immediately.
Our DNSBL webserver is set at 192.168.192.168 and i can ping it from my computer, it is also set to allow all traffic to/from this address in the floating rules.
My internal network is 10.0.0.0/12 and Guest network is in 172.16.0.0/16 range.
I have followed all the instructions in this forum, and all is working well, except this delay between DNSBL and the computer.
Are there any more tests i can do to check the connection to the DNSBL webserver? If I browse to the IP or request a jpg file it times out.
Thanks for any help or suggestions.
-Chad
-
I am seeing an issue where the pages with ads can take up to 20 seconds to timeout prior to loading the page…
If I browse to the IP or request a jpg file it times out.Hi Chad,
You have to check your Firewall/NAT rules to see if a rule is blocking access to the DNSBL VIP from your LAN. Without fixing that you will get timeouts as you are experiencing. You have to ensure that you can ping and browse to the VIP.
-
I have hard coded some domains so that they can't be suppressed ;) <kidding>Run the following command to see if the domain is in the DNSBL list:
grep "example.com" /var/unbound/pfb_dnsbl.confpinging those domains should also resolve to the DNSBL VIP address.
If the domain is listed there, then check to see if your device DNS is pointed to the Resolver, and is not resolving from to a different DNS server…</kidding>
if I ping the domains its resolve the DNSBL IP 10.10.10.1
now somehow stuff start working and it blocks the sites.
maybe it needed time ? I didn't change anything from side . -
I have posted a Pull Request #1223:
-
Fix 'Match Outbound' rule variable name
-
Remove duplicate Alexa variable as its already in pfb_global()
-
Determine if a DNSBL background reload is running before updating DNSBL
-
Change "${cmd}" variable name to "{$cmd}"
-
Add $pfb['ccdir'] explicitly due to file caching issue.
-
Fix dShield URL
-
Add tcpiputils to DNSBL Domain threat lookup
-
Add log size settings (5000 & 10000 lines) for limited storage installations. (Not recommended)
-
Add Paypal 'support the Developer' link
-
Remove extra whitespace
Also, the following issue was included in Pull Request #1219
-
-
-
I have a question of running VPN next to/ together with DNSBL.
On my laptop and Iphone/ Ipad i have client software running PIA VPN. With the VPN on nothing seem to be blocked, when switched of DNSBL is working. This is confirmed in the alerts tab and log.
Is it not possible to run VPN and DNSBL ? My understanding of the system is still limited. Hope you guys could point me in the right direction.
Greets,
HJ
-
Is it not possible to run VPN and DNSBL ? My understanding of the system is still limited.
I don't personally use a VPM like PIA, but the only way for DNSBL to function, is if the pfSense Resolver is used for the DNS requests. If the VPN uses a different DNS server, then DNSBL will not block any domains…
-
Ok thanks for the reply.
Is it possible to use another DNS then the one from the VPN server to use both systems.
When i run the VPN and DNSBL the NTp clock syc & dns resoler go "off" and then nothing works anymore.
Hj
-
I don't personally use a VPM like PIA, but the only way for DNSBL to function, is if the pfSense Resolver is used for the DNS requests. If the VPN uses a different DNS server, then DNSBL will not block any domains…
I think I understand the issue when running a PIA client on a device behind the pfSense system.
Does this issue exist if you use pfSense/OpenVPN system as the VPN client? If so, is the fix as simple as using PIA's DNS values in pfSense?
-
Using the DNS of the VPN server did not change the problem.
When i start running the Openvpn client "unbound" will stop together with the "ntp sync" internet. Running no client VPN everything runs well.
My goal is to run client VPN to PIA plus use the capabilities of pfblockerng dnsbl for ad blocking, trackers etc. In don,t want my IP to be shown.
-
Using the DNS of the VPN server did not change the problem.
My goal is to run client VPN to PIA plus use the capabilities of pfblockerng dnsbl for ad blocking, trackers etc. In don,t want my IP to be shown.
Thanks for the information. This is the direction I'm heading too. I have a few more things to clean up before I (re)invoke my PIA connection. I'll keep an eye on your issue.
Rick
-
I am seeing an issue where the pages with ads can take up to 20 seconds to timeout prior to loading the page…
Example: landsend.com will sit blank for 20 seconds waiting on assets.adobedtm.com, then loads immediately.
-Chad
I have been dealing with this issue also, I have found using wireshark that the VIP network request is being sent to 10.10.10.1 on port 80 (not the port that you have set) so the website waits until it times out to load the page.
A previous post mentioned that it was squid that was sending traffic to port 80. (to see the 1x1 gif you need to go to 10.10.10.1 on the port you have set.)
I have also set the VIP ip to 172.16.0.5 and the ports to 8178 and 8179 (suricata off, squid off, squidguard off) using wireshark I still see traffic going to 10.10.10.1 port 80
Also on systems that do not use the squid3 have the same delay also on windows or centos. The websites I am using to test are newegg and foxnews.
-
Hi cmpufxr,
Check your desktop DNS settings, and make sure they are only pointing to the pfSense Resolver IP address. If you have a multi-segmented LAN network, make sure to add the DNSBL allow firewall rule to allow those other subnets access to the VIP address. Make sure you can ping and browse to the DNSBL VIP, without that connectivity working properly, you will see those timeouts.
Keep in mind that there are two NAT rules that forward traffic from the VIP address on ports 80/443 to the 127.0.0.1 DNSBL Port/SSL ports.
-
Hi cmpufxr,
Check your desktop DNS settings, and make sure they are only pointing to the pfSense Resolver IP address. If you have a multi-segmented LAN network, make sure to add the DNSBL allow firewall rule to allow those other subnets access to the VIP address. Make sure you can ping and browse to the DNSBL VIP, without that connectivity working properly, you will see those timeouts.
Keep in mind that there are two NAT rules that forward traffic from the VIP address on ports 80/443 to the 127.0.0.1 DNSBL Port/SSL ports.
Dns settings are good, I have access to the VIP I can only browse to the VIP with 10.10.10.1:8178 and https:\10.10.10.1:8179 (vip address and port)
It looks like the NAT rule is not forwarding from 80 and 443 to 8178 and 8179.
Thanks, I have at least identified the issue, now to fix it.
I run a website so incoming port 80 on the wan is already forwarded. I also block all outgoing and have rules setup to allow 80 and 443 out on the LAN. I wonder if these are causing issues.
-
Ok, I found out the reason for why some websites were slow. ( newegg cnn fox landsend )
On my system I block all incoming and outgoing traffic on lan, vlan1, vlan2
When I setup pfblockerng with dnsbl I did not add the outgoing rules on the lan interface for both the dnsbl ip's.
So in the floating rule that I already had for outgoing system ports I added both the ip's for the dnsbl web server.
so the only traffic that was going out was the 80 and 443 and when the NAT rule tried to forward this traffic to 127.0.0.1 port 8178 and 8179 (this is also outgoing traffic from lan to 127.0.0.1) so I had to add 8178 and 8179 to my LAN outgoing port rules.
before the rule I could ping 10.10.10.1 but NOT browse to the 1x1 pixel gif unles I added the port (ie 10.10.10.1:8178). After I can browse and ping to 10.10.10.1 without having to add the port
I hope this helps someone else… Thank you for all your help
Thanks BBcan177 for getting me on the right track...
-
When creating the Alias are you add them into a larger group or are you leaving them as they are individual items? Is there a limitation on amount of alias you can add to a group?
Here are more DNSBL Feeds that can be used in pfBlockerNG.
(Copy and paste URLS as plain text)- Create a new alias for these.
These are not necessarily ADvert domains. So I named mine "Malicious"
hpHosts
http://hosts-file.net/download/hosts.zipSWC
http://someonewhocares.org/hosts/hostsspam404
https://spam404bl.com/blacklist.txtmalc0de
https://malc0de.com/bl/BOOTMDS (use 'Flex' state)
https://mirror1.malwaredomains.com/files/justdomainsMVPS
http://winhelp2002.mvps.org/hosts.txtMDL
http://www.malwaredomainlist.com/hostslist/hosts.txtGJTech
http://adblock.gjtech.net/?format=unix-hostsdShield_SD (They also have a conservative list available)
https://www.dshield.org/feeds/suspiciousdomains_High.txtZeus
https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist- These two feeds post full URLs, so there can be some more false positives.
Create a new Alias, and use Alexa as a recommendation.
PhishTank
https://data.phishtank.com/data/online-valid.csv.bz2OpenPhish
https://www.openphish.com/feed.txtMPatrol (You need to register - Free or Paid subscription. Use Danguardian feed)
https://lists.malwarepatrol.net-
This is a feed that I manage (as time permits)
MS_2
https://gist.githubusercontent.com/BBcan177/4a8bf37c131be4803cb2/raw -
Use this in its own Alias:
BBC_DGA (This is a large feed of DGA for the likes of Cryptolocker et al…)
http://osint.bambenekconsulting.com/feeds/dga-feed.gzBBC_C2
http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt- Use this feed in its own alias as it is updated more frequently.
So you can update it more often than once per day.
hpHosts_partial
http://hosts-file.net/hphosts-partial.aspIf users find other feeds, please post back so that others may benefit also.
Its also important to donate to the feeds provider (IP and/or Domain) as they all need support. - Create a new alias for these.
-
Did you create a group per each item that is shown below?
Few things I found out while beta testing V2
- First rule is take your time and be patient. Adding to may rules at once will casue alot of blocks you may or may not way. Wildcards are not supported, or wanted in most cases. For example adding google.com does not filter all the hosts for the domain, need to add each host that is being blocked to the filter lists
- Make sure to double check ASN's, especially for large companies that have acquired other companies, more than one ASN might exist
*** Unbound has an issue in my setup using OpenVPN connected to PIA. It does not start correctly when a network issue is sensed, em1 or openvpn interfaces flap for example. The core problem seems to be how the rc scripts handle unbound reload. They do a HUP vs reload to get around the cache being lost. Disabling "DHCP Registration" or "Static DHCP" was a workaround that allowed unbound to start correctly. - Checking "DHCP Registration" or "Static DHCP" causes other issues with unbound, this is a know issue being talked about in the DNS forum. I added all my hosts host to unbound statically as a work around
- I wanted to have my allow rules > pfb block rules > my block rules > my per interface rules. Did this by enabling "Floating Rules" and using "pfb_pass/match | pfsense pass/match | pfb_block/reject"
- If you have WIFI make sure to enable "DNSBL Firewall Rule" and select both LAN and WIFI or the NAT: Port Forward rule for 10.10.10.1 is not setup correctly. If you don't change this setting things still work but everything runs slow over wifi.
- Make sure to whitelist all the DNSBL sites FQDN via an allow aliases, some IPv4 block-lists include the IP space the DNSBL are found in
- DBSBL is basicly a man in the middle for DNS so any site that is block but running https will create warrnings. Depending on how the brower is setup it might just ignore or not display the error. CTRL+SHIFT+I on both Chrome and Firebox will bring up dev panel where you can see which pages are not loading. Chrome will show an error, Firefox will show a red slash throught the lock icon.
- Noticed a list of domains in the Alexa top 1K that serve up ads, since I don't want this I added the following custom block list to my Ads DNS Group which will remove them from any DNS group that is using the Alexa top 1K filter. To confirm make sure "Enable Alexa Whitelist" is NOT checked for this DNS group.
popcash.net www.popcash.net cdn.popcash.net outbrain.com www.outbrain.com onclickads.net www.onclickads.net googleadservices.com www.googleadservices.com adcash.com www.adcash.com popads.net www.popads.net popmyads.com www.popmyads.com- List of domains added to DNSBL Custom Domain Suppression to stop them from being blocked
goo.gl google.com www.google.com mail.google.com docs.google.com sites.google.com fonts.googleapis.com cache.google.com clients.google.com clients0.google.com clients1.google.com clients2.google.com clients3.google.com clients4.google.com clients5.google.com clients6.google.com clients7.google.com clients8.google.com clients9.google.com www.maxmind.com s3.amazonaws.com fls-na.amazon.com login.live.com redis.io pgl.yoyo.org someonewhocares.org www.thingamajob.com winhelp2002.mvps.org hosts-file.net www.hosts-file.net adaway.org sysctl.org adblock.gjtech.net www.dshield.org malwaredomainlist.com malwaredomains.com bambenekconsulting.com malwarepatrol.net zeustracker.abuse.ch malc0de.com curl.haxx.se dl.dropboxusercontent.com whois.cymru.com github.com collector-cdn.github.com pivotal.github.com cloud.github.com raw.githubusercontent.com raw.github.com stopforumspam.com www.stopforumspam.com sourceforge.net www.sourceforge.net iweb.dl.sourceforge.net chase.com www.chase.com mint.com www.mint.com americanexpress.com www.americanexpress.com online.americanexpress.com linuxquestions.org www.linuxquestions.org optimizely.com www.optimizely.com api.optimizely.com cdn.optimizely.com cdn2.optimizely.com cdn3.optimizely.com slashdot.org www.slashdot.org ebay.com www.ebay.com rover.ebay.com srx.main.ebayrtm.com openbl.org www.openbl.org www.us.openbl.org delta.com www.delta.com aa.com www.aa.com cruisesonly.com www.cruisesonly.com ripe.net www.ripe.net weather.com www.weather.com lacnic.net www.lacnic.net tvrage.com services.tvrage.com www.tvrage.com publicbt.com device.maxmind.com www.boingo.com xda-developers.com www.xda-developers.com forum.xda-developers.com opengapps.org download.mono-project.com-
Ads DNS Group
-
Update Frequency: Once a Day
-
Enable Alexa Whitelist: NOT Enabled
-
Add Custom Block List noted above
http://pgl.yoyo.org/adservers/serverlist.php?hostformat=hosts&mimetype=plaintextoup yoyo http://hosts-file.net/ad_servers.txt hphosts_ats https://adaway.org/hosts.txt adaway http://sysctl.org/cameleon/hosts syctrl http://adblock.gjtech.net/?format=unix-hosts gjtech-
Privacy Fraud DNS Group
-
Update Frequency: Once a Day
-
Enable Alexa Whitelist: Enabled
https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt disconnect_basic http://hosts-file.net/fsa.txt hphost_fsa http://hosts-file.net/hjk.txt hphost_hjk http://hosts-file.net/pha.txt hphost_psh https://www.dshield.org/feeds/suspiciousdomains_High.txt dshield_sdh-
Malware Exploit DNS Group
-
Update Frequency: Once a Day
-
Enable Alexa Whitelist: Enabled
https://s3.amazonaws.com/lists.disconnect.me/simple_malvertising.txt disconnect_malvertising http://www.malwaredomainlist.com/hostslist/hosts.txt malwaredomainlist http://mirror1.malwaredomains.com/files/justdomains malwaredomains https://s3.amazonaws.com/lists.disconnect.me/simple_malware.txt disconnect_malware http://hosts-file.net/emd.txt hphosts_emd http://hosts-file.net/exp.txt hphosts_exp http://hosts-file.net/mmt.txt hphosts_mmt https://lists.malwarepatrol.net/cgi/getfile?receipt=f1442112770&product=8&list=dansguardian malwarepatrol https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist zeustracker https://malc0de.com/bl/BOOT malc0de-
SPAM DNS Group
-
Update Frequency: Once a Day
-
Enable Alexa Whitelist: Enabled
http://hosts-file.net/grm.txt hphost_grm http://hosts-file.net/hfs.txt hphost_hfs https://spam404bl.com/blacklist.txt spam_404-
Malicious DNS Group
-
Update Frequency: Once a Day
-
Enable Alexa Whitelist: Enabled
http://hosts-file.net/hphosts-partial.txt hphost_partial http://winhelp2002.mvps.org/hosts.txt mvps_hosts http://someonewhocares.org/hosts/hosts SomeoneWhoCares https://gist.githubusercontent.com/BBcan177/4a8bf37c131be4803cb2/raw BBcan177-
BambenekConsulting DNS Group
-
Update Frequency: Once a Day
-
Enable Alexa Whitelist: Enabled
http://osint.bambenekconsulting.com/feeds/dga-feed.gz bambenek_dga http://osint.bambenekconsulting.com/feeds/c2-dommasterlist.txt bambenek_c2Thanks
Tony M.** -
When creating the Alias are you add them into a larger group or are you leaving them as they are individual items? Is there a limitation on amount of alias you can add to a group?
That is personal preference… There is no limit to either method...
I prefer to create an "ADs" alias, "Malicious" Alias etc instead of making an Alias with a single feed in it.... The other post that you linked has the "DNS Group Name" listed also for your reference...
-
Getting error when rebooting with latest PFblockerng not certain what the error is. Any help on how to fix would be appreciated.
Thanks
12-26-15 11:48:19 [ PF was wedged/busy and has been reset.]
12-26-15 11:48:40 [ There were error(s) loading the rules: pfctl: DIOCXCOMMIT: Device busy - The line in question reads [0]: ]
12-26-15 11:49:15 [ Unresolvable source alias pfB_Top_v4 for rule pfB_Top_v4 auto rule]
12-26-15 11:49:17 [ Unresolvable source alias pfB_Top_v6 for rule pfB_Top_v6 auto rule]
12-26-15 11:49:19 [ Unresolvable destination alias pfB_Top_v4 for rule pfB_Top_v4 auto rule]
12-26-15 11:49:21 [ Unresolvable destination alias pfB_Top_v6 for rule pfB_Top_v6 auto rule] -
That error is normal. It's because when you reboot pfsense, pfblock needs some time to make the alias and the firewall complains about it.
