PfBlockerNG configuration for a newbie :)

[belgiumrom]

Hi again,

@RonpfS

Removing steepto.com from TLD blocking and force reloading does not do the trick…

@BBcan177

My LAN is as follows:
Pfsense as gateway/firewall/dhcp/pfblocker/squid/squidguard/dns resolver in forwarding mode. There is a clamd embedded in squid proxy though. No other antivirus inside LAN.
Another Ubuntu 16.04 server with bind9 dns server installed but this is forwarding all DNS requests from LAN to the pfsense DNS. All LAN devices are Linux mint and they are getting DNS servers automatically along with a fixed IP address from pfsense dhcp in this order: ubuntu first, pfsense second.
True, I don't use the 192.168.x.x range in my LAN, I use 10.10.x.x, that's why I set the 192.168.0.1 as VIP.

Now, I come to realise there are some weird behaviours in my LAN.
To start, if I browse directly to www.steepto.com from any device within LAN, I get the 1x1 pixel page. Up to here we're good.

Doing whois steepto.com from LAN device, I get:

user@lan_machine ~ $ whois steepto.com
  Domain Name: STEEPTO.COM
  Registry Domain ID: 1848412964_DOMAIN_COM-VRSN
  Registrar WHOIS Server: whois.godaddy.com
  Registrar URL: http://www.godaddy.com
  Updated Date: 2017-10-31T16:39:37Z
  Creation Date: 2014-02-27T11:22:59Z
  Registry Expiry Date: 2018-02-27T11:22:59Z
  Registrar: GoDaddy.com, LLC
  Registrar IANA ID: 146
  Registrar Abuse Contact Email: abuse@godaddy.com
  Registrar Abuse Contact Phone: 480-624-2505
  Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
  Domain Status: clientRenewProhibited https://icann.org/epp#clientRenewProhibited
  Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
  Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
  Name Server: NS.STEEPTO.COM
  Name Server: NS3.STEEPTO.COM
  DNSSEC: unsigned
  URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/

Last update of whois database: 2018-01-03T23:03:29Z <<<

For more information on Whois status codes, please visit https://icann.org/epp

NOTICE: The expiration date displayed in this record is the date the
registrar's sponsorship of the domain name registration in the registry is
currently set to expire. This date does not necessarily reflect the expiration
date of the domain name registrant's agreement with the sponsoring
registrar.  Users may consult the sponsoring registrar's Whois database to
view the registrar's reported date of expiration for this registration.

TERMS OF USE: You are not authorized to access or query our Whois
database through the use of electronic processes that are high-volume and
automated except as reasonably necessary to register domain names or
modify existing registrations; the Data in VeriSign Global Registry
Services' ("VeriSign") Whois database is provided by VeriSign for
information purposes only, and to assist persons in obtaining information
about or related to a domain name registration record. VeriSign does not
guarantee its accuracy. By submitting a Whois query, you agree to abide
by the following terms of use: You agree that you may use this Data only
for lawful purposes and that under no circumstances will you use this Data
to: (1) allow, enable, or otherwise support the transmission of mass
unsolicited, commercial advertising or solicitations via e-mail, telephone,
or facsimile; or (2) enable high volume, automated, electronic processes
that apply to VeriSign (or its computer systems). The compilation,
repackaging, dissemination or other use of this Data is expressly
prohibited without the prior written consent of VeriSign. You agree not to
use electronic processes that are automated and high-volume to access or
query the Whois database except as reasonably necessary to register
domain names or modify existing registrations. VeriSign reserves the right
to restrict your access to the Whois database in its sole discretion to ensure
operational stability.  VeriSign may restrict or terminate your access to the
Whois database for failure to abide by these terms of use. VeriSign
reserves the right to modify these terms at any time.

The Registry database contains ONLY .COM, .NET, .EDU domains and
Registrars.

Doing whois steepto.com from pfsense, I get:

[2.4.2-RELEASE][root@pfsense]/root: whois steepto.com
% IANA WHOIS server
% for more information on IANA, visit http://www.iana.org
% This query returned 1 object

refer:        whois.verisign-grs.com

domain:      COM

organisation: VeriSign Global Registry Services
address:      12061 Bluemont Way
address:      Reston Virginia 20190
address:      United States

contact:      administrative
name:        Registry Customer Service
organisation: VeriSign Global Registry Services
address:      12061 Bluemont Way
address:      Reston Virginia 20190
address:      United States
phone:        +1 703 925-6999
fax-no:      +1 703 948 3978
e-mail:      info@verisign-grs.com

contact:      technical
name:        Registry Customer Service
organisation: VeriSign Global Registry Services
address:      12061 Bluemont Way
address:      Reston Virginia 20190
address:      United States
phone:        +1 703 925-6999
fax-no:      +1 703 948 3978
e-mail:      info@verisign-grs.com

nserver:      A.GTLD-SERVERS.NET 192.5.6.30 2001:503:a83e:0:0:0:2:30
nserver:      B.GTLD-SERVERS.NET 192.33.14.30 2001:503:231d:0:0:0:2:30
nserver:      C.GTLD-SERVERS.NET 192.26.92.30 2001:503:83eb:0:0:0:0:30
nserver:      D.GTLD-SERVERS.NET 192.31.80.30 2001:500:856e:0:0:0:0:30
nserver:      E.GTLD-SERVERS.NET 192.12.94.30 2001:502:1ca1:0:0:0:0:30
nserver:      F.GTLD-SERVERS.NET 192.35.51.30 2001:503:d414:0:0:0:0:30
nserver:      G.GTLD-SERVERS.NET 192.42.93.30 2001:503:eea3:0:0:0:0:30
nserver:      H.GTLD-SERVERS.NET 192.54.112.30 2001:502:8cc:0:0:0:0:30
nserver:      I.GTLD-SERVERS.NET 192.43.172.30 2001:503:39c1:0:0:0:0:30
nserver:      J.GTLD-SERVERS.NET 192.48.79.30 2001:502:7094:0:0:0:0:30
nserver:      K.GTLD-SERVERS.NET 192.52.178.30 2001:503:d2d:0:0:0:0:30
nserver:      L.GTLD-SERVERS.NET 192.41.162.30 2001:500:d937:0:0:0:0:30
nserver:      M.GTLD-SERVERS.NET 192.55.83.30 2001:501:b1f9:0:0:0:0:30
ds-rdata:    30909 8 2 E2D3C916F6DEEAC73294E8268FB5885044A833FC5459588F4A9184CFC41A5766

whois:        whois.verisign-grs.com

status:      ACTIVE
remarks:      Registration information: http://www.verisigninc.com

created:      1985-01-01
changed:      2017-10-05
source:      IANA

Domain Name: STEEPTO.COM
  Registry Domain ID: 1848412964_DOMAIN_COM-VRSN
  Registrar WHOIS Server: whois.godaddy.com
  Registrar URL: http://www.godaddy.com
  Updated Date: 2017-10-31T16:39:37Z
  Creation Date: 2014-02-27T11:22:59Z
  Registry Expiry Date: 2018-02-27T11:22:59Z
  Registrar: GoDaddy.com, LLC
  Registrar IANA ID: 146
  Registrar Abuse Contact Email: abuse@godaddy.com
  Registrar Abuse Contact Phone: 480-624-2505
  Domain Status: clientDeleteProhibited https://icann.org/epp#clientDeleteProhibited
  Domain Status: clientRenewProhibited https://icann.org/epp#clientRenewProhibited
  Domain Status: clientTransferProhibited https://icann.org/epp#clientTransferProhibited
  Domain Status: clientUpdateProhibited https://icann.org/epp#clientUpdateProhibited
  Name Server: NS.STEEPTO.COM
  Name Server: NS3.STEEPTO.COM
  DNSSEC: unsigned
  URL of the ICANN Whois Inaccuracy Complaint Form: https://www.icann.org/wicf/

Last update of whois database: 2018-01-03T23:04:28Z <<<

Domain Name: STEEPTO.COM
Registrar URL: http://www.godaddy.com
Registrant Name: Bedigital Corporation
Registrant Organization: Bedigital Corporation
Name Server: NS.STEEPTO.COM
Name Server: NS3.STEEPTO.COM
DNSSEC: unsigned

For complete domain details go to:
http://who.godaddy.com/whoischeck.aspx?domain=STEEPTO.COM

The data contained in GoDaddy.com, LLC's WhoIs database,
while believed by the company to be reliable, is provided "as is"
with no guarantee or warranties regarding its accuracy.  This
information is provided for the sole purpose of assisting you
in obtaining information about domain name registration records.
Any use of this data for any other purpose is expressly forbidden without the prior written
permission of GoDaddy.com, LLC.  By submitting an inquiry,
you agree to these terms of usage and limitations of warranty.  In particular,
you agree not to use this data to allow, enable, or otherwise make possible,
dissemination or collection of this data, in part or in its entirety, for any
purpose, such as the transmission of unsolicited advertising and
and solicitations of any kind, including spam.  You further agree
not to use this data to enable high volume, automated or robotic electronic
processes designed to collect or compile this data for any purpose,
including mining this data for your own personal or commercial purposes.

Please note: the registrant of the domain name is specified
in the "registrant" section.  In most cases, GoDaddy.com, LLC
is not the registrant of domain names listed in this database.

But…

Doing a "dig steepto.com" from any machine within LAN, I get (please note the answering DNS is Ubuntu and the IP address returned is my VIP, 182.168.0.1):

user@lan_machine ~ $ dig steepto.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> steepto.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 8485
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 13, ADDITIONAL: 16

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;steepto.com. IN A

;; ANSWER SECTION:
steepto.com. 9 IN A 192.168.0.1

;; AUTHORITY SECTION:
com. 74130 IN NS b.gtld-servers.net.
com. 74130 IN NS e.gtld-servers.net.
com. 74130 IN NS c.gtld-servers.net.
com. 74130 IN NS h.gtld-servers.net.
com. 74130 IN NS k.gtld-servers.net.
com. 74130 IN NS a.gtld-servers.net.
com. 74130 IN NS d.gtld-servers.net.
com. 74130 IN NS i.gtld-servers.net.
com. 74130 IN NS l.gtld-servers.net.
com. 74130 IN NS f.gtld-servers.net.
com. 74130 IN NS m.gtld-servers.net.
com. 74130 IN NS j.gtld-servers.net.
com. 74130 IN NS g.gtld-servers.net.

;; ADDITIONAL SECTION:
a.gtld-servers.net. 7499 IN A 192.5.6.30
a.gtld-servers.net. 5006 IN AAAA 2001:503:a83e::2:30
b.gtld-servers.net. 17421 IN A 192.33.14.30
c.gtld-servers.net. 77756 IN A 192.26.92.30
d.gtld-servers.net. 79 IN A 192.31.80.30
e.gtld-servers.net. 2470 IN A 192.12.94.30
f.gtld-servers.net. 990 IN A 192.35.51.30
g.gtld-servers.net. 3781 IN A 192.42.93.30
h.gtld-servers.net. 82835 IN A 192.54.112.30
i.gtld-servers.net. 8375 IN A 192.43.172.30
j.gtld-servers.net. 17421 IN A 192.48.79.30
k.gtld-servers.net. 2470 IN A 192.52.178.30
l.gtld-servers.net. 3810 IN A 192.41.162.30
l.gtld-servers.net. 1840 IN AAAA 2001:500:d937::30
m.gtld-servers.net. 17426 IN A 192.55.83.30

;; Query time: 0 msec
;; SERVER: 10.10.10.100#53(10.10.10.100)
;; WHEN: Wed Jan 03 18:07:46 EST 2018
;; MSG SIZE  rcvd: 544

Doing "dig @pfsense steepto.com" (directing dig to pfsense DNS) from the same LAN machine, I get:

user@lan_machine ~ $ dig @pfsense steepto.com

; <<>> DiG 9.10.3-P4-Ubuntu <<>> @pfsense steepto.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 47814
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;steepto.com. IN A

;; ANSWER SECTION:
steepto.com. 60 IN A 192.168.0.1

;; Query time: 0 msec
;; SERVER: 10.10.10.1#53(10.10.10.1)
;; WHEN: Wed Jan 03 18:08:13 EST 2018
;; MSG SIZE  rcvd: 56

Please notice again the IP address returned for steepto.com is my VIP.

Other tools are reporting the same result:

user@lan_machine ~ $ host steepto.com
steepto.com has address 192.168.0.1

user@lan_machine ~ $ nslookup steepto.com
Server: 10.10.10.100
Address: 10.10.10.100#53

Non-authoritative answer:
Name: steepto.com
Address: 192.168.0.1

Redirecting DNS query to pfsense DNS:

user@lan_machine ~ $ nslookup steepto.com pfsense
Server: pfsense
Address: 10.10.10.1#53

Name: steepto.com
Address: 192.168.0.1

I read somewhere that pfblockerng does not like running together with squid/squidguard. Could this be my issue?

Thanks for your suggestions.

[BBcan177]

I asked to run a "whois example.com" but I guess my brain was somewhere else when I wrote that  :o

Should run a "host -t A" or nslookup or dig or ping command to see if the Domains reply with the DNSBL VIP…

So I see from your commands that its replying with the DNSBL VIP.... so that part is working... You just need to see if the proxy or something else is causing your issue?  Do you have an AV solution that is protecting for DNS sinkhole behaviour?

[belgiumrom]

LOL, yes, it sounded strange in my brain too…

Yes, there is the clamd antivirus that's embedded into squid proxy, and it's enabled but I fail to see why that would interfere with dnsbl.
Funny thing, in my endeavours of blocking this steepto.com domain, I was actually blocking it in squid access control lists too under "blacklist" but that didn't make any difference so I removed it.
To test, I disabled first clamd. No difference. Then I disabled squid proxy and squidguard. No difference as well.

...and yes, my pfblocker is working beautifully except this stubborn domain...

I am stumped again...

[belgiumrom]

I posted a pic with all the services I have enabled in pfsense. Do you see anything that could intefere?

Also, when the lists are reloaded, I see that the hard limit of domains is overpassed, could that be an issue too?

–----------------------------------------
Assembling database... completed
Executing TLD
TLD analysis....xxxxxxxxxxx completed
** TLD Domain count exceeded. [ 400000 ] All subsequent Domains listed as-is **
Finalizing TLD…  completed

[belgiumrom]

Did you guys do anything to the lists?!?!?

My pfblocker started blocking ads from steepto.com in filmehd.net all of a sudden without me doing anything…

Now ads in facebook.com started showing again... lol

Am I doing something wrong? LOL

[RonpfS]

@belgiumrom:

Did you guys do anything to the lists?!?!?

You have to ask that question to the "guys" who maintain the lists.

@belgiumrom:

Also, when the lists are reloaded, I see that the hard limit of domains is overpassed, could that be an issue too?

–----------------------------------------
Assembling database... completed
Executing TLD
TLD analysis....xxxxxxxxxxx completed
** TLD Domain count exceeded. [ 400000 ] All subsequent Domains listed as-is **
Finalizing TLD…  completed

That's because you don't have enough memory to get a complete TLD set. So from Cron update to Cron Update, some more domains are converted to TLD and that may demand different whitelisting.

So look at the logs to see what lists were downloaded when steepto.com became blocked.

Try a Force Reload DNSBL to see if things change.

Lower the total number of DNSBL entries by removing some big lists.

[belgiumrom]

Thanks RonpfS for your suggestions. Will try tonight.

I was asking the "guys" here because I have some lists made or maintained by BBcan177.

…and you are right, I am running pfsense on a tiny microcomputer with only 4GB of RAM, but that's going to change soon, will upgrade to 8GB. I only found out about pfblocker after I purchased the hardware.

[BBcan177]

@belgiumrom:

Thanks RonpfS for your suggestions. Will try tonight.

I was asking the "guys" here because I have some lists made or maintained by BBcan177.

…and you are right, I am running pfsense on a tiny microcomputer with only 4GB of RAM, but that's going to change soon, will upgrade to 8GB. I only found out about pfblocker after I purchased the hardware.

TLD requires more memory as Unbound creates a pointer in memory for each Zone entry. To prevent Unbound from consuming all of the available memory and crashing the box, I have defined conservative Zone limit according to the amount of memory available.

So as soon as you reach the the max TLD limit, no further Zones are created and the balance of the Domains will be blocked as per the explicit domain entry and not the whole sub-domain as TLD is intended to accomplish.

So, as TLD is processing, this domain might have been processed before the TLD limit and then added as a Zone. And at times it might be processed after the TLD limit at which time it won't be blocking the whole sub-domain.

So you either need more memory, or put the most important Domains to be processed by TLD first so that they will be added as a Zone before the TLD limit is reached.

[belgiumrom]

Thanks for suggestions BBcan177.
I upgraded to 8 GB the other day, but steepto.com still shows on that filmehd.net website…
I apologize but I didn't understand much of your explanation other than upgrading RAM will help... lol, too technical, and my knowledge is somewhat limited in TLD (learning now).
One good news, there is no hard limit message anymore when TLD list is processed.

[belgiumrom]

…looking back at comments, I found out this domain blocked as well, somewhat similar config... (192.168.0.1 being my VIP)

grep steepto.com /var/db/pfblockerng/dnsbl/.txt /var/db/pfblockerng/dnsblorig/.orig /var/unbound/pfb_dnsbl.conf /usr/local/pkg/pfblockerng/dnsbl_tld

/var/db/pfblockerng/dnsbl/easy_list.txt:local-data: "steepto.com 60 IN A 192.168.0.1"
/var/db/pfblockerng/dnsblorig/ad_servers.orig:127.0.0.1 imgg.steepto.com
/var/db/pfblockerng/dnsblorig/easy_list.orig:||steepto.com^$third-party
/var/db/pfblockerng/dnsblorig/easylist.orig:||steepto.com^$third-party
/var/db/pfblockerng/dnsblorig/hpHosts.orig:127.0.0.1 imgg.steepto.com
/var/db/pfblockerng/dnsblorig/hpHosts_ads.orig:127.0.0.1 imgg.steepto.com
/var/unbound/pfb_dnsbl.conf:local-zone: "steepto.com" redirect local-data: "steepto.com 60 IN A 192.168.0.1"