BIND filter-aaaa

AberDino

I copped the script to /var/unbound.

That's the bit I was missing. All working beautifully now, thank you!

Gertjan

Same thing here : completely forgot about that one !
But, as I'm using tunnelbroker.he.net also, and need to block some - not only Netflix, IPv6 AAAA"s.
Example : when I visit my ISP main web site using AAAA, it's often just doesn't work well.

@johnpoz said in BIND filter-aaaa:

Grab that script that is linked too, put in a file called no-aaaa.py and copy that file to your /var/unbound dir

I still think it's needed to copy the python script over to the /var/unbound and /var/unbound/var/unbound - the lather has to be created on start, and has to have the right owner, unbound.

As of today - using 2.4.5 - it's still working :

Btw I'm using 'ln' to link copy to populate /var/unbound/var/unbound from the copy in /var/unbound

#!/bin/sh

# https://forum.pfsense.org/index.php?topic=134352.msg737158#msg737158
echo "Inserting netflix-no-aaaa.py ...."

#make sure the directory for the python libraries is in the chroot
mkdir -p /var/unbound/usr/local/lib/python2.7

#link the actual python library directory to the chroot's directory
mount -t nullfs /usr/local/lib/python2.7 /var/unbound/usr/local/lib/python2.7

#copy the python script to the /var/unbound directory so 
#unbound-checkconf can find it
rm -f /var/unbound/netflix-no-aaaa.py
cp /root/netflix-no-aaaa.py /var/unbound/netflix-no-aaaa.py

#make sure unbound can read it
chown unbound:unbound /var/unbound/netflix-no-aaaa.py

#create a /var/unbound directory in the /var/unbound directory so that 
#unbound can find the script
mkdir -p /var/unbound/var/unbound

#copy the python module into the /var/unbound/var/unbound directory under the chroot #directory
rm -f /var/unbound/var/unbound/netflix-no-aaaa.py
ln -s /var/unbound/netflix-no-aaaa.py /var/unbound/var/unbound/netflix-no-aaaa.py

echo "Done ...."

The thing is set up when pfSense starts using an earlyshellcmd command :

/root/unbound-p.sh is the script shown above.
It's placed in /root, together with the /root/netflix-no-aaaa.py script.

johnpoz

It was working without that..

Gertjan

Do you mean that putting in place the "netflix-no-aaaa.py" is enough ?

No more coping the script to /var/unbound/var/unbound/
and no more mounting of the /usr/local/lib/python2.7 dir ?

johnpoz

That is all I did - then you see it in the new 2.4.5 gui and bobs your uncle.

Gertjan

Ok, nice. That cleans up things.

Will include a "Note" with the source of that python file, to complete my "in case of total rebuild" procedure.

geekmug

@AberDino said in BIND filter-aaaa:

Any thoughts on how I could get this working again?

@Gertjan said in BIND filter-aaaa:

I don't use bind myself - not for pfSense, that is, but have one question back : bind (named) isn't chrooted on some location ?

I just upgraded and ran into this myself. Yes, pfSense uses /cf/named as the chroot. The issue is that /usr/local/lib/named is a new directory that holds the plugins that needs to get copied into /cf/named/usr/local/lib/named. I've pushed up a pull request to fix this:

https://github.com/pfsense/FreeBSD-ports/pull/816

Until the pull request gets merged and an updated package is published, you could patch the bind.inc locally, and then run /etc/rc.packages bind post-install to regenerate the chroot.

bruor

It would be fantastic if the filtering of AAAA could be added as a native script whose list of domains to filter could be edited in the interface. This works great but I worry about having to recreate this if I ever have to restore from a backup in a few years.

AndrewZ

@johnpoz said in BIND filter-aaaa:

Grab that script that is linked too, put in a file called no-aaaa.py and copy that file to your /var/unbound dir

Upgraded to 2.4.5_1 and noticed that this file [that was there for quite some time] - no-aaaa.py - is gone. Copied it over, rebooted and noticed that the file is gone again. Finally, realized that /var/ is mounted in RAM. So, where this file should be stored?

jimp

Would be safe in /root or /conf or a subdirectory of those.

johnpoz

Not sure what your taking about on 2.4.5p1 and file is right where it was in /var/unbound

2.4.5-RELEASE][admin@sg4860.local.lan]/root: cd /var/unbound
[2.4.5-RELEASE][admin@sg4860.local.lan]/var/unbound: ls -la
total 84
drwxr-xr-x   5 unbound  unbound  1024 Jun 16 05:37 .
drwxr-xr-x  31 root     wheel     512 Jun  2 16:54 ..
-rw-r--r--   1 root     unbound   185 Jun 14 07:14 access_lists.conf
drwxr-xr-x   2 unbound  unbound   512 Jun  2 16:54 conf.d
drwxr-xr-x   2 root     unbound   512 Mar 28 07:52 dev
-rw-r--r--   1 root     unbound   698 Jun 14 07:14 dhcpleases_entries.conf
-rw-r--r--   1 root     unbound    65 Jun 14 07:14 domainoverrides.conf
-rw-r--r--   1 root     unbound  6838 Jun 14 07:14 host_entries.conf
-rw-r--r--   1 root     unbound  1404 Mar 28 07:51 no-aaaa.py
-rw-r--r--   1 root     unbound     0 Jun 16 15:00 pfb_dnsbl.conf
-rw-r--r--   1 root     unbound   300 Sep 17  2017 remotecontrol.conf
-rw-r--r--   1 unbound  unbound   758 Jun 16 05:37 root.key
-rw-r--r--   1 unbound  unbound   759 Dec  4  2019 root.key.67289-2
-rw-r--r--   1 root     unbound  3369 Feb 28  2019 sslcert.crt
-rw-------   1 root     unbound     0 Feb 28  2019 sslcert.key
-rw-r--r--   1 root     unbound  5297 Jun 14 07:14 unbound.conf
-rw-r-----   1 unbound  unbound  2455 Sep 17  2017 unbound_control.key
-rw-r-----   1 unbound  unbound  1330 Sep 17  2017 unbound_control.pem
-rw-r-----   1 unbound  unbound  2459 Sep 17  2017 unbound_server.key
-rw-r-----   1 unbound  unbound  1318 Sep 17  2017 unbound_server.pem
drwxr-xr-x   3 root     unbound   512 Mar 28 07:52 usr
[2.4.5-RELEASE][admin@sg4860.local.lan]/var/unbound: cat no-aaaa.py 
def init(id, cfg):
    return True

def deinit(id):
    return True

def inform_super(id, qstate, superqstate, qdata):
    return True

domains = [
    "netflix.com.",
    "nflxso.net.",
]

def operate(id, event, qstate, qdata):
    if event == MODULE_EVENT_NEW or event == MODULE_EVENT_PASS:
        if qstate.qinfo.qtype != RR_TYPE_AAAA:
            qstate.ext_state[id] = MODULE_WAIT_MODULE
            return True

        for domain in domains:
            if qstate.qinfo.qname_str == domain or qstate.qinfo.qname_str.endswith("." + domain):
                msg = DNSMessage(qstate.qinfo.qname_str, RR_TYPE_A, RR_CLASS_IN, PKT_QR | PKT_RA | PKT_AA)
                if not msg.set_return_msg(qstate):
                    qstate.ext_state[id] = MODULE_ERROR
                    return True
                # We don't need validation, result is valid
                qstate.return_msg.rep.security = 2
                qstate.return_rcode = RCODE_NOERROR
                qstate.ext_state[id] = MODULE_FINISHED
                log_info("no-aaaa: blocking AAAA request for %s" % qstate.qinfo.qname_str)
                return True

        qstate.ext_state[id] = MODULE_WAIT_MODULE
        return True

    if event == MODULE_EVENT_MODDONE:
        qstate.ext_state[id] = MODULE_FINISHED
        return True

AndrewZ

I see something different on my system (apu2)...
Then I realized that I have Use memory file system for /tmp and /var set. Now all is clear. It looks like I never rebooted my router after the previous upgrade!

AndrewZ

@jimp said in BIND filter-aaaa:

Would be safe in /root or /conf or a subdirectory of those.

OK, stored it in /conf and created an earlyshellcmd script (similar to the old one) to restore the file from the persistent location. Solved!

mikekoke

With the new version of pfblocker if you have enabled the python mode you can no longer select no-aaa in Unbound. How can the ipv6 removal script be integrated with pfblocker?

Gertjan

Hummm.
Dmnd.
You're right.

"no-aaaa" is a strange beast.
The idea that unbound can use external python script to 'filter' DNS traffic for unbound is what is pfBlockerNG V3 all about.

Comparing no-aaaa.py and pfb_unbound.py shows similarities.

no-aaaa is small.
There is a table with domain names.
And there is that small function that does the work : "operate".
I'm not a python expert, but I can read it.

Right now, our no-aaaa stops if the request isn't an 'AAAA' type.

This is what the first 'if' test does :

if qstate.qinfo.qtype != RR_TYPE_AAAA:

it should be inverted.
This way, execution continues with comparing the DNS request with the domains stored in the domains[] array.

To make a long story short :

# DNSBL Validation for specific RR_TYPES only
   if pfb['python_blacklist'] and qstate is not None and qstate.qinfo.qtype is not None and qstate.qinfo.qtype in pfb['rr_types']:
       q_name_original = get_q_name_qstate(qstate)
       q_type = qstate.qinfo.qtype

       if q_type == RR_TYPE_AAAA:
           for domain in domains:
               if qstate.qinfo.qname_str == domain or qstate.qinfo.qname_str.endswith("." + domain):
                   msg = DNSMessage(qstate.qinfo.qname_str, RR_TYPE_A, RR_CLASS_IN, PKT_QR | PKT_RA | PKT_AA)
                   if not msg.set_return_msg(qstate):
                       qstate.ext_state[id] = MODULE_ERROR
                       return True
                   # We don't need validation, result is valid
                   qstate.return_msg.rep.security = 2
                   qstate.return_rcode = RCODE_NOERROR
                   qstate.ext_state[id] = MODULE_FINISHED
                   log_info("[pfBlockerNG] : no-aaaa: blocking AAAA request for %s" % qstate.qinfo.qname_str)
                   return True

   # Create list of Domain/CNAMES to be evaluated

The first and last line are comments are already present in the code.
Just replace / place what's in between them.
The 'operate' function is at the bottom.

edit : python is tabindent sensible.

Also, just above the 'operate' function, add your domains array :

domains = [
    "netflix.com.",
    "nflxso.net.",
    "nflxext.com.",
    "www.netflix.com.",
    "assets.nflxext.com.",
    "scdn.nflximg.net.",
    "secure.netflix.com.",
    "www2-ext-s.nflximg.net.",
    "so-s.nflximg.net.",
]

I advise strongly to to make a backup of your pfb_unbound.py file.
Keep in mind : for every domain that is AAAA filtered, an 'A' is retrieved, and no more pfblocker python code - the rest of the operate function, gets executed.
Until proven wrong, this breaks everything .

edit : but think about it : the domains listed in our domains array are sites we WANT to visit - and only using 'A', NOT 'AAAA'. No real need - for me - to log or do other things with the domains that are (white !) listed using the domains [].

The log line shows that something is working :

Now me hoping someone more python minded steps in.

The domain array should be lifted into the pfBlockerNG GUI.

I don't even now if this kind of scripting belongs into pfBlokcerNG ....

And why can't there be 2 python scripts : the already known no-aaaa.py and the new kind on the block : pfb_unbound.py ?
Why should we have to choose ? Is this an unbound limitation ? A pfSense limitation ?
For more edits, I'll continue to get home first. Right now, people start to wake up all arround me, and I'm exploding the DNS way to often == "no internet". And me learing python in the proces is not a good mix.

edit : I'm using no-aaaa because I have to use tunnel.he.net for my IPv6 access.
My ISP doesn't know what IPv6 is - and the IPv6 they can propose is a #@!^@!& broken simple /64. My ISP - the biggest in Europe (?) doesn't quiet understand that companies can have more then one LAN .....

Without no-aaaa, my network setup 'prefers' IPv6 if available, and it is as I'm using tunnel.he.net for my IPv6 access. Netflix is also IPv6-ready. Netflix doesn't want to stream over tunnel.he.net because it's considers I'm hiding myself as what happens when I would use a typical VPN service. Content fails to stream .... That's why no-aaaa exists.

Netflix works now for me using the patched pfb_unbound.py file

edit (again) :
I didn't notice this before :
Where I am right now, @work, I never watch Netflix.
I restarted pfSense with pfBlocker completely disabled.
My browser couldn't even connect to ww.netflix.com. That's new. Before, I could login, see the screen with series and movies, only the actual streaming wouldn't work as it would use IPv6 etc.
With pfBlocker enabled - same thing - netflix isn't accessible, but the first test is needed to be sure the netflix domain or IP's is not in some of the feeds I use.

With the patched code above, Netflix is accessible and works.

Gertjan

Added to the above :

This : /var/unbound/unbound.conf :

# Python Module
python:
python-script: no-aaaa.py
python-script: pfb_unbound.py

would be nice.
No need to edit @BBcan177 's python script !

Guess what : unbound, the version we use - only accepts ONE python module. Not multiple modules.

When I modify the config generating code so I obtain an unbound.conf as shown above, only the python script "no-aaaa.py" gets loaded. Not the second "pfb_unbound.py".

Accoring to :

See the last paragraph here https://nlnetlabs.nl/documentation/unbound/unbound.conf/ - it should be possible.
Our 2.4.5-p1 unbound version is probably to old.

mikekoke

I hope the next version of pfsense will integrate the functionality of using more than one python script having an updated version of unbound.

Gertjan

@mikekoke said in BIND filter-aaaa:

I hope the next version o

The next version of pfBlockerNG-devel, probably the upcoming 3.0.0_4 will have the "No AAAA" build in.
No more script files to manage. Just enter your list with host names that won't work well using IPv6, and you're done :

mikekoke

@gertjan
Great news as soon as I can I will update pfblocker, can I know which domains you have added to the list?

Gertjan

https://forum.netgate.com/topic/159016/pfblockerng-devel-v3-0-0_5

@mikekoke said in BIND filter-aaaa:

which domains

Normally : none.
Read https://forum.netgate.com/topic/118566/netflix-and-he-net-tunnel-fixed-using-unbound-python-module