How to block entire domains?



  • Hello guys,

    I spent the better part of the day trying to block entire domains at the perimeter from within pfsense but I am not having so much success.

    Basically, let say I want to block facebook.com, I could issue a nslookup (or host or whois) command and get the IP address behind the facebook.com domain (sent from my DNS I guess?)

    [workstation-user@workstation bashscript]# nslookup facebook.com
    Server:		192.168.0.100
    Address:	192.168.0.100#53
    
    Non-authoritative answer:
    Name:	facebook.com
    Address: 31.13.69.197
    

    The problem is that if I block the IP address reported by the above command, sooner than later facebook.com will be accessible via another IP address because they own more than a single IP..  So I will constantly end up in the terminal looking up their IP's to add them to pfsense aliases (I did for months, it was unmanageable and a literal nightmare).

    Looking up for a scripted solution, I ended up on a thread of this forum where the OP was suggested to use a combination of tools to get all IP addresses and their range under a single domain.  Basically,

    1.  use nslookup to lookup the IP address for the domain name as resolved by the DNS server
    2. use "whois -h whois.radb.net $IP" using the IP previously found to get the AS number (ASN) associated with the domain to block
    3. use "whois -h $whoisdb – '-i origin $ASN'" to get the entire range if IP's associated to the previously found ASN.

    Im not sure if this is a good way of doing this or even if this is smart.  On paper it works well but in real use its another story.....  Trying with for example oracle.com:

    [workstation-user@workstation bashscript]$ nslookup oracle.com | grep 'Address: ' | tr -d 'Address: '
    137.254.120.50
    

    Then

    [workstation-user@workstation bashscript]$ whois -h whois.radb.net 137.254.120.50 | grep 'origin:' | tr -d 'origin:         '
    AS792
    AS19905
    AS794
    

    Then running the following command for all three ASN:

    
    [workstation-user@workstation bashscript]$ whois -h whois.radb.net -- '-i origin AS792' | grep ^route | grep -v route6 | cut -d" " -f7
    209.17.0.0/20
    137.254.5.0/24
    156.151.0.0/17
    156.151.0.0/16
    129.155.0.0/18
    137.254.128.0/17
    141.146.128.0/17
    137.254.0.0/17
    141.146.0.0/17
    198.17.210.0/24
    
    [workstation-user@workstation bashscript]$ whois -h whois.radb.net -- '-i origin AS794' | grep ^route | grep -v route6 | cut -d" " -f7
    141.146.128.0/17
    137.254.128.0/17
    
    [workstation-user@workstation bashscript]$ whois -h whois.radb.net -- '-i origin AS19905' | grep ^route | grep -v route6 | cut -d" " -f7
    156.154.161.0/24
    156.154.162.0/24
    156.154.163.0/24
    156.154.164.0/24
    156.154.165.0/24
    [ .... and many other 100's ...]
    
    

    The problem is that even if I block all those IP's, I still can access oracle.com because the IP found in step 1 is NO included in the lists of step 3…. Why?????  This happens for a few domains.  For facebook it actually works.  The IP found with nslookup is contained in the lists given by whois but for several other domains, its not working.

    I managed to script evetyhing in bash and use awk/sed and other tools to clean the output and use my main webserver to deliver nicely formatted txt files so pfsense can use them for aloases, only to discover this strange problem..

    If someone is familiar enough with this or has a better way of doing things, I'd love to hear!

    Thanks



  • @lpallard:

    The problem is that even if I block all those IP's, I still can access oracle.com because the IP found in step 1 is NO included in the lists of step 3….

    I managed to script evetyhing in bash and use awk/sed and other tools to clean the output and use my main webserver to deliver nicely formatted txt files so pfsense can use them for aloases, only to discover this strange problem..

    Why not have the script include the IP(s) found in step 1?



  • Why not using solution that has been design for such purpose?

    I mean that you try to block, using FW rules, HTTP(S) access to, e.g. Facebook and this is obviously complex (even if feasible) because IP addresses change.
    On the other hand, if you were using HTTP proxy, this is as easy (and efficient) as configuring rule preventing to access Facebook.

    Real life is slightly more complex because if you want to really block Facebook (or whatever web site) you will have to block also external proxies but Squid and Squidguard are designed to handle such control, relying on blacklists. This is not 100% but quite efficient.

    BTW, blacklist concept could be used to maintain list of IP like done with pfBlockerNG but HTTP proxy will provide more flexibily.



  • pfblocker saved the day!  It is coming very soon with very interesting new features solving the problem altogether!

    Regarding using squid to block domains, it has worked for me in the past (unreliably) but now with pfsense 2.2+ its no longer doable.

    Squid 2 is very old and no longer maintained actively (my understanding) so its not recommended to use it, plus if you happen to run any kind of linux machines behind the proxy server, you will sooner than later hit a brick wall with package managers.  I've learn the hard way when my clients failed to update for months before I realized that squid was the culprit.  Posted on squid's bugzilla and I was told straight away that still using squid 2 in 2014-2015 was not recommended as Squid3 was released more than a year before I reported the issue with 2.0…

    Plus the issue with package managers was very well known from the folks on the squid's bugzilla so no surprise for them.

    Squid 3 may or may not work well but there are very well documented instances where the package in pfsense's repo cannot be installed and wont run!  There are several dozens of errors and misconfigurations happening when installing that package.  I reported the issue(s) on pfsense's bugzilla and on this forum, several commits happened in the following months and the squid's thread documents this problem as well but until last week, this is still not working for me.

    Unfortunately lets forget squid on pfsense for now as there is nothing working.  Perhaps a standalone squid server just behind pfsense would help but I dont need it right now.

    IMO pfblocker is heading the right way and so far version 2 (upcoming) seems to fulfill completely my needs in blocking domains.  Good work BBcan177!!!

    Why not have the script include the IP(s) found in step 1?

    If I understand the way this works, using the IP found in step 1 will only block the IP my DNS sends me to …  This wouldnt block the entire domain's IP range.



  • @lpallard:

    Unfortunately lets forget squid on pfsense for now as there is nothing working.  Perhaps a standalone squid server just behind pfsense would help but I dont need it right now.

    My point with Squid was not "Squid on pfSense" BTW. ;)

    Running Squid on pfSense may fit for quite small infrastructure but will definitely not in case you target something stable and scalable. Furthermore I do share your analysis.

    IMO pfblocker is heading the right way and so far version 2 (upcoming) seems to fulfill completely my needs in blocking domains.  Good work BBcan177!!!

    pfBlocker provides some quite interesting features.  I tested beta version some month ago and I've been impressed by what it does and configuration, management easiness.
    However, blocking IP address rather than URL or even web page content is a totally different approach.

    It may fit for you but to me, this is not one either the other: these are different solution to different problems and you may have to use both.

    Thinking that one will replace the other is, in my opinion, valid in some cases only  ;)


Log in to reply