Netgate Discussion Forum
    • Categories
    • Recent
    • Tags
    • Popular
    • Users
    • Search
    • Register
    • Login

    How To: Manually create SquidGuard blacklist from EasyList, etc.

    Cache/Proxy
    1
    1
    2.7k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • M
      mirkwoody
      last edited by

      Here is how you can create custom blacklists from popular list already on the internet. I've asked a lot of questions here, now I want to try and add something back.

      It is by no means a definitive guide on "How to", but gives some idea. Originally I wanted something automated but to me it seems none have successfully gotten that to work 100%.
      .. There will be a lot of Searching and replacing in NotePad++.

      If anyone has some ninja skills and ideas of how to make this easier, then please tell.

      Step 1. - Find a list you want.
      You could for example find some here https://filterlists.com/. Click "View" and copy all or parts of it.
      Sometimes lists like "EasyList" can be cumbersome and you only want parts of the list, then you can head to  https://github.com/easylist/easylist and find only the parts you want.

      Step 2. - Download "Notepad++"
      Link here: https://notepad-plus-plus.org/
      It will be much easier and faster to work with than for example Word.

      Step 3. - Copy & Edit.
      Copy the stuff you want into NotePad++.
      For SquidGuard to accept you list you need to have a simple list with domain followed by a space "Google.com " for example, not "?Google.com=what." or something odd.

      Some lists are simple and easy, If it is then you can maybe skip to end of this Step.
      Other list are not, let's take example with some mess it won't accept.

      ||zugo.com^$third-party
      ||z???waar.org^$third-party
      ||zxxds.net^$third-party
      foengeri.
      f34f3.
      f43..sfs.
      ||zyiis.net^$third-party
      ||zypenetwork.com^$third-party
      ! adsterra
      

      Open the Replace tool by pressing [Ctrl + H]
      Find: "||"
      Replace with: … nothing.
      Replace all.

      Do this with everything you can, also "^$third-party".
      Also remove the double dot
      Find: ".."
      Replace with: "."

      You should get something like this..

      zugo.com
      z????waar.org
      zxxds.net
      foengeri.
      f34f3.
      f43.sfs.
      zyiis.net
      zypenetwork.com
      ! adsterra
      

      To remove the "! adsterra" line use…
      Find: "!.*$"
      Replace: nothing.
      Search mode: Regular expression.

      So search for everything invalid and delete accordingly. Same with anything like "_ ? = ( ) & # , * ..." any special character and symbol.

      Now you have this.

      zugo.com
      zwaar.org
      zxxds.net
      foengeri.
      f34f3.
      f43.sfs.
      zyiis.net
      zypenetwork.com
      
      

      Now there are top domains missing "foengeri." for example.  (This can be a pest to find in large lists)

      Select "Regular expression" at the bottum on replace tool window.
      Find "$"
      Replace: " "  .. with a space.. you will now add a space behind everything
      Click replace all.

      Now switch it back to "Normal" search mode.
      Find: ". "  dot followed by space.
      Replace with ".com" or other… maybe just delete since you wont know which top domain was suppose to be there.

      ..

      Now Find " " all spaces
      Replace with nothing to remove all the spaces you made.

      ...

      Now add a space behind everything again. Switch to "Regular expression" again,
      Find: "$" 
      Replace with: " " a space (So that SquidGuard can understand)

      zugo.com 
      zwaar.org 
      zxxds.net 
      zyiis.net 
      zypenetwork.com 
      

      Now there should be a space behind every domain.

      Step 4. - Insert into SquidGuard and test

      Services - SquidGuard Proxy Filter - Target catagories - "+ Add"
      Type name and copy text into "Domain list"
      Hit Save… (If all is well it lets you save.. otherwise it tells you what is wrong. If only a few errors, then just do in browser search, usually Ctrl + F)
      Go to "Common ACL", hit the "+" at target rule list, and choose to "deny" or other.
      ... Apply and save in General settings tab.

      Important tip/note

      Don't spend hours looking thru a list before trying to save, SquidGuard will tell you exactly whats wrong in the list, and you can easily search for the incorrect parts in NotePad++.

      ….

      A lot of this could probably be automated more, and there are many search and replace options I am not familiar with.
      Also.. when it is not auto updating it's maybe not the best, but worth some.

      ===  Useful regular expressions in Notepad++ ===

      Wiki how to: http://docs.notepad-plus-plus.org/index.php/Regular_Expressions

      $                                                                            End of line.
      ^                                                                          Front of line.
      /.$                                                                      All after test.com/
      \b(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(.(?1)){3}\b      Remove IPv4 IP adresses.
      [.

      to something else if you like… < > ( )...

      ^.{0,3}$                                                                Any line shorter than 3 ...
      ^(.?)$\s+?^(?=.^\1$)                                        Duplicate finding and replacing.

      Remove empty lines:  Edit > Line Operations > Remove empty lines.

      1 Reply Last reply Reply Quote 0
      • First post
        Last post
      Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.