Squidguard category filtering silently fails with large blacklist - a workaround
-
@dbmandrake the more I read about this issue the more I wonder if containers would be a good solution for this. Maybe a container that could spin up just for the download needs and install the blacklist and delete itself just to replace the updated URLs in a blacklist. Something like Docker, Kerbernets, Mesosphere, or FreeBSD Jails. Maybe not to the level of container orchestration, but on a smaller one container use need. It would be stored on the drive, and be isolated and protected. Some may say RAM disk was the birth of virtualization. I always thought so during DOS 3.11 days. So why not just move into container utilization for blacklists. What are your thoughts on something like containerized blacklists to help with how large the url blacklists are getting?
What about something like FreeBSD Jails? I have been researching them. They can even share IP addresses with the primary host. Also risky as they can data marshal the network cards.
(Image: FreeBSD Jails Testing with iocage management)Hypothetical solution:
If you needed to have access to the container for a URL update just start the Jail download, it downloads while the other Jail is in use, after the update is pushed swap to the other Jail as the primary and leave the other as standby for next time. Why even wait for the new download right? Just like AMO the OS that runs the Siemens Hicom PBXs, they use of dual boards, they have a primary and a secondary. To update software changes in a Hicom PBX you run the command "EXE-updat: BP, ALL;" two times so it sends the updates and changes to each board, when the command is run it changes the active board from 1 to 2 and after back to 1. same with EXE-updat : A1, ALL; they have a active and a stand by. We have the software to do this with blacklists now and containers, lets update the firewalls like this. Right?
This is a proven solution within currently in use as a hardware design for large scale PBXs. We just need to retool this within the context of containers configurations for use with PfSense firewalls software improvements.
(Image: Logged into FreeBSD Jail)Hypothetical:
If the Jail is set up just to hold a blacklist database, it could self-check changes within that database and so on within the two containers, once an update starts for the standby jail, it just swaps in as the primary and you never see any downtime, the primary becomes the secondary and waits until the next scheduled update.Maybe the solution is that we need a different kind of download, "a Jailed containerized download," One container holds the primary useable list, and once it is outdated it just swaps to the other container.
-
@jonathanlee Holy Cow there are plenty of porn websites.
Here's what I did as another workaround for just too many blocks in memory that slip through and fail. Maybe it's of interest to someone here.
By default, PFSense Blocks everything right?
This is My specific Case WAN RULES:
1) Allow Both USA IPS - This would work for most unless you command total global e-commerce!
Maybe you would allow a few others in. I already worked hard and added every country in PFSense ALIASES for you guys to take whatever you need. PFSENSE-GEOIP-List-of-Countries
2) Allow only these Ports - Created Aliases Called Allowed Ports***3) Create a Drop Rule last rule (yes it drops anyway) and Tagged in Log RealTime "Dropped Wrong Ports" Or "Dropped Wrong Country!"
4) Finally, I only drop what's needed because of the small open footprint. Don't block the world!
Then, I created white lists for updates for Ubuntu, PFSense, Debian, Proxmox, etc at the Top of the Drop rule.
5) Lastly, ON the LAN side or WAN side use pfBlocker to protect your open inbound ports or just block outbound Lan. It's a choice but remember those rouge IPs some are USA based (We passed "USA IPs and Selected Ports")but most are not. Like I said it's a choice but now you should be able to use all these ips without a rouge Ip slipping through. The SNORT on the LAN side only.
6) Now make absolutely sure your IPS lists are working @ Diagnostics / Tables.
7) System Advanced Firewall adjust "Firewall Maximum Table Entries!" The maximum number of table entries for systems such as aliases, sshguard, snort, etc, combined. Note: Leave this blank for the default. On this system, the default size is: 50,000,000.
I Use 50 Million (WTF) with 32 gigs of Ram but I'm sure I'm not even close. I do know you will have to tune this if you have less RAM for sure. At some point, if you keep adding growings lists then you will have "got aways," block what you really need.Everything should work w/o adding multiple massive lists.
By locking down the IPS to Alllow, and by only allowing only those IPs limited access to the selected few ports you can reduce
your footprint significantly and not bog down your firewall. I hope this helps someone.
-
@mikeinnyc yes there is so many websites that need to be blocked it does a memory overflow when loading the blacklights, unreal right? I do a reduced list and add in specific needs.
-
@mikeinnyc Hi Mike great post, how much resources are being consumed with this configuration?
-
Here is a better example of Jails in use. Each is running within the host machine. Maybe a solution to attack this problem this way?
They can even change the IP addresses of the Jails while they are live, so you could set it to one or the other with Lan based IP addresses also.
-
When I worked on wall street for a really fast three decades before the days of IT Lockdowns and dialup 56k, T1s, and the most expensive DSL connections $800 a month. We "May Have" used gambling sites and others. (See the movie the wolf of Wall Street!") I do not recall ever seeing porn on computer screens!" Well after a couple of industry lawsuits here's how it rolled in every co afterward:
- DENY EVERYTHING - Guess what we have this using pfSense by default!
- Beg Compliance for Opening website access - Always denied. Thank gawd for cell phones :)
- Read approved websites list of maybe 1 page in total - all business related.
WHY- Because LOG FILES must be kept for life on Wall Street. So it's possible that "Girls Gone Really Wild," may come back to haunt me! :) Probably would help me hahaha.
Now, my recommendations - I have vast experience in getting sued as a CEO.
IF - You have employees or people you can physically tap on the shoulder (1099) then use the above. Only have approved outgoing websites. If they complain, they won't because personal cell phone 5G is outside the scope and jurisdiction of record keeping
unless it's business related. Besides they screen record everything and video record. Using encrypted sites like signal and datchat for personal use is unstoppable.If Web Hosting - Deny all countries by default - except this Whitelist block of IPS from USA. You probably should use cloud flare if more than a few countries. So many bad actors from certain countries.
Deny all ports - Except this whitelist of ports. You can further add ips to lock in down more.
Then, protect those web hosting ports with rate Liming stick tables (HAProxy) and other filters.
The point is that with web hosting you should rarely need outbound denys lists why because by default deny all except this IP and that Ip and this port. Your production Network traffic should not leak into Private lans period.
Buy enterprise hardware, not VMs. You will always have problems so check logs. -
Guys, all this talk about firewall rule policies, HA Proxy, corporate policies etc is nothing to do with what this topic is about and is only serving to derail and dilute the thread.
Please try to stick discussion about the bug in the SquidGuard package which allows a too small ramdisk to overflow during extraction without any warnings or errors and then imports a corrupted database into squidGuard, this is what this thread is about. Thanks.
-
It is about Business Impact Analysis and Mission Essential Functions.
The rabbit hole:
Yes, configure your system on a as need basis. Take into consideration many factors some being Data Classifications, Geographical Considerations, Data Sovereignty, Organizational Consequences. The Serverless Architecture of our modern systems today also start to play key roles, when using something like Azure as a Domain. Moreover, Software Defined Networking is playing more of a role than most end users know about today with the use of hyper-convergence. "User-facing" problems are far different than confidentiality and data integrity problems and they require different considerations within risk mitigation. It is not only that machine virtualization that needs consideration, but container virtualization, and full application virtualization. They take different roles within risk mitigation as they can perform data marshalling over the NIC cards easily. Look to how many cloud service models they have today, Infrastructure as a service, Software as a service, Platform as a service, Anything as a service, and even Security as a service. The toxic idea of just avoiding virtualization, and or ignoring it, no longer applies. The risk mitigation plans have to include virtualization today. All needs must be taken into consideration within implementing authorization solutions. It's Discretionary and Role-Based Access Control. Windows 11 helped to solve some of the issues with virtualization risks on an end user platform, as some issues were occurring inside of Windows 10.Access Control Lists:
I have also noticed that some websites if you simply just block the IP address inside of the access control lists, they can still be accessed over HTTPS or HTTP, that is why I am using Squidguard it checks the http/https get requests, but it needs a blacklist to function, manual or downloadable.One of the reasons why I am studying software and computer science, is to help find a really good solution, and it seems as soon as we get a good one working some prototype protocol evolves that needs to be accounted for that is not following the rules or compliance within Internet Assigned Numbers Authority. Why have rules like the ones from IANA if there is no compliance. Now in comes the need for something like internet backbone compliance servers or cards installed right on a Ciena system, that can track and block prototype protocol abuses. Lets agree on one thing the wild west days before GDPR and CCPA are gone forever.
-
This post is deleted! -
@dbmandrake Please take time to check my hypothetical solution with Squidgard lists above, let me know what you think.
-
@jonathanlee I know this sounds off-topic but it's dead on.
Don't block the world and call it a bug :)
This is a hardware limitation issue. Try adding more memory first and then add many IP aliases.The real question is network design.
By Default pfSense Blocks everything. How can we add more whitelists?
Do you really want employees accessing everything or is this "Home use only?"
Sorry to be abrupt but this solves your problem. Better hardware and more ram. -
@jonathanlee Re: Containers / Jails.
This seems like a massive degree of overkill, what problem do they solve exactly ?
The reason why the ramdisk exists at all is for small devices with limited storage where the temporary disk space needed to extract the plain ascii version of the blocklists (around 300MB for the blocklist I'm using) would cause the device to run out of disk space.
I'm just not seeing how a container solves the problem of lack of disk space.
It also speeds up the extraction and importing process since what are essentially temp files don't have to be written out to disk.
However on a server with a decent sized SSD there isn't really any advantage to using the ramdisk apart from a slight speed increase, but the disadvantage is it can fail with larger blocklists and due to inadequate error checking the failure is not detected and the incomplete blocklist is imported into squidguard without complaint which then silently breaks your filter categories. This is a big problem in an environment like a school where a school has a duty of care to not allow pupils to access certain kinds of websites.
If I do write a patch to add a ramdisk enable/disable preference option I will also write a patch to fix the error checking so that a failure due to exceeding the ramdisk size (when enabled) is reported to the user and the incomplete blocklists do not overwrite the currently active ones.
I would like to do this it's just a matter of finding the time to work on it as I'm bogged down with too many things at the moment.
-
@mikeinnyc Sorry but you comments are way off the mark - "Better hardware and more ram" doesn't solve anything. You clearly haven't read and/or understood the original post and grasped the issue with the fixed size ramdisk which is currently a part of the blocklist import process.
The hardware is absolutely capable of working with a blocklist of the size in question - without even breaking a sweat. Once the limitation of the small, fixed size ramdisk is removed, that is.
-
@dbmandrake im thinking that post was some type of SPAM. I could be wrong
-
@dbmandrake I actually forgot about the speeds of SSD drives today, the hypothetical solution I hoped would also help solve the issue with downtime when updating blacklists, on my firewall everything goes offline during blacklist updates, and the firewall can't use the full blacklist because of the same issue you described and solved. My system is the MAX so it has an extra 30GBs SSD on it. Additionally, it could protect the blacklist uptime if something got corrupted with a bad blacklist update, this way it could default back to that other container if that issue should ever occur. Kind of like a HA-Proxy just for blacklists, primary and secondary. High availability.
Thanks for looking at that post, I just wanted to have some input on it with Squidguard, alongside more visibility on FreeBSD Jails for the possibly retooling them for something else.
-
@jonathanlee said in Squidguard category filtering silently fails with large blacklist - a workaround:
@dbmandrake I actually forgot about the speeds of SSD drives today, the hypothetical solution I hoped would also help solve the issue with downtime when updating blacklists, on my firewall everything goes offline during blacklist updates, and cant use the full blacklist because of the issue you described.
I've been using the full size blacklist since before I started this thread without issue - with the patch to disable the ramdisk. No issues have cropped up yet, in fact the firewall hasn't been rebooted since before this thread was started. I actually have a second firewall running this patch as well as I've had to temporarily set up a second proxy server for a slightly different use case.
Regarding going offline during the update, I'd have to check but as far as I know Squid doesn't go offline during the extraction of the tar file - which is the longest part of the process.
I think it's only offline for a few seconds at the end of the import process for the same amount of time as if you'd pressed the Apply button in the squidguard config page, which forces squidguard to re-read the on disk version of the blacklist binary database into memory.
But I should run a test to time how long the proxy is out of action. I have mine scheduled to do the blacklist update automatically at 2am anyway so if the proxy is down for a few seconds at 2am nobody cares.
Not sure what you mean when you say "everything" goes offline on your firewall when the blacklist updates - only the proxy (and transparent proxy) will be affected, all other traffic is unaffected.
-
@dbmandrake everything on my network is pointed at the proxy, plus I run a WPAD, what I mean is when Squidguard updates that blacklist the proxy starts to update and users have no internet access until it restores, I am running a Netgate SG2100-MAX it only has 4GBs ram with it. It takes a bit longer for me around 5 mins, it takes long enough that it will stop a streaming movie. I need to set it to update during the AM too, again I am running a DSL 6meg.
-
@jonathanlee For automatic scheduled update see my post in another thread.
-
@dbmandrake thanks for the information on the auto update.
