Improve Custom refresh pattern
-
@jonathanlee assuming your using pfsense AND squid and not just using squid by itself on something like ubuntu, it might be possible to use the ACL tab and block or restrict access to specific domains used for the kali docker image and or find the urls that the kali docker image seeks for updates and block it that way, though at this point I feel this is less a job for squid refresh patterns and more a job for an actual firewall given your after blocking and not so much caching given you want to disallow access to kali. Unless I am mistaken, I feel this would be better served by firewall side and access control lists. In any case, it feels like you want to use access control lists, if you only want to allow certain authorized hosts to use kali, then you want access control lists to lock down access to specific ip addresses, ranges, or hosts.
-
@high_voltage spot on. I added custom URL blocking for Docker, ruby gems, and hotjar, as well as expressions and blocking of url shorteners. Thanks for your reply. I also keep having Windows 10 trying to update all the sudden over http, and the firewall is blocking that as a policy violation.
It's weird this update was working before. What's confusing is I can click on that link and it will download directly also.
I was able to direct the Raspberry pi to update only over https using a different mirror. But Windows it just keeps trying http.
-
@jonathanlee have you made sure that you cleared out the squid disk cache (again, assuming you're using squid over pfsense, which I would in hindsight hope you are doing now that I think about it given your posting here on the pfsense support forum, DUH) after making any and all changes? though....hindsight, you might not need to do that since your not messing with refresh patterns this time but instead messing with ACL's.... uh, not sure if this would be useful in this case here or not, but possibly, if you're saying things are being strange (not entirely sure, my mind is on 20 different projects right now as I make this latest reply right here) but maybe if things ARE being weird, try resetting the states table in pfsense? I have had the oddball situation that things after paying a ton of attention (more than I should have to be honest) I finally said "F it" and reset the firewall states table to try and test if that cleared out oddball situations, and it has, so maybe try that?
apologies for the hastily typed up reply, but as I said, my mind is on 20 different projects at the moment that I type this up.
-
@high_voltage thanks for the reply,
Yes this Netgate 2100 max firewall is running Squid over pfSense with custom refresh patterns and Squid guard. I also reset the disk cache and have this installed WPAD for on and off the wifi with smartphone I even added SSL certificates to everything. I forced all traffic into the proxy port 3128-3129 everything works Hulu, Disney plus, Amazon video. I watch XFILES alot lately over Hulu with this config. What's weird I can take that link and manually download the update over the browser, but somehow the system won't download the update. It's almost paused just gets to 0. My Raspberry Pi was doing this also, I changed the mirrors to only use https downloads under some settings and that now updates fine. Again Windows 10 pro is not using https for the anti virus signature updates right now. I am still new to this refresh options. When you force traffic over the proxy port 80 is still working for everything else I can access http over web url use. It can see Squid guard blocks and will display viruses on clam AV tests. This thing is a tank. This last configuration fix will make it work perfectly. Why does Microsoft use http for updates? Most of the internet moved to HTTPS. It's weird right? I am about to factory default it and try again. Nat I tested with a port redirect also. I tested using just transparent. Everything works except the Windows 10 pro updates. It worked last night however Microsoft started using https for a couple hours.
-
@jonathanlee ill be totally honest, part of why i recently disabled squid was for similar issues, for whay ever reason, linux updates broke recently when using squid, not sure why but this definitely seems to be an issue for some reason right now.
-
@high_voltage I got my Raspberry Pi to work with a different mirror I edited sources to one that allowed https. When I run apt-get update it uses a different mirror now I use the constant com's mirror.
Edit this file
/etc/apt/sources.listAdd a https source from the update mirrors for example in Raspberry Pi Linux I changed it to a https source.
Check out other countries some are almost all https like Germany.
Make sure to upvote
-
@jonathanlee it has got to work the same for refreshers for other Linux flavors also.
-
@jonathanlee that was the only way to get Linux updates to work with Squid for me, it was doing the same thing as Windows updates, Squid would show a http and when you looked at Squid guard's live connections it would only show 0.
-
Made new post with this specific issue.
https://forum.netgate.com/topic/169166/warning-possible-bypass-attempt-found-multiple-slashes-where-only-one-is-expected-http-dl-delivery-mp-microsoft-com-filestreamingservice-files/3?_=1642466910316
-
Here it is, per your request, a Windows 10 update cached and delivered to another machine. Notice the HIT
(IMAGE: Windows dynamic refresh patterns to work recently)
Make sure to upvote
-
@jonathanlee For whatever reason, it's worth noting I literally only just discovered 2 weeks ago that apparently a good chunk of my problems were due to transparent squid and clam AV, having clamAV set to scan all mode was causing random issues I cannot even begin to pinpoint. Setting it to scan Web only fixed everything, but having it set to scan all mode for whatever reason would cause apt packages To fail at trying to receive header information. Even http connections failed due to this.
-
@high_voltage I think it is the same as if you were to do a ClamAV scan on Kali Linux. So many packages and tools come up as issues when they are in fact only Pen Testing tools. In PFsense Curl, and many other items are included in packages and may scan as false positives also as they are not on a client machine however part of a firewall. It should have a scan Squid Cache option, that is what should be scanned right? Think about the number of items stored in the content accelerator that could be invasive. Why does squid not include scan local cache as an option?
-
@jonathanlee no, i mean it broke traffic entirely.
-
@high_voltage wow that's different. I had issues where I needed to clear the cache before the traffic would flow again, almost like a container was in the cache.
-
@jonathanlee Huh? last time I posted in this thread was 4 years ago.
-
@kom sorry I thought you wanted to see a Windows 10 update run that was cached.
-
@jonathanlee Perhaps four years ago I did. I don't remember since it's been four years. I don't even use squid anymore. It's completely useless other than as a base for squidguard URL filtering.
-
@kom I respectfully disagree with "useless", I use it for HTTPS cache anti-virus scanning of HTTPS websites and HTTP. Dynamic caching, URL filtering, and blocking. Don't get me wrong it is rather complicated to understand, however the vast abilities that it has to customize a network environment by need is what sets it apart. It can do many things. It is just a challenge to learn. It has also protected my system from many hidden issues that Clam AV stops and reports with HTTPS alongside pup detection as well as generates clear reports. It's Mirrored Analytics down to a granular level.
-
@kivimart is it working for squid version - 4.45
-
@dmalick You can use the latest here https://github.com/mmd123/squid-cache-dynamic_refresh-list
Yes it works with the latest squid
