@johnpoz side note I finally found my invasive container it's on my 2019 Motorola g-power the thing is registering all sorts of Snort open AppID items I am not using, everything else on my network is matched to app use. It even saw Opera browser I don't even use that, alongside Snapchat, LinkedIn on and on even a bunch of Stripe payment service, and endless Skype. It was the smartphone.

Looks like Squid's website just released version 6.5 on Nov 4th 2023

That was 10 days ago. . .

Screenshot 2023-11-14 at 4.29.12 PM.png

I am confused as it was said it was not updated in 2 years. . .
Screenshot 2023-11-14 at 4.34.47 PM.png
Was updated again Nov 6 2023

Also many security issues have been resolved per the GitHub.

Screenshot 2023-11-14 at 4.31.04 PM.png

I am thinking install it on a raspberry pi 5 8gb and NAT to it from the firewall

This seems to be working now/resolved.

I bounced the whole firewall and my (pihole) dns servers and came back to it 30 minutes later and now it is working.

I don't understand what bouncing the pihole servers, or the full firewall ( given I previously bounced the dns resolver / haproxy services) might have done but with the haproxy backend happy, everything is now working.

hopefully this helps the next guy :)

Still noone?

Nice!

I forgot to link to the issue ticket: https://redmine.pfsense.org/issues/14764

What is the next official Netgate product that will continue to support a proxy with SSL intercept that can be purchased? Now that this is being twightlighted?

What version should I upgrade too for proxy cacheing abilities? I have a SG-2100 currently. Should users move to Palo Alto?

@Kababayan

I did not know about the built in store id helper program.

I was testing this one out from git hub

https://github.com/rudiservo/pfsense_storeid/tree/master

Quick Question . . .

What are you doing with the acl that points to redbot.org?

acl dontrewrite url_regex redbot.org

Why create a one time ACL just to block it?

per Squid's website

refresh_pattern ^http://(youtube|ytimg|vimeo|[a-zA-Z0-9\-]+)\.squid\.internal/.* 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private acl rewritedoms dstdomain .dailymotion.com .video-http.media-imdb.com .c.youtube.com av.vimeo.com .dl.sourceforge.net .ytimg.com .vid.ec.dmcdn.net .videoslasher.com store_id_program /usr/local/squid/bin/new_format.rb store_id_children 40 startup=10 idle=5 concurrency=0 store_id_access allow rewritedoms !banned_methods store_id_access deny all

They just use the refresh and one acl with all the domains in it that are needed.

If you want to up the cache time you may need to use Squids StoreID helper program,

https://forum.netgate.com/topic/182487/storeid-and-squid-helper-program/

I have been researching this. It does increase cache rates. However I use SSL intercept.

You may also want to enable Dynamic cache for Windows updates.

#WINDOWS refresh_pattern -i windowsupdate.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i windows.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com.akadns.net/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i deploy.akamaitechnologies.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i msedge.b.tlu.dl.delivery.mp.microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims #FACEBOOK refresh_pattern ^http.*facebook.com.* 720 100% 4320 #FACEBOOK IMAGES refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920 refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920 refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif|jpg?) 241920 99% 241920 store-stale refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png|jpg?) 241920 99% 241920 refresh_pattern ^http.*profile.ak.fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920 refresh_pattern ^http.*fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920 #FACEBOOK VIDEO refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 refresh_pattern (audio|video)/(webm|mp4) 129600 99% 129600 store-stale #refresh_pattern -i ^http.*squid.internal.* 241920 100% 241920 store-stale #YAHOO refresh_pattern ^http.*mail.yahoo.com.* 720 100% 4320 refresh_pattern ^http.*yahoo.* 720 100% 4320 refresh_pattern ^http.*yimg.* 720 100% 4320 #apple update refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 refresh-ims #refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 refresh_pattern -i phobos\.apple\.com 129600 100% 129600 refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 #GOOGLE STUFF refresh_pattern ^http.*gmail.* 720 100% 4320 refresh_pattern ^http.*google.* 720 100% 4320 #AntiVirus refresh_pattern ^http.*kaspersky.* 43200 100% 43200 refresh_pattern ^http.*geo\.kaspersky\.com.* 43200 100% 43200 refresh_pattern kaspersky.*\.avc$ 43200 100% 43200 refresh_pattern kaspersky 43200 100% 43200 #Office/Windows refresh_pattern ^http.*officecdn\.microsoft\.com.* 43200 100% 43200 refresh_pattern ^http.*officecdn\.microsoft\.com\.edgesuite\.net.* 43200 100% 43200 refresh_pattern ^http.*delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200 refresh_pattern ^http.*download\.windowsupdate\.com.* 43200 100% 43200 refresh_pattern ^http.*download\.windowsupdate\.com\/.* 43200 100% 43200 refresh_pattern ^http.*au\.download\.windowsupdate\.com\/.* 43200 100% 43200 refresh_pattern ^http.*tlu\.dl\.delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200 #Website refresh_pattern -i (\.|-)(xml|js|jsp|txt|css|swf|html|htm|txt|gif|tiff)(\?.*)?$ 360 80% 1440 #end new refresh patterns refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 #MISC FILE CACHING HERE refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)(\?|$) 43800 100% 129600 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$) 43800 100% 129600 refresh_pattern -i \.(m1v|M2V|M2P|MOD|MOV|FLV)(\?|$) 43800 100% 129600 refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf)(\?|$) 43800 100% 129600 refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))(\?|$) 43800 100% 129600 refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)(\?|$) 43800 100% 129600 refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))(\?|$) 43800 100% 129600 refresh_pattern -i \.(woff|txt|exe|dmg|webm)(\?|$) 43800 100% 129600 refresh_pattern -i \.(doc|pdf)(\?|$) 10080 90% 43200 # DOC | PDF refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|docx|tiff)$ 10080 90% 43200 refresh_pattern -i .index.(html|htm)$ 0 40% 10080 refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(wav|class|dat|zsci|ver|advcs)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 refresh-ims refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 refresh-ims refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2) 43200 100% 432000 refresh_pattern -i (.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd) 43200 100% 432000 refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?) 43200 100% 432000 refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt) 43200 100% 432000 #new refresh patterns 2 refresh_pattern -i (\.|-)(ini|def|sig|upt|mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|esd|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 #end new refresh patterns 2 #new refresh patterns refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|flv) 129600 100% 129600 refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880 # GENERIC CACHING BELOW refresh_pattern -i \.(cdn) 10800 100% 43800 refresh_pattern -i (cdn) 10800 100% 43800 refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 refresh_pattern ^.*safebrowsing.*google 129600 100% 129600 refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 129600 100% 129600 refresh_pattern ytimg\.com.*\.jpg 129600 100% 129600 #YOUTUBE refresh_pattern \.ytimg\? 10800 90% 10800 refresh_pattern -i (yimg|twimg).com.* 1440 100% 129600 refresh_pattern -i (ytimg|ggpht).com.* 1440 80% 129600 refresh_pattern -i (get_video?|videoplayback?|videodownload?|.mp4|.webm|.flv|((audio|video)/(webm|mp4))) 241920 100% 241920 store-stale refresh_pattern -i ^https?://..googlevideo.com/videoplayback. 10080 99% 43200 store-stale refresh_pattern -i ^https?://..googlevideo.com/videoplayback.$ 241920 100% 241920 store-stale #XBOX refresh_pattern -i lancache-microsoft 525600 100% 525600 refresh_pattern -i images-eds.xboxlive.com 525600 100% 525600 #catch all refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 range_offset_limit 0

Ref:
https://github.com/mmd123/squid-cache-dynamic_refresh-list/blob/master/pfsense%20squid%20refresh%20patterns

https://wiki.squid-cache.org/Features/StoreID#:~:text=“Store%20ID”%20is%20another%20name,algorithm%20via%20a%20helper%20program.

https://github.com/rudiservo/pfsense_storeid

http://www.squid-cache.org/Doc/config/store_id_program/

https://wiki.squid-cache.org/Features/StoreID/DB

https://forum.netgate.com/topic/106614/dynamic-cache-not-work/4

I hope that gives you some good info on ways to get better hit rates.

https://forum.it-monkey.net/index.php?topic=23.0

Check out this how to guide also.

If you are not using IPv6 try to disable AAAA access.

Services/DNS Resolver/General Settings

Under custom

server: do-ip4: yes prefer-ip4: yes do-ip6: no prefer-ip6: no private-address: ::/0 dns64-ignore-aaaa: *.* do-not-query-address: :: do-not-query-address: ::1 do-not-query-address: ::/0

Screenshot 2023-11-01 at 10.04.45 AM.png

6cf0e84c-91b7-4081-b7f1-b828b07fd80c-image.png
This isn't functioning correctly. Am I heading in the right direction?

It seems, per HAProxy documention, that the user needs to exist and it must be able to login without password.
This is against all our policies so I changed the mysql-check back to a basic ping check.

This fixes the Health check, but not the biggest problem: No connection is possible trough HAProxy.