Nice!

I forgot to link to the issue ticket: https://redmine.pfsense.org/issues/14764

What is the next official Netgate product that will continue to support a proxy with SSL intercept that can be purchased? Now that this is being twightlighted?

What version should I upgrade too for proxy cacheing abilities? I have a SG-2100 currently. Should users move to Palo Alto?

@Kababayan

I did not know about the built in store id helper program.

I was testing this one out from git hub

https://github.com/rudiservo/pfsense_storeid/tree/master

Quick Question . . .

What are you doing with the acl that points to redbot.org?

acl dontrewrite url_regex redbot.org

Why create a one time ACL just to block it?

per Squid's website

refresh_pattern ^http://(youtube|ytimg|vimeo|[a-zA-Z0-9\-]+)\.squid\.internal/.* 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-must-revalidate ignore-private acl rewritedoms dstdomain .dailymotion.com .video-http.media-imdb.com .c.youtube.com av.vimeo.com .dl.sourceforge.net .ytimg.com .vid.ec.dmcdn.net .videoslasher.com store_id_program /usr/local/squid/bin/new_format.rb store_id_children 40 startup=10 idle=5 concurrency=0 store_id_access allow rewritedoms !banned_methods store_id_access deny all

They just use the refresh and one acl with all the domains in it that are needed.

If you want to up the cache time you may need to use Squids StoreID helper program,

https://forum.netgate.com/topic/182487/storeid-and-squid-helper-program/

I have been researching this. It does increase cache rates. However I use SSL intercept.

You may also want to enable Dynamic cache for Windows updates.

#WINDOWS refresh_pattern -i windowsupdate.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i windows.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i microsoft.com.akadns.net/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i deploy.akamaitechnologies.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims refresh_pattern -i msedge.b.tlu.dl.delivery.mp.microsoft.com/.*\.(cab|exe|dll|ms[i|u|f|p]|[ap]sf|wm[v|a]|dat|zip|psf) 43200 80% 129600 reload-into-ims #FACEBOOK refresh_pattern ^http.*facebook.com.* 720 100% 4320 #FACEBOOK IMAGES refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920 refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js|jpg?) 241920 80% 241920 refresh_pattern -i ((facebook.com)|(85.131.151.39)).(jpg|png|gif|jpg?) 241920 99% 241920 store-stale refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png|jpg?) 241920 99% 241920 refresh_pattern ^http.*profile.ak.fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920 refresh_pattern ^http.*fbcdn.net.*(jpg|gif|png|jpg?) 241920 99% 241920 #FACEBOOK VIDEO refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200 refresh_pattern (audio|video)/(webm|mp4) 129600 99% 129600 store-stale #refresh_pattern -i ^http.*squid.internal.* 241920 100% 241920 store-stale #YAHOO refresh_pattern ^http.*mail.yahoo.com.* 720 100% 4320 refresh_pattern ^http.*yahoo.* 720 100% 4320 refresh_pattern ^http.*yimg.* 720 100% 4320 #apple update refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200 refresh-ims #refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200 refresh_pattern -i appldnld\.apple\.com 129600 100% 129600 refresh_pattern -i phobos\.apple\.com 129600 100% 129600 refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600 #GOOGLE STUFF refresh_pattern ^http.*gmail.* 720 100% 4320 refresh_pattern ^http.*google.* 720 100% 4320 #AntiVirus refresh_pattern ^http.*kaspersky.* 43200 100% 43200 refresh_pattern ^http.*geo\.kaspersky\.com.* 43200 100% 43200 refresh_pattern kaspersky.*\.avc$ 43200 100% 43200 refresh_pattern kaspersky 43200 100% 43200 #Office/Windows refresh_pattern ^http.*officecdn\.microsoft\.com.* 43200 100% 43200 refresh_pattern ^http.*officecdn\.microsoft\.com\.edgesuite\.net.* 43200 100% 43200 refresh_pattern ^http.*delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200 refresh_pattern ^http.*download\.windowsupdate\.com.* 43200 100% 43200 refresh_pattern ^http.*download\.windowsupdate\.com\/.* 43200 100% 43200 refresh_pattern ^http.*au\.download\.windowsupdate\.com\/.* 43200 100% 43200 refresh_pattern ^http.*tlu\.dl\.delivery\.mp\.microsoft\.com\/filestreamingservice\/files.* 43200 100% 43200 #Website refresh_pattern -i (\.|-)(xml|js|jsp|txt|css|swf|html|htm|txt|gif|tiff)(\?.*)?$ 360 80% 1440 #end new refresh patterns refresh_pattern \.(ico|video-stats)$ 129600 100% 129600 #MISC FILE CACHING HERE refresh_pattern -i \.(3gp|7z|ace|asx|avi|bin|cab|dat|deb|rpm|divx|dvr-ms)(\?|$) 43800 100% 129600 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso)(\?|$) 43800 100% 129600 refresh_pattern -i \.(m1v|M2V|M2P|MOD|MOV|FLV)(\?|$) 43800 100% 129600 refresh_pattern -i \.(jp(e?g|e|2)|gif|pn[pg]|bm?|tiff?|ico|swf)(\?|$) 43800 100% 129600 refresh_pattern -i \.(mp(e?g|a|e|1|2|3|4)|mk(a|v)|ms(i|u|p))(\?|$) 43800 100% 129600 refresh_pattern -i \.(og(x|v|a|g)|rar|rm|r(a|p)m|snd|vob|wav)(\?|$) 43800 100% 129600 refresh_pattern -i \.(pp(s|t)|wax|wm(a|v)|wmx|wpl|zip|cb(r|z|t))(\?|$) 43800 100% 129600 refresh_pattern -i \.(woff|txt|exe|dmg|webm)(\?|$) 43800 100% 129600 refresh_pattern -i \.(doc|pdf)(\?|$) 10080 90% 43200 # DOC | PDF refresh_pattern -i .(iso|avi|wav|mp3|mp4|mpeg|swf|flv|x-flv)$ 43200 90% 432000 refresh_pattern -i .(deb|rpm|exe|zip|tar|tgz|ram|rar|bin|ppt|doc|docx|tiff)$ 10080 90% 43200 refresh_pattern -i .index.(html|htm)$ 0 40% 10080 refresh_pattern -i .(ppt|pptx|doc|docx|docm|docb|dot|pdf|pub|ps)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(xls|xlsx|xlt|xlm|xlsm|xltm|xlw|csv|txt)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(app|bin|deb|rpm|drpm|exe|zip|zipx|tar|tgz|tbz2|tlz|iso|arj|cfs|dar|jar)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(bz|bz2|ipa|ram|rar|uxx|gz|msi|dll|lz|lzma|7z|s7z|Z|z|zz|sz)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(exe|msi)$ 0 90% 200000 refresh-ims refresh_pattern -i .(cab|psf|vidt|apk|wtex|hz|ova|ovf)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(xml|flow|asp|aspx)$ 0 90% 200000 refresh-ims refresh_pattern -i .(json)$ 0 90% 200000 refresh-ims refresh_pattern -i .(asx|mp2|mp3|mp4|mp5|wmv|flv|mts|f4v|f4|pls|midi|mid)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(mpa|m2a|mpe|avi|mov|mpg|mpeg|mpg3|mpg4|mpg5)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(m1s|mp2v|m2v|m2s|m2ts|mp2t|wmx|rm|rmvb|3pg|3gpp|omg|ogm|asf|war)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(wav|class|dat|zsci|ver|advcs)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(gif|png|ico|jpg|jpeg|jp2|webp)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(jpx|j2k|j2c|fpx|bmp|tif|tiff|bif)$ 100000 90% 20000 refresh-ims refresh_pattern -i .(pcd|pict|rif|exif|hdr|bpg|img|jif|jfif)$ 100000 90% 200000 refresh-ims refresh_pattern -i .(woff|woff2|eps|ttf|otf|svg|svgi|svgz|ps|ps1|acsm|eot)$ 100000 90% 200000 refresh-ims refresh_pattern -i (\.|-)(mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2) 43200 100% 432000 refresh_pattern -i (.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd) 43200 100% 432000 refresh_pattern -i (.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?) 43200 100% 432000 refresh_pattern -i (.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt) 43200 100% 432000 #new refresh patterns 2 refresh_pattern -i (\.|-)(ini|def|sig|upt|mid|midi|mpg|mpeg|ram|cav|acc|alz|apk|at3|bke|arc|ass|ba|big|bik|bkf|bld|c4|cals|clipflair|cpt|daa|dmg|ddz|dpe|egg|egt|ecab|ess|esd|gho|ghs|gz|ipg|jar|lbr|lqr|lha|lz|lzo|lzma|lzx|mbw|mc.meta|mpq|nth|osz|pak|par|par2|paf|pyk|pk3|pk4|rag|sen|sitx|skb|tb|tib|uha|uue|viv|vsa|z|zoo|nrg|adf|adz|dms|dsk|d64|sdi|mds|mdx|cdi|cue|cif|c2d|daa|b6t)(\?.*)?$ 43200 100% 432000 #end new refresh patterns 2 #new refresh patterns refresh_pattern -i (\.|-)(mp3|m4a|aa?c3?|wm?av?|og(x|v|a|g)|ape|mka|au|aiff|zip|flac|m4(b|r)|m1v|m2(v|p)|mo(d|v)|arj|appx|lha|lzh|on2)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(exe|bin|(n|t)ar|acv|(r|j)ar|t?gz|(g|b)z(ip)?2?|7?z(ip)?|wm[v|a]|patch|diff|mar|vpu|inc|r(a|p)m|kom|iso|sys|[ap]sf|ms[i|u|f]|dat|msi|cab|psf|dvr-ms|ace|asx|qt|xt|esd)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(ico(.*)?|pn[pg]|css|(g|t)iff?|jpe?g(2|3|4)?|psd|c(d|b)r|cad|bmp|img)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(webm|(x-)?swf|mp(eg)?(3|4)|mpe?g(av)?|(x-)?f(l|4)v|divx?|rmvb?|mov|trp|ts|avi|m38u|wmv|wmp|m4v|mkv|asf|dv|vob|3gp?2?)(\?.*)?$ 43200 100% 432000 refresh_pattern -i (\.|-)(docx?|xlsx?|pptx?|rtf|xml|pdf|tiff?|txt)(\?.*)?$ 43200 100% 432000 refresh_pattern -i \.(rar|jar|gz|tgz|tar|bz2|iso|m1v|m2(v|p)|mo(d|v)|flv) 129600 100% 129600 refresh_pattern (Release|Packages(.gz)*)$ 0 20% 2880 # GENERIC CACHING BELOW refresh_pattern -i \.(cdn) 10800 100% 43800 refresh_pattern -i (cdn) 10800 100% 43800 refresh_pattern (get_video\?|videoplayback\?|videodownload\?|\.flv?) 129600 100% 129600 refresh_pattern (get_video\?|videoplayback\?id|videoplayback.*id|videodownload\?|\.flv?) 129600 100% 129600 refresh_pattern ^.*(utm\.gif|ads\?|rmxads\.com|ad\.z5x\.net|bh\.contextweb\.com|bstats\.adbrite\.com|a1\.interclick\.com|ad\.trafficmp\.com|ads\.cubics\.com|ad\.xtendmedia\.com|\.googlesyndication\.com|advertising\.com|yieldmanager|game-advertising\.com|pixel\.quantserve\.com|adperium\.com|doubleclick\.net|adserving\.cpxinteractive\.com|syndication\.com|media.fastclick.net).* 129600 20% 129600 refresh_pattern ^.*safebrowsing.*google 129600 100% 129600 refresh_pattern ^http://((cbk|mt|khm|mlt)[0-9]?)\.google\.co(m|\.uk) 129600 100% 129600 refresh_pattern ytimg\.com.*\.jpg 129600 100% 129600 #YOUTUBE refresh_pattern \.ytimg\? 10800 90% 10800 refresh_pattern -i (yimg|twimg).com.* 1440 100% 129600 refresh_pattern -i (ytimg|ggpht).com.* 1440 80% 129600 refresh_pattern -i (get_video?|videoplayback?|videodownload?|.mp4|.webm|.flv|((audio|video)/(webm|mp4))) 241920 100% 241920 store-stale refresh_pattern -i ^https?://..googlevideo.com/videoplayback. 10080 99% 43200 store-stale refresh_pattern -i ^https?://..googlevideo.com/videoplayback.$ 241920 100% 241920 store-stale #XBOX refresh_pattern -i lancache-microsoft 525600 100% 525600 refresh_pattern -i images-eds.xboxlive.com 525600 100% 525600 #catch all refresh_pattern -i (/cgi-bin/|\?) 0 0% 0 refresh_pattern . 0 20% 4320 range_offset_limit 0

Ref:
https://github.com/mmd123/squid-cache-dynamic_refresh-list/blob/master/pfsense%20squid%20refresh%20patterns

https://wiki.squid-cache.org/Features/StoreID#:~:text=“Store%20ID”%20is%20another%20name,algorithm%20via%20a%20helper%20program.

https://github.com/rudiservo/pfsense_storeid

http://www.squid-cache.org/Doc/config/store_id_program/

https://wiki.squid-cache.org/Features/StoreID/DB

https://forum.netgate.com/topic/106614/dynamic-cache-not-work/4

I hope that gives you some good info on ways to get better hit rates.

https://forum.it-monkey.net/index.php?topic=23.0

Check out this how to guide also.

If you are not using IPv6 try to disable AAAA access.

Services/DNS Resolver/General Settings

Under custom

server: do-ip4: yes prefer-ip4: yes do-ip6: no prefer-ip6: no private-address: ::/0 dns64-ignore-aaaa: *.* do-not-query-address: :: do-not-query-address: ::1 do-not-query-address: ::/0

Screenshot 2023-11-01 at 10.04.45 AM.png

6cf0e84c-91b7-4081-b7f1-b828b07fd80c-image.png
This isn't functioning correctly. Am I heading in the right direction?

It seems, per HAProxy documention, that the user needs to exist and it must be able to login without password.
This is against all our policies so I changed the mysql-check back to a basic ping check.

This fixes the Health check, but not the biggest problem: No connection is possible trough HAProxy.

@mcury Thanks for the information

@PF4PFS
Enter a path to a log (/var/run/log) file at "Remote syslog host" in the HAproxy general settings.

As mentioned above, I would remove all actions and only state a default backend for investigation, to see if the site works basicall through HAproxy.