Wireless router

johnpoz

@Antibiotic Unless your like on dialup, I don't see how you think a proxy is going to speed up anything these days.. Most everything id dynamic and created on the fly, and you have the added overhead of having to do stuff to get https to work.. And there is nothing really still on http that is for sure.

JKnott

@Antibiotic said in Wireless router:

129.168.20.1

I trust that's a typo.

What addresses do you have on the LAN side of pfSense? All the addresses I see are in the same subnet.

johnpoz

@JKnott hahha - good catch on the typo ;) yeah that has to be a typo.. just transposed the 9 and the 2

JonathanLee

@Antibiotic I purchased an official Netgate appliance just to learn about proxy use when I was completing my AA in cyber security.

Squid is one of the most complex packages to configure. But it is a dream to see run once it is configured. It is my favorite thing about my firewall the greatest puzzle for me ever. I love it. Like the hypothetical ultimate Rubik's cube.

You can use advanced options in Squid for just about anything you can dream of.

So yes, you can configure specific conditions in Squid.

I have timers set so that it turns off the cache after hours, some devices are always splice like an amazon tablet. I have devices locked to mac addresses with IP addresses to use the cache.

I have a book for Squid it 442 pages of amazing things you can do.

It is complex like @johnpoz said.

I also use it to do dynamic updates, so I do not have to sit and wait for a Windows 11 update over and over between different machines.

Here is an example of my custom config if it helps.

Just forget the cachemgr and the updates and all that focus on the IP addresses and mac addresses stuff also the timers.

cachemgr_passwd disable offline_toggle reconfigure shutdown #for use with cachemgr.cgi if you have linked the file
cachemgr_passwd REDACTED all #for use with cachemgr.cgi if you have linked the file
eui_lookup on #for use with mac addresses
acl no_miss url_regex -i gateway\.facebook\.com\/ws\/realtime\? #for use with never caching facebooks real-time checker
acl no_miss url_regex -i web-chat-e2ee\.facebook\.com\/ws\/chat #for use with never caching Facebook messages that is bad and creepy if you do that it is unethical so splice and move on
acl CONNECT method CONNECT #for use with windows update cache use over the proxy to help
acl wuCONNECT dstdomain www.update.microsoft.com #same use with windows updates
acl wuCONNECT dstdomain sls.microsoft.com #also use with windows updates
http_access allow CONNECT wuCONNECT localnet #allows connections for windows updates to the cache to connect to it
http_access allow CONNECT wuCONNECT localhost
http_access allow windowsupdate localnet #for use with my update file I hand crafted within the dynamic cache area
http_access allow windowsupdate localhost #same thing
http_access allow HttpAccess localnet #other services that require http access like antivirus updates and other enterprise services that require it
http_access allow HttpAccess localhost #same thing allow it to local host too
http_access deny manager #block manager access over http
http_access deny to_ipv6 #my isp does not use ipv6 so I block it
http_access deny from_ipv6

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH #for use with squid seeing a broken domain connection when updates occur 
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all
#ok to the acls most of the above you can research on Squids website
acl splice_only src 192.168.1.8 #Tasha iPhone
acl splice_only src 192.168.1.10 #Jon iPhone
acl splice_only src 192.168.1.11 #Amazon Fire
acl splice_only src 192.168.1.15 #Tasha HP
acl splice_only src 192.168.1.16 #iPad

acl splice_only_mac arp MAC ADDRESS HERE
acl splice_only_mac arp MAC ADDRESS HERE
acl splice_only_mac arp MAC ADDRESS HERE
acl splice_only_mac arp MAC ADDRESS HERE
acl splice_only_mac arp MAC ADDRESS HERE

#this uses a no bump file I created for specific websites that ethically should always never intercept like banks etc don't do that. 
acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/reg.url.nobump"
acl NoBumpDNS dstdomain "/usr/local/pkg/dns.nobump"


acl markBumped annotate_client bumped=true #I created a flag here for use later
acl active_use annotate_client active=true
acl bump_only src 192.168.1.3 #webtv
acl bump_only src 192.168.1.4 #toshiba
acl bump_only src 192.168.1.5 #imac
acl bump_only src 192.168.1.9 #macbook
acl bump_only src 192.168.1.13 #dell

acl bump_only_mac arp MAC ADDRESS HERE
acl bump_only_mac arp MAC ADDRESS HERE
acl bump_only_mac arp MAC ADDRESS HERE
acl bump_only_mac arp MAC ADDRESS HERE
acl bump_only_mac arp MAC ADDRESS HERE

ssl_bump peek step1 #peek at step one look for my block items if its not there let it connect
miss_access deny no_miss active_use #do not cache the no miss I hate seeing miss saved for that
ssl_bump splice https_login active_use #splice the http here also and move on flag it active
ssl_bump splice splice_only_mac splice_only active_use now we splice specific ip addresses and the mac so mac and ip must match and flag it
ssl_bump splice NoBumpDNS active_use same here splice the no bumps 
ssl_bump splice NoSSLIntercept active_use spice the no intercept file
ssl_bump bump bump_only_mac bump_only active_use #always bump stuff like my research equipment for cybersecurity I want to see everything going on I want to see the containers in action I want every detail from it
acl activated note active_use true ok take all the active use put it into acl activated
ssl_bump terminate !activated kill everything not activated incase squid has a container or something in it terminate the connection. 

acl markedBumped note bumped true #other research I was doing
url_rewrite_access deny markedBumped

#for use with my quality of service to help speed up the proxy

read_ahead_gap 64 KB
negative_ttl 1 second
connect_timeout 30 seconds
request_timeout 60 seconds
half_closed_clients off
shutdown_lifetime 10 seconds
negative_dns_ttl 1 seconds
ignore_unknown_nameservers on
pipeline_prefetch 100


# I am not using this I was creating an always bump file but didn't really finish it. 
#acl SSLIntercept ssl::server_name_regex -i "/usr/local/pkg/url.bump"
#ssl_bump bump SSLIntercept

Ok so now that you got some basics of that remember order matters so dynamic cache advanced configure come before general advanced config so some ACLs are created here but used in the other advanced config.

acl block_hours time 01:30-05:00 #my network lock hours
ssl_bump terminate all block_hours #terminate all during block outs so nothing gets in nothing gets out. At least I hope so
http_access deny all block_hours #same block the http access during specific hours
acl getmethod method GET #set get for use with storeID
acl to_ipv6 dst ipv6 #for user with blocking ipv6 and stuff 
acl from_ipv6 src ipv6

#ciphers for ssl intercept and tls use
tls_outgoing_options cipher=HIGH:MEDIUM:!RC4:!aNULL:!eNULL:!LOW:!3DES:!MD5:!EXP:!PSK:!SRP:!DSS
tls_outgoing_options options=NO_SSLv3,SINGLE_DH_USE,SINGLE_ECDH_USE

#my acls for my HttpAccess for antivirus it pulls up my huge file so I don't have a million reg expressions 
acl HttpAccess dstdomain "/usr/local/pkg/http.access"
acl windowsupdate dstdomain "/usr/local/pkg/windowsupdate"
acl rewritedoms dstdomain "/usr/local/pkg/desdom"

#my store_id program it comes built in with squid but you need the files created and a database to use it
store_id_program /usr/local/libexec/squid/storeid_file_rewrite /var/squid/storeid/storeid_rewrite.txt
store_id_children 10 startup=5 idle=1 concurrency=0
always_direct allow !getmethod
store_id_access deny connect
store_id_access deny !getmethod
store_id_access allow rewritedoms
reload_into_ims on
max_stale 20 years
minimum_expiry_time 0
#this is refresh patters after this for updates etc or other items you don't want to download a million times let the proxy reserve it securely. 
refresh_pattern -i squid\.internal 10080 80% 79900 override-lastmod override-expire ignore-reload ignore-no-store ignore-private

#FACEBOOK
refresh_pattern ^https.*.facebook.com/* 10080 80% 43200

#FACEBOOK IMAGES  
refresh_pattern -i pixel.facebook.com..(jpg|png|gif|ico|css|js|jpg?) 10080 80% 43200
refresh_pattern -i .akamaihd.net..(jpg|png|gif|ico|css|js|jpg?) 10080 80% 43200 
refresh_pattern -i facebook.com.(jpg|png|gif|jpg?) 10080 80% 43200 store-stale
refresh_pattern static.(xx|ak).fbcdn.net.(jpg|gif|png|jpg?) 10080 80% 43200
refresh_pattern ^https.*profile.ak.fbcdn.net.*(jpg|gif|png|jpg?) 10080 80% 43200
refresh_pattern ^https.*fbcdn.net.*(jpg|gif|png|jpg?) 10080 80% 43200

#FACEBOOK VIDEO
refresh_pattern -i .video.ak.fbcdn.net.*.(mp4|flv|mp3|amf) 10080 80% 43200
refresh_pattern (audio|video)/(webm|mp4) 10080 80% 43200

#APPLE STUFF
refresh_pattern -i apple.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip|dist)$ 0 80% 43200  refresh-ims

#apple update
refresh_pattern -i (download|adcdownload).apple.com/.*\.(pkg|dmg) 4320 100% 43200
refresh_pattern -i appldnld\.apple\.com 129600 100% 129600
refresh_pattern -i phobos\.apple\.com 129600 100% 129600
refresh_pattern -i iosapps\.itunes\.apple\.com 129600 100% 129600


#my favorite Windows update cache I hate redownloading items a millions times, this makes it green energy efficient. 
# Updates: Windows
refresh_pattern -i microsoft.com/..(cab|exe|msi|msu|msf|asf|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|msi|msu|msf|asf|wma|wmv)|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i windows.com/..(cab|exe|msi|msu|msf|asf|wmv|wma|dat|zip)$ 4320 80% 43200  refresh-ims
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i windows.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 4320 80% 43200 
refresh_pattern -i .*windowsupdate.com/.*\.(cab|exe) 259200 100% 259200   
refresh_pattern -i .*update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 259200 100% 259200   
refresh_pattern windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern download.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern www.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern au.download.windowsupdate.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200 
refresh_pattern bg.v4.pr.dl.ws.microsoft.com/.*\.(cab|exe|dll|msi|psf) 4320 100% 43200
#windows update NEW UPDATE 0.04
refresh_pattern update.microsoft.com/.*\.(cab|exe) 43200 100% 129600    
refresh_pattern ([^.]+\.)?(download|(windows)?update)\.(microsoft\.)?com/.*\.(cab|exe|msi|msp|psf) 4320 100% 43200  
refresh_pattern update.microsoft.com/.*\.(cab|exe|dll|msi|psf) 10080 100% 43200 
refresh_pattern -i \.update.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.download.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
refresh_pattern -i \.ws.microsoft.com/.*\.(cab|exe|ms[i|u|f]|[ap]sf|wm[v|a]|dat|zip) 525600 100% 525600       
    
refresh_pattern ([^.]+\.)?(cs|content[1-9]|hsar|content-origin|client-download).[steampowered|steamcontent].com/.*\.* 43200 100% 43200     
refresh_pattern ([^.]+\.)?.akamai.steamstatic.com/.*\.* 43200 100% 43200

refresh_pattern -i ([^.]+\.)?.adobe.com/.*\.(zip|exe) 43200 100% 43200
refresh_pattern -i ([^.]+\.)?.java.com/.*\.(zip|exe) 43200 100% 43200
refresh_pattern -i ([^.]+\.)?.sun.com/.*\.(zip|exe) 43200 100% 43200
refresh_pattern -i ([^.]+\.)?.oracle.com/.*\.(zip|exe|tar.gz) 43200 100% 43200

refresh_pattern -i appldnld\.apple\.com 43200 100% 43200
refresh_pattern -i ([^.]+\.)?apple.com/.*\.(ipa) 43200 100% 43200
 
refresh_pattern -i ([^.]+\.)?.google.com/.*\.(exe|crx) 10080 80% 43200
refresh_pattern -i ([^.]+\.)?g.static.com/.*\.(exe|crx) 10080 80% 43200

acl https_login url_regex -i ^https.*(login|Login).*
cache deny https_login


#windows updates a huge but you don't want to allow just anything to get in you cache so set it to only allow specific acls to access the larger limits and set the rest as standard
range_offset_limit 512 MB windowsupdate
range_offset_limit 4 MB
range_offset_limit 0
quick_abort_min -1 KB

It was a quest to make that thing work not to mention you need to also worry about blocking DoH and use a blacklist.

So that is my research so far.

But transparent mode is 100 percent easier over this right?

My recommendation is play with the built in guy and use the ACL page set up some items to bypass the proxy if needed under general config.

I am a student remember, and I am learning so I could and most likely have some mistakes I made this post so you can take the parts you want to play with and use that to research with. Most of it is made with a goal of caching any and all containers before they reach a machine, and also making my network green and efficient, let the proxy serve the items already downloaded but do it ethically right don't intercept banks and email and messages and stuff just avoid it and move on.

Antibiotic

@JonathanLee Oh, thank you for public your hard work. I will take a look slowly)))

JonathanLee

@Antibiotic Don't forget I am a student so to be on the official side use Netgate's GUI options with the transparent options it works great like that but you could also use that IP address mac address, and timed options too. I think my brain melted after getting it to work. @johnpoz and others helped me a lot over the years.

johnpoz

@JonathanLee said in Wireless router:

I also use it to do dynamic updates, so I do not have to sit and wait for a Windows 11 update over and over between different machines.

You understand windows does that on its own unless you have turned it off, it will pull updates from other machines on your network vs having to pull from the internet again..

https://support.microsoft.com/en-us/windows/windows-update-delivery-optimization-and-privacy-bf86a244-8f26-a3c7-a137-a43bfbe688e8

JonathanLee

@johnpoz Yes but it is cool to see it say refresh 304 and hit 200s for a huge update and it download in 2 seconds it's crazy fast to see it run John It is amazing. It reminds me of a Juniper content accelerator. Again, it is a lot to configure, if users could just click a radio button for the best of the best caching/acceleration options that would make it easy for end users.

Antibiotic

@JonathanLee Hello, any benefits for anonymity to check : Disable VIA Header and to make OFF X-Forwarded Header Mode. I will use non transparent proxy for one of my subnet, mostly for basic protection and gaming! I'm a little bit in doubt, will cache working in non transparent mode or he is working only in transparent mode? I have set for local cache "Memory Cache Size" 4GB but memory usage on main dashboard the same or squid use this dinamically? How to check that cache is working and increasing? Also for safety set Hard Disk Cache System to diskd , is it OK?

JonathanLee

@Antibiotic I use X-Forwarded Header Mode set to transparent as it was having issues with my IP looking like a private address with Netgate forums website.
That fixed it

I do not disable the VIA header so my requests follow RFC2616.

I do suppress the version however.

Hope that helps, The X-Forwarded Header Mode was causing issues with my system and setting it to transparent helped Netgate's staff helped me with that because I could not see that my IP was showing up incorrectly and causing issues, I am not behind a lot of equipment so I don't need it enabled.