Problem accessing Spotify via web browser app through Squid

michmoor

@oldschoolrouterjockey
You're most likely MITM yourself. I get it, want to peek inside a TLS session but some sites (majority) wont work for numerous reasons such as certificate pinning.
Either way you need to whitelist the site. Thats it.

oldschoolrouterjockey

@michmoor actually nothing as extensive as that; I'm just doing a simple squid with the Transparent HTTP Proxy option set, and not setting the HTTPS/SSL Interception, so not doing any MITM. or at least I think not. I mean non of my banking sites or anything TLS related ever complains about invalid certs.

michmoor

@oldschoolrouterjockey
Yep , transparent proxy is also an issue as well.

Squid Proxy will be depreciated in future releases on pfSense so its best to start moving away from it. see here
Transparent proxy on pfsense is currently broken. See here

Your option is to create an alias, call it 'BypassProxy' and put the spotify IP(s) in there. Then go to your Squid configuration and in the 'Bypass Proxy for These' put in the alias you created. You dont want that traffic going through a proxy. Also ensure that you have port 443 open in your firewall rules so that bypassed traffic can go outbound.

oldschoolrouterjockey

@michmoor
OK I did all that, I added a ton of different spotify hosts I found in a HAR trace to the firewall alias I created then added that alias to the bypass. It didnt really help anything though so I then added my local ip address to the bypass for the source IP option. Still didnt help.

(im going to make a new post as now I have a TON more questions on squid/proxy/transparent in general)

So I just disabled squid completely and went back to trying to use the local dynamic socks option via ssh to my local pfsense, and found that sometimes I can get logged in. Sometimes. But then after I do get logged then then things only seem to sort of work, sort of. It takes a few minutes after clicking play on something for its audio to actually start. Then if a commercial ever comes on, after its over it takes a few mins to start playing again. Its just really strange and is making me question all I think I know about networking, proxy, etc.

So even using the dynamic proxy, somehow they know? Or maybe is not so much as "they know and are intentionally trying to cause problems when people use proxies" as much as something is just weird going on?

michmoor

@oldschoolrouterjockey

Forward proxies being notoriously difficult to troubleshoot and are known to be problematic in web browsing do initially seem like the culprit in your setup. As you said you added the IPs to the Bypass List and it still isnt working correctly therefore it isnt Squid causing an issue. My next question would be do you have any browser extensions that could be at play here? What happens when you go into incognito mode in the browser?

Gertjan

@oldschoolrouterjockey said in Problem accessing Spotify via web browser app through Squid:

(im going to make a new post as now I have a TON more questions on squid/proxy/transparent in general)

With this update upcoming : Deprecation of Squid Add-On Package For pfSense Software ?

oldschoolrouterjockey

@michmoor said in Problem accessing Spotify via web browser app through Squid:

@oldschoolrouterjockey

Forward proxies being notoriously difficult to troubleshoot and are known to be problematic in web browsing do initially seem like the culprit in your setup. As you said you added the IPs to the Bypass List and it still isnt working correctly therefore it isnt Squid causing an issue. My next question would be do you have any browser extensions that could be at play here? What happens when you go into incognito mode in the browser?

OK So I loaded a brand new FF profile, zero extensions on it, then set the FF config proxy settings to socks (localhost:3128 to correspond to my local SSH session to my netgate), also set the proxy dnx when using socksv5. I can login to spotify just fine, but still takes nearly 2 minutes after hitting play to actually have a stream of music. I can then pause/play instantaneously, but forwarding to the next song again gets me the 2 minute wait.

With the corp vpn disconnected and FF proxy settings set to direct, it all works fine as youd expect.

michmoor

@oldschoolrouterjockey
So the problem isnt with pfSense or Squid.
Im sure you have a specific use case to go through a socks proxy but that could be whats causing you the pain.
Or you are visiting the site while on the corp vpn.
In either case a direct connection through pfsense doesnt cause an issue it seems.

enzo5

If Spotify isn't working through Squid, ensure its domains are allowed, skip SSL bumping for those domains, and check the Squid logs for any issues. You can also clear the cache and update Squid to the latest version. If the problem persists, consider bypassing Squid temporarily or using the Spotify desktop app instead.

JonathanLee

@oldschoolrouterjockey

I hope this helps,

Squid has been updated upstream to fix all the concerns. Again, some sites do certificate pinning.

Squid has been updated to address the security concerns that Netgate was concerned about in the past.

All you need to do to fix this is add some custom settings, Keep in mind both michmoor and I do not work for Netgate I wish I did but I am in school still. I do know that the depreciated notice was sent long before Squid dev team fixed the issues a couple versions ago. I personally purchased my system because at the time the web cache was advertised.

To fix this add the URL requests domain to the do not cache list on squid and also create a custom list, or you can attempt to enable what is called tproxy, you will have to research that I have used that in the past it may not be on the new version of Squid.

Try this create a list of domain issues save it as a text file in some location on the firewall.
I did /usr/local/pkg/
Inside the text file I add the sites that cause issues,

acl BrokenButTrustedServers dstdomain "/usr/local/pkg/dstdom.broken"
acl DomainMismatch ssl_error SQUID_X509_V_ERR_DOMAIN_MISMATCH
sslproxy_cert_error allow BrokenButTrustedServers DomainMismatch
sslproxy_cert_error deny all

or you can create your pass lists like this

acl NoSSLIntercept ssl::server_name_regex -i "/usr/local/pkg/reg.url.nobump"
acl NoBumpDNS dstdomain "/usr/local/pkg/dns.nobump"
acl SSL_Intercept_Terminate dstdomain "/usr/local/pkg/url.bump"
acl block_hours time 00:30-05:00
ssl_bump terminate all block_hours
http_access deny all block_hours
acl getmethod method GET
acl to_ipv6 dst ipv6 #my ISP does not provide ipv6 I have a different boot environment for hurricane tunnel broker 
acl from_ipv6 src ipv6

acl HttpAccess dstdomain "/usr/local/pkg/http.access"
acl windowsupdate dstdomain "/usr/local/pkg/windowsupdate"
acl https_login url_regex -i ^https.*(login|Login).* #do not save or look at anything with login in it do not even miss cache it
cache deny https_login


acl splice_only src IP ADDRESS PLACE HERE

acl splice_only_mac arp MAC ADDRESS PLACE HERE

acl active_use annotate_client active=true

acl bump_only src IP ADDRESS PLACE HERE

acl bump_only_mac arp MAC ADDRESS PLACE HERE

collapsed_forwarding on
negative_dns_ttl 5 minutes
shutdown_lifetime 1 seconds
#pipeline_prefetch 50 Do Not Use this causes risk at times 
happy_eyeballs_connect_timeout 10
memory_pools on
digest_generation off

ssl_bump peek step1
ssl_bump terminate SSL_Intercept_Terminate
miss_access deny no_miss active_use
ssl_bump splice https_login active_use
ssl_bump splice splice_only_mac splice_only active_use
ssl_bump splice NoBumpDNS active_use
ssl_bump splice NoSSLIntercept active_use
ssl_bump bump bump_only_mac bump_only active_use
acl activated note active_use true
ssl_bump terminate !activated

For this example, configuration you are going to need certificates installed on the devices.

Squid works really well, as it does not use DNS requests to inspect traffic and block, it's an amazing set of software. Squid has a support email also. Squid is no joke it needs to be respected. I hope this helps.

Also make sure your webcache does not call out to a wpad for security if you are on the proxy do not use wpad anymore. Added security.

acl wpad urlpath_regex ^/wpad.dat$
acl wpad urlpath_regex ^/proxy.pac$
acl wpad urlpath_regex ^/wpad.da$
deny_info TCP_RESET wpad
reply_header_access Content-Type deny wpad
http_access deny wpad
http_access deny !safeports
http_access deny CONNECT !sslports
http_access allow localhost manager
http_access deny manager
#cachemgr_passwd disable offline_toggle reconfigure shutdown
#cachemgr_passwd CLASSIFIED PASSWORD WAS HERE would require php adaption to use squidlite
eui_lookup on
acl no_miss url_regex -i gateway\.facebook\.com\/ws\/realtime\?
acl no_miss url_regex -i web-chat-e2ee\.facebook\.com\/ws\/chat #do not cache or save any chat anything
acl CONNECT method CONNECT
acl wuCONNECT dstdomain www.update.microsoft.com #http connect
acl wuCONNECT dstdomain sls.microsoft.com
http_access allow CONNECT wuCONNECT localnet
http_access allow CONNECT wuCONNECT localhost
http_access allow CONNECT windowsupdate localnet
http_access allow CONNECT windowsupdate localhost
http_access allow CONNECT HttpAccess localnet
http_access allow CONNECT HttpAccess localhost
http_access deny manager
http_access deny to_ipv6
http_access deny from_ipv6

It is complex and took me years to get it to work, I will never pretend I know everything about it. I am in school as a full computer science major student. Again, I am a student, so this was the only way to learn about proxy code for me was to start tinkering with this. Most everything for proxy is closed source that works like this, this was used as an educational tool for cybersecurity for me. I am sure there was many times I had messed up items. I have learned a lot and still do. At this point some websites are slow for me and others are lighting fast.

It is like a tank, keep in mind the config is pulled from several places. ACLS come before the bump slice stuff always I got some mixed up in this post, some of my devices few are always bump everything and others are splice always and do not inspect anything at all stuff that is not mine is bypassed that way.

JonathanLee

@michmoor You ever configure Squid with http tproxy ? I tested this it was amazing again any reboot or enable reset puts it back to the old way.