[SOLVED]Help needed: bypass squid and squidGuard for iTunes, AppleStore, Android

[GL]

Hello All,
I have been able to apply the Windows Upgrade bypass without problems in the SquidGuard.conf file, so that all my Win devices can update bypassing Squid (running with https transparent proxy and squidGuard).

I have problems with my Apple deivces which cannot access itunes, apple store nor upgrade.
The only solution found on the net is to list Apple ip addresses in the squid webgui.

Apple, however, is using Akamai network, as Microsoft, and trying to bypass squid via IP addresses is pratically impossible (Akamai network is huge). If you input some Akamai address you find via nslookup in squid bypass, the patch is working for a while before you will be redirected to some other server.

I am sure there exist a way to let Apple devices bypass squid and squidGuard via squid.conf and squidGuard.conf files, but I have not been able to reach the result. The only success for me has been to stop squid from restarting up to when I did not restore the old squid.conf file…

Is there anybody who can tell us how, what and where we should add/modify in the squid.conf and SquidGuard.conf files, as per the Wind Update bypass, to get the result ( ie bypassing squid when addressing apple web sites)?

Thanks in advance to the nice and kind guy who will give us the instructions.
I am sure that your answer will become one of the most viewed posts on this board....

Ciao,

GL

[GL]

just un update: I would like to bypass squid only when reaching itunes and AppleStore for upgrading and/or downloading new apps.
during normal navigation I would like to be proxied by squid.

If I remove SSL transparet proxy option and leave squid as http transparent proxy only, everything works fine.

[slim2016]

squid proxy server /  general /[Transparent Proxy Settings] - Bypass Proxy for These Destination IPs

That should do the trick

[GL]

Hello ,
thanks for your reply but this is not the definitive solution.
apple is using akamai network and their servers ip are continuosly changing…
adding the IPs to squid works for a while, then you are back with the problem.

The IP s are also geolocated and changing country by country.

Is there a definitive set of squid directives to be added to squid.conf and squidGuard.conf to fix the problem forever?

After adding the Windows Update fixing to squidGuard.conf, I am sure this can be done also for Apple, but my knowledge of squid  is not allowing me to do that alone.

Pls, help me and the rest of pfsense supporters who are also Apple users.
thans in advance.

Ciao,
GL

[sichent]

Not a user of transparent proxy but not sure why adding these as SNI bypass should not work on the recent Squid
https://docs.diladele.com/faq/squid/sslbump_exlusions/apple_app_store.html

[GL]

thanks for the reply, however actually the webgui allows you to add exclusions only with IPs, and because Apple is using Akamai, servers IP are floating and they are potentially hundreds…
we need to add the exclusion in the config files, as per the Windows Update workaround, the problem is that me and most of the not advanced users do not know how.
I tried to replicate that solution using the windows update workaround, but it is not working. I am missing something...
Is there anyone able to help us?
Thanks in advance.
Ciao,
GL

[GL]

I found the solution: simply there is no solution.
If you use Squid as https transparent proxy, the only info not encrypted that reaches Squid is the IP of the servers, so Squid has no opportunity to read the domain names of the servers.
If you want to use Squid on https, you must use it as explicit proxy.
This is also why in pfsense webgui you can exclude Squid only based on IPs and not domain names.

[sichent]

Not exactly correct; the version of Squid in pfSense is good enough to use the SNI info from the HTTPS connection being established.
The acl you need to use is ssl::server_name

For example this what we use when filtering all connections except spliced in Web Safety (does not apply to SquidGuard!!):

acl ssl_exclude_domains ssl::server_name "/opt/websafety/etc/squid/ssl/exclude/domains.conf"

ssl_bump peek step1 all
ssl_bump splice ssl_exclude_domains
ssl_bump stare step2 all
ssl_bump bump all

I am not sure where to put that in pfSense UI - but should not be very complex to find.

[GL]

thanks for your input, I will try this we and will let everybody know the outcome.
Cross our fingers…
I think this will be a very important input for most of the "home" users.
ciao,
GL

[GL]

I implemented the solution and for now it seems working, I am testing it.
here a step by step guide of what i implemented.

1. Go to Services->Squid Proxy Server

2. Enable and configure HTTPS transparent proxy

3. Go to the bottom of the page, click Show Advanced Options

4. Cut and past the following text in the box "Custom Options (Before Auth)":

acl ssl_exclude_domains ssl::server_name "/usr/local/etc/squid/exclude_domains.conf"

ssl_bump peek step1 all
ssl_bump splice ssl_exclude_domains
ssl_bump stare step2 all
ssl_bump bump all

5. wait to save

6. connect with secure shell to pfsense and login

7. choose option 8 "Shell"

8. cd /usr/local/etc/squid

9. ee exclude_domains.conf

10. input the following text :

.apple.com
.mzstatic.com
.icloud.com
.dropbox.com
.microsoft.com
.oneDrive.com
.live.com
.messenger.live.com
.skype.com
.trouter.com
.login.live.com
.whatsapp.com
.whatsapp.net

11. press "esc" then press "a", again press "a"

12. go back to pfsense webgui and save your squid configuration

13. restart squid service

The configuration in exclude_domains.conf should let you use your Apple devices with iTunes and Apple store, let you use WhatsApp, sync with iCloud (pls check also allowed ports in your firewall rules), sync with OneDrive, let your skype work with https transparent proxy, allow you to update with Microsoft in Win10.

I am still testing, if someone will implement it, pls let us know the outcomes.

[GL]

Adding support also for Android devices.

Add following entries to exclude_domains.conf file:

.ggpht.com
.play.googleapis.com
android.clients.google.com
www.googleapis.com
.gvt1.com

It seems working for me, up to now.

[GL]

Adding following entries to exclude_domains.conf you should bypass Netflix. I am still testing this.

.netflix.com
.llnwd.net
.edgesuite.net
.nflximg.com
.nflxvideo.net

[GL]

Up to today no main issues found.
Everything seems working well.
Thanks to sichent for his help and suggestion!

[tapout72]

@gl said in [SOLVED]Help needed: bypass squid and squidGuard for iTunes, AppleStore, Android:

acl ssl_exclude_domains ssl::server_name "/usr/local/etc/squid/exclude_domains.conf"
ssl_bump peek step1 all
ssl_bump splice ssl_exclude_domains
ssl_bump stare step2 all
ssl_bump bump all

GL, I know this is an old thread but I tried your solution to try and bypass netflix. However, I get a Privacy error NET::ERR_CERT_AUTHORITY_INVALID on every site I try to access after applying the Custom Options (Before Auth).

Any ideas on where I went wrong?

[rob_kae]

@tapout72 did you manage to sort this? I have the same problem.

[vlurk]

I implemented this solution in pfSense 2.7.0 and you can make this solution resilient to package updates by leveraging the whitelist in the ACL tab. In the default config of the package, you can already splice exceptions and bump everything by default. The alias of the acl it creates for sslbump is simply sslwhitelist. And you can reference it in custom options, so this solution would look like this:

ssl_bump peek step1 all
ssl_bump splice sslwhitelist
ssl_bump stare step2 all
ssl_bump bump all

Note that the ACL accepts regex! Which is what I use exclusively. Here is an example of a regex to match microsoft.com domain and subdomains:

(^|.)microsoft.com$

I now get all the domains I have defined logged as TCP_TUNNEL, so I can tell this is working well too.

[azmodeuz]

@vlurk Thank you for this guide. I have the same issue but with Viber. How can I use your settings for viber desktop App