Error with some website

rjcab

Hello,

I use pfsense as FW, Transparent Proxy, and DHCP server.
For several sites it doesn't display all content such as amazon or netflix:

I suppose proxy filters ads but I don't know where I could modify options to get the whole content.

In the proxy log I found that line:

slack-files2.s3-us-west-2.amazonaws.com:443

Thanks

stephenw10

If you disable the proxy or add that host to the bypass list does it load normally?

rjcab

@stephenw10 thanks a lot, if I disable the proxy, it works.
But I want to get all visited website on my network, how to proceed ?

Many Thanks

stephenw10

Well it seems very likely parts of the site are being filtered. Amazon have a lot of ads on their pages!

I assume you have Squidguard installed with ad lists loaded? If you disable filtering does the page load correctly?

What is logged as blocked when you visit that page?

rjcab

@stephenw10 you mean pfBlockerNG ? Not I don't have.
Only these packages installed:

stephenw10

No I meant Squidguard.

Are you using blacklists directly in Squid to filter URLs then?

rjcab

@stephenw10
In the proxy menu I haven't changed anything.

stephenw10

@rjcab said in Error with some website:

I suppose proxy filters ads

Where are you doing that filtering then?

rjcab

@stephenw10

Well pretty good question. I am newbee on Pfsense

I assume that in disabling squid it works so the config should be below on one of these tab:

But I don't see, I have just configured the proxy to catch all visited websites.

Gertjan

@rjcab said in Error with some website:

the proxy to catch all visited websites

be aware : there are many 'sites' that can't be 'proxied'. And this list grows every day.
To make things worse, it's easy for a web site administrator to forbid his web site being proxied. Use a HSTS flagged certificat, and a web browser can't use a proxy anymore to get that site. There will be a certificat failure.
Or the web site, like your amazonaws example, detects that a proxy is used, and they do not allow that : they inform the client, using a very cryptic message, that visiting "amazonaws" only works if the MITM is disabled.

Quiet understandably, as bank web sites, medical data web site, and actually any web site, and any visiting client (that is you) doesn't want a MITM as that opens the door to all kind of issues.

So, when you decide to use a proxy, you have to baby-sit it every day and collect the web sites that "don't work when handled by the proxy" and add them to the "don't proxy this web site" list. This list will eventually grow to the "all the sites avaible on the internet" and that will be the day MITM has been dealt with.

rjcab

@Gertjan Thanks, I now understand the approach.
I have done this:

It seems that is doen't work, I will try with other websites.

Gertjan

@rjcab said in Error with some website:

It seems that is doen't work

The "whitelist" accepts URLs like that ? Or host names ?

"amazon.fr" probably uses and redirects to something other host name(s). And of course, these other host name(s) can be different tomorrow.

To see what happens : (I used Firefox) :
Open amazon.fr
Goto Menu > More tools > Web Developers Tools
Select Network (between Memory and Storage) and hit Crtl-F5

You saw the list with host names flying by ? You probably have to add them all.

See it like this : 'they', amazon, but also Google, Apple, Microsoft, and actually all the big players, all hired the best network engineers just to make your live, doing MITM, hard.
Outsmarting them .... are you sure you want to go down that path ?

rjcab

@Gertjan many thanks, I will try your advise

rjcab

Hello,

I made some tests and add URL as below

When I tried to access with a mobil device it doesn't work as maybe the URL or content is different. I will continue to investigate

stephenw10

The whitelist should be domains not URLs. So:

amazon.fr
leboncoin.fr
netflix.com

rjcab

@stephenw10 thank you. I tried but still the issue. I think there are more URL with netflix on mobile device to whitelist

Also I have tested to connect to my company VPN. It works but once connected I don't have access to Gmail whereas if I disable squid it works. Maybe I have to whitelist the network adress of my company which provide internet services ?