Squid HTTPS/SSL question
-
Hey all, I am pretty new to Pfsense, so excuse me if I am just tooootally missing something, or just having bad luck googling…. But I can't seem to conceptually figure how squid currently works for HTTPS/SSL. I literally just deployed my first Pfsense box yesterday, as I said I am very new to it all, but I can't seem to identify if/how the current version of squid handles encrypted communication.
Reason for my asking, I want to set up the https features, but I have read of issues with things suck as Dropbox, windows updates etc, as well as iPhone/android issues with the required cert management.
I'm sure I'm just missing something, but it seems like nearly every article or post I read are on an old version of Pfsense and an old version of squid. I am not on my network and haven't set up ssh yet to check, but I am on the newest stable version of Pfsense and squid.
-
Squid works either explicitly or transparently. In explicit mode, all clients must be told about the proxy or discover it for themselves via WPAD. With transparent mode, pfSense redirects all web traffic to squid without the client's knowledge. This presents a problem with HTTPS sites because your web browser will flag a possible Man in the Middle attack. The solution to MitM warnings is to install a pfSense-generated certificate in every client that will use the proxy. Depending on the number of clients you have, this could be a tremendous hassle. I personally much prefer explicit mode coupled with WPAD. Let the clients (except for Android grrr) discover the proxy on their own and all is good, no certificate hassles, no fussing with every client.
-
@KOM:
Squid works either explicitly or transparently. In explicit mode, all clients must be told about the proxy or discover it for themselves via WPAD. With transparent mode, pfSense redirects all web traffic to squid without the client's knowledge. This presents a problem with HTTPS sites because your web browser will flag a possible Man in the Middle attack. The solution to MitM warnings is to install a pfSense-generated certificate in every client that will use the proxy. Depending on the number of clients you have, this could be a tremendous hassle. I personally much prefer explicit mode coupled with WPAD. Let the clients (except for Android grrr) discover the proxy on their own and all is good, no certificate hassles, no fussing with every client.
Well, to be fair I am an iPhone user anyway so android issue are much less concern. When I have droid friends over I'll tell them ;)
But that being said what exactly does explicit mode do? I guess I have not gone down the google hole far enough to read about this yet, but it sounds like the solution for me.
-
Explicit mode just means that your network traffic must be directed to the proxy server from the client end. With transparent mode, the redirection happens at the gateway and your clients are unaware that they are going through a proxy.
-
@KOM:
Explicit mode just means that your network traffic must be directed to the proxy server from the client end. With transparent mode, the redirection happens at the gateway and your clients are unaware that they are going through a proxy.
Ok, to make sure I understand this, in Explicit mode the client machines (web browsers) will have to be instructed to use the Proxy. I understand how the web browsers will take advantage of this, but would things like Windows updates see the proxy and utilize the Caching ability? On that point, if something, say a phones browser or Windows update doesn't see the proxy would it just go around it, or it would it not be able to load the content at all?
-
in Explicit mode the client machines (web browsers) will have to be instructed to use the Proxy.
Yes, either manually, via WPAD, or through GPO if you're in a Windows domain environment.
but would things like Windows updates see the proxy and utilize the Caching ability?
Windows updates and some other updates are not easy to cache. I have not played a lot with Squid.conf settings but I haven't managed to figure out the right combination of settings to get them to cache. A lot of content is getting harder to cache. Under pfSense 2.2.x I had a disk cache and my hit rate was 5-7% om average. Now under 2.3.2 I don't have a disk cache at all, I just use squid as the platform for squidguard URL filter.
say a phones browser or Windows update doesn't see the proxy would it just go around it, or it would it not be able to load the content at all?
Depends on whether or not you want to enforce usage of the proxy. If you do, you would add a firewall rule that blocks tcp 80/443 on LAN. That way there's no going around it. Otherwise don't add any firewall rule and you can then just turn proxy support on & off at the client.
-
@KOM:
in Explicit mode the client machines (web browsers) will have to be instructed to use the Proxy.
Yes, either manually, via WPAD, or through GPO if you're in a Windows domain environment.
but would things like Windows updates see the proxy and utilize the Caching ability?
Windows updates and some other updates are not easy to cache. I have not played a lot with Squid.conf settings but I haven't managed to figure out the right combination of settings to get them to cache. A lot of content is getting harder to cache. Under pfSense 2.2.x I had a disk cache and my hit rate was 5-7% om average. Now under 2.3.2 I don't have a disk cache at all, I just use squid as the platform for squidguard URL filter.
say a phones browser or Windows update doesn't see the proxy would it just go around it, or it would it not be able to load the content at all?
Depends on whether or not you want to enforce usage of the proxy. If you do, you would add a firewall rule that blocks tcp 80/443 on LAN. That way there's no going around it. Otherwise don't add any firewall rule and you can then just turn proxy support on & off at the client.
Ah, ok. I think I get it.
But…. let me also add in here I am a single dude, and this is for personal use. I am starting to think with the "its getting harder to cache data" it may not even be worth going any further than just leaving it in Transparent mode and just let it cache what it can, unless I am misunderstanding?Maybe I didn't get it…. I would have to get squid to work correctly in Explicit mode to really benefit from squidguard wouldn't I.
I have yet to even get to squidguard... That is probably my next package. Like I said, I am very new to all of this, and in all honesty its more a "for fun" project than anything.
-
I would have to get squid to work correctly in Explicit mode to really benefit from squidguard wouldn't I.
The operating mode doesn't matter; squidguard depends on squid.
If it's just you at home then you will receive almost no benefit from running squid & squidguard, other than as an exercise. If it's just you or a low number of clients then perhaps transparent mode with a certificate installed on every client is the way to go, but I avoid doing that.
-
@KOM:
I would have to get squid to work correctly in Explicit mode to really benefit from squidguard wouldn't I.
The operating mode doesn't matter; squidguard depends on squid.
If it's just you at home then you will receive almost no benefit from running squid & squidguard, other than as an exercise. If it's just you or a low number of clients then perhaps transparent mode with a certificate installed on every client is the way to go, but I avoid doing that.
I guess one thing I am still not sure about, why wouldn't it be of any benefit? Obviously, I won't have economy's of scale on my side, as in I won't have LOTS of people being able to pull cached data from a local networked SSD which would save a lot of going out and fetching data from the Internet, but wouldn't I still be able to see personal benefits from caching? Maybe it's just not at all a big deal though, like I said I'm still new to it all.
-
why wouldn't it be of any benefit?
Because the general hit rate is very low. I mentioned in a previous post that I'm seeing 5-7% and that's on a corporate LAN with lots of users. It's not easy to cache the dynamic web these days. When you always get a miss on your cache, the extra time to do an object lookup starts to add up and you end up working slower for no real benefit. Now, your mileage may vary based on what sites you typically go to, how they are served, and what kind of refresh_pattern + other options magic you can figure out. Most people give up on Windows Updates and either go with a WSUS server or just give up trying to cache them. Windows 10 updates seem to be easier to cache. Linux updates are dead simple. Do some Googling for 'squid' and 'updates' to see the kind of problems people are having. It can be done from what I understand, but you will end up deep-diving into squid to get there.
http://www.squid-cache.org/
-
Hi there,
I hope my reply does not come across as a hijacking one.
Similar scenario as LIGISTX. pfsense +squid in transparent more + SSL MITM. I just had to deploy certificates to Windows, Linux, iOS and android devices. My home environment consists of 10 machines so super tiny.
I guess I wont see much of a caching benefit?
The primary reason for squid for me was the use of built in Antivirus. I could not find HAVP in the package manager, like many guides reference too. -
I guess I wont see much of a caching benefit?
It depends. Every environment is different. Get it running as best you can and then monitor performance from the console after letting it run for a while. Caching is a tricky art and there is no magic checkbox to tick. Some deep-diving into squid's documentation and some Googling will get you going with refresh patterns and store_ids plus tweaked squid.conf options.
I dislike AV on the firewall because it slows everything down and I don't believe the level of protection is equal to what's offered by the usual commercial companies. Put your AV on the client and let the firewall route packets.
-
@KOM:
I dislike AV on the firewall because it slows everything down
Thank you for the response.
If I throw more resources at pfSense (CPU, RAM) because it's a VM, will it make a difference? -
@KOM:
I dislike AV on the firewall because it slows everything down
Thank you for the response.
If I throw more resources at pfSense (CPU, RAM) because it's a VM, will it make a difference?I'm going to assume no. If the CPU still has to look at all the data even if it was a beast setup, I assume it will still slow it down. But, I personally am not sure of this at all lol.
-
If I throw more resources at pfSense (CPU, RAM) because it's a VM, will it make a difference?
If you can saturate your bandwidth and not have pfSense CPU break a sweat then you're probably good.
-
@KOM:
@KOM Well, I am not sure why this jacked a bunch of stuff up, but I had the server down for the weekend (was working on the hardware), plugged it back in this afternoon and accidentally had the WAN plugged into the LAN which seems to have tripped a lot of stuff out, mostly snort which was blocking all sorta of connections. I cleared the block list (since nothing was previously blocked when I took the pfsense box down), but now squid is acting up on me. It will load a HTTP webpage the first time just fine, but if I try and reload it I get an error, "Connection to "IP ADDRESS" failed, The system returned (1) operation not permitted" error. I am fairly sure it all worked before I took the box down, and I even tried restoring my settings from a previous backup. Turning SQUID off results in no error, but the page not loading.
Any ideas?
-
Uninstalling Squid, after checking the do not save config button and a restart of the client and the pfsense box didn't fix it :/. I am clearly doing something totally wrong. I can ping say newegg.com through cmd though.
-
I would have manually cleared the cache first. Also check for anything in the System log and /var/squid/logs/cache.log.
-
Well, I have since reinstalled Squid and cleared the cache, didn't help :/
I think the easiest thing will be to reinstall pfsense. I really don't know what I did wrong
-
Hi there,
I hope my reply does not come across as a hijacking one.
Similar scenario as LIGISTX. pfsense +squid in transparent more + SSL MITM. I just had to deploy certificates to Windows, Linux, iOS and android devices. My home environment consists of 10 machines so super tiny.
I guess I wont see much of a caching benefit?
The primary reason for squid for me was the use of built in Antivirus. I could not find HAVP in the package manager, like many guides reference too.Can I ask please how you installed on Android? I've installed my certificates, but when I disconnect from my wifi my devices 'connect' but on the devices they say they have no IP address. They work with transparent HTTP but screw up when I add HTTPS, so I have to add them to the bypass filter.
Thanks in advance.
