Squid HTTPS/SSL filtering 2017 [Is this it?]



  • Hi everyone,

    I'm new to the forums, and new to Pfsense. My goals are to setup HTTPS/SSL filtering at home. As most of you know, since Google pushed HTTPS, more and more websites started using it. On top of that hackers are getting more and more advanced, and are using HTTPS themselves to bypass detection from IDS's etc. So I find it extremely stupid to have filtering that doesn't cover HTTPS, as there's also all those web based proxies and even ClamAV won't detect viruses sent over HTTPS (tested with eicar).
    I have read a lot of threads, however I am running into a lot of issues and just can't seem to crack it. I've spent a few days frantically trying to sort it out.

    For starters, I have Squid and Squidguard setup, I've got WPAD setup and devices seem to be picking up the proxy settings just fine, and I have also created a CA certificate and have it installed on my devices. Here's the issue I am getting as of now, it seems squid is changing the domain from the domain typed in to "http" :

    Some of the other issues I am getting is slow browsing intermittently. Could it be related to PowerD? I have tried changing around the settings but it doesn't seem to have helped much. Furthermore what kind of hardware is recommended for HTTPS decryption? I'm literally just running it on a old machine that I've thrown in an extra dual NIC.

    Another thing I want to ask is how are you all getting past HSTS? Do you guys setup bypasses for HSTS sites such as Google? It's really frustrating. My college has a similar to what I want to do here, they run Smoothwall and everything seems to work perfectly. Anyone is able to bring their own device into college, and use the internet as long as they install the certificate. And there's barely any errors at all. Their blocking system seems to work even with random websites, for example if iadoi3.com was a porn site, it would detect it. Not based on looking at the URL but actually looking through the website and inspecting it. Which I think is incredibly clever and advanced.
    How would I setup something like this to work efficiently? Is it even possible with pfsense? As of now, squid seems to be quite finnicky.
    I've of course also got friends and guests coming over, I don't want to spend ages fiddling around with their device trying to set it up with my network. In terms of the certificate, I have got a link on my captive portal for people to download and install it.

    Oh, and another thing I nearly forgot. Android devices are really getting a hard time picking up the proxy settings. Could I just force the traffic to go through Squid? How would I do that? Just with port forward?

    Here's my setup:
    2.3.2-RELEASE-p1 (i386)
    Intel(R) Celeron(R) CPU 420 @ 1.60GHz
    2GB of RAM
    140GB Hard drive
    NIC : HP NC360T
    Squid version: 0.4.29_1
    SquidGuard version: 1.14_4

    Installed packages:

    Hope someone could really help me out with this, I'm sure many people are going through the same trouble.
    Thank you in advance! Happy new year!


  • Banned

    @pfsensation:

    it seems squid is changing the domain from the domain typed in to "http"

    No, it's not.  See https://redmine.pfsense.org/issues/6777#note-2

    If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)



  • @doktornotor:

    @pfsensation:

    it seems squid is changing the domain from the domain typed in to "http"

    No, it's not.  See https://redmine.pfsense.org/issues/6777#note-2

    If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

    This is just extremely strange. What would be best to use as a replacement for Squid? There really does seem no proper fix for this.


  • Banned

    There is no need to replace Squid, this is not a Squid issue. Anyway, see https://www.freshports.org/www/e2guardian/ (the GUI/pfSense package got never finished - https://github.com/pfsense/pfsense-packages/pull/866, https://forum.pfsense.org/index.php?topic=87526.0). As for squidGuard replacements, there's https://www.freshports.org/www/ufdbguard/ (not available in pfSense and obviously no GUI for that.)



  • I wouldn't bother with MITM with a CA to be honest…  If you are using WPAD your browser and SQUID both should be letting HTTPS traffic pass through properly.  This happens even for HSTS based websites like Google.com.  However this also negates the ability of ClamAV working, which, is useless anyways.



  • @C0RR0SIVE:

    I wouldn't bother with MITM with a CA to be honest…  If you are using WPAD your browser and SQUID both should be letting HTTPS traffic pass through properly.  This happens even for HSTS based websites like Google.com.  However this also negates the ability of ClamAV working, which, is useless anyways.

    Is it even possible to work like this? From what I understand, I cannot do any HTTPS filtering without MITM as the traffic is encrypted.

    @doktornotor:

    There is no need to replace Squid, this is not a Squid issue. Anyway, see https://www.freshports.org/www/e2guardian/ (the GUI/pfSense package got never finished - https://github.com/pfsense/pfsense-packages/pull/866, https://forum.pfsense.org/index.php?topic=87526.0). As for squidGuard replacements, there's https://www.freshports.org/www/ufdbguard/ (not available in pfSense and obviously no GUI for that.)

    Thanks for that, I will have a look at e2guardian, from the description, it does look like what I would wanna use but it's quite outdated. However, what I really wanted to know is how others are filtering HTTPS content using Pfsense in this day and age. I'm not an expert, but I know there's many hoops to jump through as HTTPS was designed so that inspection couldn't be done easily (HSTS for example).

    I would really appreciate if someone who is already doing this could message, and give me pointers to the system they're using and how they're jumping through the hoops I mentioned above.



  • shrugs I don't use MITM and CA's on my network, but I use WPAD and I seem to get a "block page" any time I hit a blocked secure website, granted it doesn't show the actual block page since you can't intercept, so SQUID just breaks the tunnel.

    Blocked Secure Site

    Allowed Secure Site

    I also have Transparent disabled…  If you are wanting to get the block page that you would normally use however, not sure how it would be possible all because of HSTS.



  • @C0RR0SIVE:

    shrugs I don't use MITM and CA's on my network, but I use WPAD and I seem to get a "block page" any time I hit a blocked secure website, granted it doesn't show the actual block page since you can't intercept, so SQUID just breaks the tunnel.

    Blocked Secure Site

    Allowed Secure Site

    I also have Transparent disabled…  If you are wanting to get the block page that you would normally use however, not sure how it would be possible all because of HSTS.

    I'm having that same issue, but I think it maybe linked to chrome. The block page correctly displays on other browers, and devices. It's kinda sad really, I'm sure others have the setup I want but not sure if they're not willing to share. Or haven't seen this thread yet.

    Squid is making me want to bash my head on the wall…

    EDIT : The page has been fixed on Chrome now, after the latest update. I'm using Chrome beta.


  • Banned

    This might be related - https://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html
    In short - browser ignores everything but 200 Connection established from initial CONNECT tunnel setup to remote HTTPS sites thorough proxy (hope this is not too technical)



  • @sichent:

    This might be related - https://docs.diladele.com/faq/squid/cannot_connect_to_site_using_https.html
    In short - browser ignores everything but 200 Connection established from initial CONNECT tunnel setup to remote HTTPS sites thorough proxy (hope this is not too technical)

    Thank you for the reply @Sichent, so is Diladele the only filtering system available for pfsense that's actually actively updated and maintained? Doktornotor seems to be working extremely hard trying to fix Squid itself, but issues like this just remain. I haven't been able to use anything else decent enough that also does SSL inspection of sorts, I'm stuck relying on SquidGuard which is half broken and OpenDNS.

    I'm not sure if others are not doing this, or just don't want to share their methods. But it seems bizarre to me, how can everyone be filtering without HTTPS… As most do just use Squid + Squid Guard.


  • Banned

    Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.



  • I've got HTTPS filtering running with squid+clamav+squidguard on pfsense.
    If you want, i can send you my configs ive changed and try to get it running that way.
    If that works for you too, we can make it public in this forum.

    I still have to exclude some sites. for example some sites with certificate pinning activated…



  • @spanguel:

    I've got HTTPS filtering running with squid+clamav+squidguard on pfsense.
    If you want, i can send you my configs ive changed and try to get it running that way.
    If that works for you too, we can make it public in this forum.

    I still have to exclude some sites. for example some sites with certificate pinning activated…

    Sounds great, I'd love to try it. How did you go about excluding sites? Excluding doesn't seem like a good method because more and more websites are jumping on the HSTS bandwagon, Github, Python, and of course Google, just to name a few. Furthermore, Squid guard also doesn't do any kind of content inspection, it just seems to work off a black list, I'm not sure, but I think I can't create different levels of user groups and block certain websites for certain people either.



  • Yes, hello. Can you please share the config to me. I am also having big https problems with squidguard. Some https sites working, others not.





  • Thanks for that.



  • But what about actually decrypting and scanning?



  • @pfsensation:

    But what about actually decrypting and scanning?

    Do you mean using the pfsense antivirus to scan https files? Or something else?



  • @aGeekHere:

    @pfsensation:

    But what about actually decrypting and scanning?

    Do you mean using the pfsense antivirus to scan https files? Or something else?

    I mean using pfsense to scan and look into SSL/HTTPS traffic to do certain things like identify new websites to block. Currently I am using OpenDNS + SquidGuard and Shalla and it pretty much sucks. If there's a new domain which isn't in those lists, it'll open. E2 Guardian seems like it's perfect for me but it's still got issues as I understand, and doesn't work as well as it should.



  • The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.



  • @aGeekHere:

    The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.

    That's pretty old and basic. I'm already using those but it's not good enough. What about uncategorized websites, that's where OpenDNS, shalla list and these methods really fail.



  • @pfsensation:

    @aGeekHere:

    The best way I found is to force google or bing into safe mode and use squidguard to catch the rest.

    That's pretty old and basic. I'm already using those but it's not good enough. What about uncategorized websites, that's where OpenDNS, shalla list and these methods really fail.

    Well you cannot find these uncategorized websites using google/bing safe search so someone would have to find a new website (not using safe search) that is not added to shalla list yet and go to that website direct. Well like with all things if there is a will there is a way.



  • @doktornotor:

    @pfsensation:

    it seems squid is changing the domain from the domain typed in to "http"

    No, it's not.  See https://redmine.pfsense.org/issues/6777#note-2

    If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

    @doktornotor:

    Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.

    So this looks like the SquidGuard package shouldn't be used at all which leads me to 2 questions:

    1. With 2.3 a lot of packages got cleaned up that weren't maintained. So if SquidGuard isn't maintained and broken why didn't it get removed as well?
    2. If SquidGuard is a no go and the other two solutions e2guarding and ufDBGuard are not available as packages how do we do content filtering on pfSense?

    While I understand all the issues mentioned just keen to understand what's the solution moving forward.

    Thanks!



  • @andyschmid:

    @doktornotor:

    @pfsensation:

    it seems squid is changing the domain from the domain typed in to "http"

    No, it's not.  See https://redmine.pfsense.org/issues/6777#note-2

    If Squidguard is blocking Google for you, that's unfortunate, you'll need to fix that issue instead. (No idea how, the package is broken and the code not maintainable.)

    @doktornotor:

    Fixing Squid (pretty much done as of 0.4.33) sadly will not do anything for the SquidGuard package. It's not just buggy, but that code is completely unreadable and something I definitely won't touch. Would hugely benefit from rewriting from scratch.

    So this looks like the SquidGuard package shouldn't be used at all which leads me to 2 questions:

    1. With 2.3 a lot of packages got cleaned up that weren't maintained. So if SquidGuard isn't maintained and broken why didn't it get removed as well?
    2. If SquidGuard is a no go and the other two solutions e2guarding and ufDBGuard are not available as packages how do we do content filtering on pfSense?

    While I understand all the issues mentioned just keen to understand what's the solution moving forward.

    Thanks!

    It's a big doddle!

    E2 Guardian is actually quite fantastic, I've been testing it in a virtual machine. It's way better than SquidGuard for sure, however it still has this issue of not working with chrome as it still uses Squid as a upstream proxy. I've found this website explaining a bit more : http://stackoverflow.com/questions/43665243/chrome-invalid-self-signed-ssl-cert-subject-alternative-name-missing

    However, anyone reading this. Remember that E2 Guardian is still unofficial and the GUI isn't perfect. But so far it seems to work half decently, and be able to phrase match - Something I always wanted. It's stupid to rely solely on URL filtering these days, you want your firewall to be able to scan a website and see if it should be blocked or allowed based on whats on the page. This also means that any new URL or proxy that may slip through the nets will also be blocked.



  • I used PFSENSE for almost two years. It is an AWESOME firewall and router, capable to support about 500 users concurrently whit just a CPU and a couple of good NICs. It's just sad to see one of their "killer apps", the https web filtering falling down defeated and without any hope. Is this the begining of the end for PFSENSE? I hope not, but unfortunaty I need to search another options. Push play baby!!! GOODBYYYYYEEEEE MOON MEN, GOODBYYYYYEEEE GOODBYYYYYEEEE…...... :'(



  • I had the same problem with the new 2.3.4 install of pfSense. I configured everything exactly how it should be configured but kept getting that https://http* error everyone seems to be getting. Looking through various websites I found nothing that was of any use. A lot of people even said that's the nature of ssl filtering with pfSense. I finally got it working. Literally, all I did was go into the common ACL tab and change the 'Default access [all]' setting from allow to deny. Save and apply the settings. Then go back in and change it back from deny to allow again. Save and apply the settings. Now it works like it should.


  • Banned

    See alternative to SquidGuard and Dansguardian / e2. GUI is own though and harder to install than desired. https://docs.diladele.com/tutorials/filtering_https_traffic_squid_pfsense/index.html