Hi vonfrank,
i'm using pfsense packages haproxy and package acme on my pfsense for this similar situation.

pfsense WebUI rconfigured to listen on Port 8443.
pfsense WebUI disabled for redirect from Port 80.

I've two frontends defined in haproxy:

WAN:80 (for acme redirection and ssl cert sign)

WAN:443 (for ssloffloading with all certificates for my websites)

one acl for each certificate. configured as "Host matches:"-Expression. value is the CN of ssl-Certificate.

one action for each acl to switch to every backend.

SSL Certificates configured as "Additional certificates", in configuration area "SSL Offloading"

checkbox "Add ACL for certificate CommonName. (host header matches the "CN" of the certificate)" checked

backends in haproxy:

one Backend for acme challenge. (I'm using the pfsense internal webserver for that)

several other Backends. one for every site. some acl's for URL rewriting

here my Backend config for acme to direct to pfsense's WebUI (listening on Port 8443):

backend BE_ACME-Server_http_ipvANY mode http log global errorfile 403 /var/etc/haproxy/errorfile_BE_ACME-Server_http_ipvANY_403_http_503 timeout connect 30000 timeout server 30000 retries 3 acl acme_not_in_path path_beg -i /.well-known/acme-challenge http-request deny  if  !acme_not_in_path server pfsenseAdminWebpage 127.0.0.1:8443 ssl check inter 1000  verify none

I've never fiddled with those specific option so I can't really give you any guidance.

https://wiki.squid-cache.org/Features/DelayPools

https://www.howtoforge.com/squid-delay-pools-bandwidth-management

For ClamAV in Squid: Services > Squid Proxy Server > Antivirus > Redirect url: <your web="" interface="" url="">/squid_clwarn.php
For example: https://pfsense.localdomain:8080/squid_clwarn.php

I'm now trying to do the same in squidGuard:
Services > SquidGuard Proxy Filter > Common ACL > Redirect mode: ext url err page (enter URL)
Services > SquidGuard Proxy Filter > Common ACL > Redirect info: https://pfsense.localdomain:8080/sgerror.php

But that throws me 'SSL_ERROR_RX_RECORD_TOO_LONG' errors.</your>

For XMLRPC to work, all nodes must be on identical versions. You cannot sync between mismatched versions.

For some reason, I remember Windows Server 2012 DNS being a pain when it comes to wpad entries…

If I am not mistaken, due to the potential for abuse of WPAD on a network, Windows Server makes it difficult to add these records as they are in the Global Query Block List.

The following may be of some assistance...

https://technet.microsoft.com/en-us/library/cc995261.aspx
https://technet.microsoft.com/en-us/library/cc995158.aspx
https://technet.microsoft.com/en-us/library/cc995062.aspx?f=255&MSPPError=-2147217396

Squid in transparent mode fails for me as well - in general, not in Azure. I am running 2.4.1-RELEASE (amd64). I also use squidGuard.

What I observed was that requests (downloads from LAN which should get caught by squid) bypass squid and get downloaded via WAN. They do no appear in the Real Time monitor (because no traffic at all shows up in the Real Time monitor). SquidGuard blocking does also not work.

I tried to disable squid (Unchecked "Check to enable the Squid proxy.", hit Save) and re-enable it, but that didn't solve it.

I tried to disable transparent mode (while keeping squid enabled, so I unckecked "Enable transparent mode to forward all requests for destination port 80 to the proxy server.", hit Save), then re-abled it (checked "Enable transparent mode to forward all requests for destination port 80 to the proxy server.", then hit Save), and, yes, suddenly transparent mode worked!

Until the next reboot at least.

I then re-disabled transparent mode, and re-enabled it again, and it was up again (content from the disk cache was getting served from the cache in transparent mode right away).

Obviously, this workaround sucks a bit. I wonder if this could be automated…or fixed. :D

Note that disabling transparent mode will clear the data "Bypass Proxy for These Source IPs" and "Bypass Proxy for These Destination IPs"! if you have anything in there, be sure to put in into the clipboard first. Or else you have to pull it from your config history. The XML tags are <defined_ip_proxy_off>and <defined_ip_proxy_off_dest>.

Edit: it seems that doing changes in the WebGUI also causes transparent mode to stop working. I can not see whether a certain change did trigger this, as several changes occured around the time when squid stopped working in transparent mode. In any case, disbaling and re-enabling transparent mode did the trick to (temporarily) fix transparent mode.

Another edit: it looks like having entries in "Bypass Proxy for These Source IPs" and "Bypass Proxy for These Destination IPs" breaks transparent mode. As the disable/re-enable cycle clears these fields, transparent mode works again…until I re-enter the required IP ranges back into these fields. Then transparent mode is disabled again.

I then tried to delete the content in "Bypass Proxy for These Source IPs" and "Bypass Proxy for These Destination IPs" (without disabling/re-enabling transparent mode) and, yes, transparent mode suddenly started working again. Oh well.</defined_ip_proxy_off_dest></defined_ip_proxy_off>

Google AMP - https://www.ampproject.org/ ? maybe

The reverse proxy is entirely separate, no need to enable that.

Steve

I deleted the squid package since it was caching less than 5% of traffic anyways. But just in case your curious I revisited a bunch of the sites from the log that were throwing the error and it never came up once.

Sure. HTTPS filtering ON, Bump, splice whitelist. The Diadele solution may work, but it's not free. That solution probably rely on the Layer 7, since it promotes "Content Filter". After every new "virus spread" like today, the Bad Rabbit, I almost faint just by remembering that SOME content can't be blocked…

I don't bother caching dynamic content, youtube, microsoft updates, etc.. seemed pointless.

Most of the web is dynamic these days.  I stopped caching two years ao when I realized that my hit rate was in the area of 3-7%, and it wasn't worth the hassle.

Thats fine, I really appreciate all the help :)

If I go back to the flash based version it seems to work without any issues. Going over to other testing sites and they all fail. doing a google search for "Speed Test" lets you run the google speed tester (it fails to ever start)

http://speedtest.xfinity.com/

https://www.speakeasy.net/speedtest/

https://www.verizon.com/speedtest/

http://speedtest.att.com/speedtest/ -Worked (we all know AT&T is horrid though, they probably lie about their speed tests to make U-Verse customers feel better)
https://fast.com/ -Worked (no clue)

In all my previous attempts I never went down the google result page and tried THIS many test sites

I appreciate all your help, thank you so much!

Not quite.  That config will allow you to get the domain but not the full URL or content.  You can use explicit with WPAD to get the domain, or transparent with Splice All.  Full URL or contents requires cert on every client, which is a major hassle.

A new version of the ACME package will be available later today which should correct this.

It is working now, the problem was the following:

"Your user does not have access to "WebCfg - Services: HAProxy package" so it does not have sufficient privileges to control the haproxy process."

https://redmine.pfsense.org/issues/7987

under normal circumstances I would say yes but because it is resolving a DNS entry that resolves to 1 IP address and gets routed based on some rules I cannot have a "split-dns" situation with pfsense. It would be nice to have pfsense give back two different ip addresses to 1 dns entry depending on the subnet but that isnt the case lol.