SQUID3 HTTPS fixed but WAN PPPOE problem with SQUID3
-
Marcelloc told me to use the DNS V4 resolve first checkbox in the squid proxy configuration page and it worked just fine! Thanks Marcellok!
Now there are not HTTPS issues any more.Also i just found out that if i set the ignore-must-revalidate option it will make a mess with facebook https account login.
Removing the ignore-must-revalidate option in squid3 fixes that issue.So i'm glad i found and fixed a bug in my custom squid config.
But sadly i'm still having issues if squid3 is installed with pppoe wan.
That is why i have to keep 2 pfsense Virtual Machines working: one to connect via PPPOE and another one behind with squid3 to cache as much as it's technically possible.
i'm still testing squid3 to make sure it works with PPPOE but since the only testing platform i have is actually a production platform, i just have few hours or less in the early morning to test that and if i'm alone and lucky i may take months to figure out what is making squid3 not to work if PPPOE WAN connection is needed.
With squid 2.7 and Lusca Cache i don't recall having PPPOE connection issues.
But with squid3, it's PPPOE working without squid or squid installed but no PPPOE connection ever established.Below i pasted the most aggressive and efficient squid custom configuration that i could build until now.
Hopefully someone can help me improve it and fix bugs or wrong lines.
refresh_pattern -i .$ 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http:// 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://- 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://-.com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://-.net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://. 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.-* 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.-.com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.-.net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..* 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..- 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..-.com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..-.net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://... 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://...-* 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://...-.com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://...-.net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://....* 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://....- 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://....com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://....net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://...com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://...net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..co.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..in.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://..org 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.co.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.gg.in.th 99999 999999% 99999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.in.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://.org 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www.....com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www.....net 99999 999999% 99999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www....com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www....net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www...co.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www...com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www...in.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www...net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www...org 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www..co.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www..com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www..in.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www..net 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^http://www..org 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^https://.com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^https://.in.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^https://www..com 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i ^https://www.*.in.th 99999 999999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(3g2|3gp|asf|asx|avi|divx|flv|iff|ifo|m3u|m4a|m4v|mov|mpa|mpeg|mpe|qt|qtm|viv|mpg|ogg|rm|rmvb|scr|swf|vob|wmv|x-flv|xvid)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims override-lastmod store-stale;
refresh_pattern -i .(aif|aiff|amr|cda|mid|wav|wma|midi|au|ram|ra|snd|mp2|mp3|mp4)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(3dm|ai|ani|art|bmp|cdr|cdt|cmf|cur|drw|dwg|dxf|eps|eps2|gif|icl|icm|ico|indd|jpeg|jpg|jpe|max|pct|pcx|png)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(ps|psd|psp|qxd|qxp|rels|svg|tga|thm|tif|tiff|wmf|wrl|xbm|xcf|xif|yuv|pnm|pbm|pgm|ppm|rgb|xpm|xwd|pic|pict)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(accdb|bfc|cbr|chm|csv|db|dbf|doc|docx|dot|hlp|kml|Kmz|lab|log|mdb|msg|odt|ost|pages|pdb|pdf|pps|txt|ppt|pptx|pst|pub|rtf|wpd|wps|wri|xlr|xls|xlsx|xlt)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(app|bat|cmd|com|exe|gadget|msi|pif|vb|wsf|torrent)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(8bi|bin|cat|cpl|dbx|dll|drv|gam|hex|hqx|lnk|nes|plugin|reg|rom|sav|sys|xll)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(arj|sit|zip|rar|rgz|psf|lzh|lha|cab|tar|tgz|gz|Z|wp|wp5|7z|pkg|rpm|sea|sitx|tar.gz|zipx|prn|srf|tex|latax|gpf|upd|jar|bz2|gzip|ace|kf|a[0-9][0-9]|r[0-9][0-9])$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(fnt|fon|otf|ttf)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(dmg|iso|toast|vcd)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(api|bas|c|cbl|class|cpp|cs|dtd|fla|java|m|pl|py|vbx)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(bak|bup|cdl|cfg|dat|deb|dss|dvf|efx|emf|eml|gho|gpx|ini|key|keychain|m4b|m4p|mcd|mim|mswmm|ori|prf|ptb|qbb|qbw|raw|sdf|ses|sql|ss|tmp|uue|uxx|vcf|xml|xsl|xtm)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i .(ht|htm|html|shtml|xhtml|css|js|jsp|asp|cer|cgi|csr|part|php|phtml|rss)$ 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern ^gopher: 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern ^ftp: 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern . 99999 99999% 9999999 override-expire ignore-no-cache ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims override-lastmod store-stale;
refresh_pattern -i (/cgi-bin/|?)$ 0 0% 0;
tcp_outgoing_address 127.0.0.1;
max_filedescriptors 65536;
quick_abort_min 0 KB;
quick_abort_max 0 KB;
quick_abort_pct 0;
ie_refresh off;
client_db off;
range_offset_limit 0;
reload_into_ims on;
retry_on_error on;
via off;
refresh_all_ims on;
half_closed_clients off;
vary_ignore_expire on;
strip_query_terms on;
server_persistent_connections on;
ipcache_size 16384;
fqdncache_size 16384;
log_fqdn off;
positive_dns_ttl 999 hours;
negative_dns_ttl 999 hours;
negative_ttl 999 hours;
dns_v4_first on;
pipeline_prefetch on; -
Some of your refresh pattern look really bad. The cover everything. So there things that will be cached but which should never be cached and must be cached. That's why you had problems with facebook.com.
You should use the refresh pattern only for files and URL you would really like to cache.
For example for windows updates or something like that but not for all sites in general. -
I use squid3 with pfsense 2.0.2 and pppoe wan without issues, what problems do you have(except for those pointed by Nachtfalke)?
-
It may be a setting i have.
I always install the latest 2.0.2 pfsense or the latest snapshot in a VM
Then i set it up to connect via PPPOE. It does connect (sometimes very quickly, sometimes i have to restart pfsense until it gets connected, or wait several minutes).
But after all, it always establishes the PPPOE connection properly getting an IP, gateways and DNS, even if i then setup static openDNS to block porn crap….Then, since i have internet, i download the squid3 package and install it, but as soon as i restart pfsense to have all changes to take effect, the PPPOE connection is not established anymore or it gets established but squid doesn't allow LAN to get any data.
If i recall properly, only disabling the transparent proxy was allowing users on LAN to browse the internet.
Even if i don't set custom options at all squid stops working as soon as i switch the lan from static IP or DHCP to a PPPOE connection.
If i uninstall the squid3 package, PPPOE wan starts working fine and clients on LAN can browse the web.
It's like squid is not getting the wan data if wan is PPPOE.
Maybe i need to setup an extra option that allows squid to be actually in the middle and properly connected to both WAN and LAN
-
you do not need to restart pfsense to all changes to take effect.
Did you tried to restart the service after you get a new wan ip?
Are you listening squid on wan?
-
Make you squid only listening on LAN interface. Not on any other.
Check "Allow users on interfaces" and check "Transparent proxy".
check "use DNS v4 first" and do NOT enable "cache dynamic content"Do not use any other custom options. Squid will run and start with the default settings very well.
I am sure your problem is not PPPoE but something wrong on squid config.
Perhaps you can post screenshots of your squid GUI configs.
Further - if you run squid you can rund squidguard or dansguardian and use predifines blacklists for porn. Then you do not need to do that with openDNS.
You can also try to use some other DNS like 8.8.8.8 or 8.8.4.4 instead of OpenDNS - just for testing.
