Squid3 - New GUI with sync, normal and reverse proxy

fragged · Feb 1, 2013, 4:10 PM

I want to not cache Steam downloads. Shouldn't this work in "Custom settings" on the General-tab:


acl steamtest url_regex -i ^http://.*/depot/.*/chunk/.*
cache deny steamtest

Squid is still caching Steam downloads (url http://81.171.70.221/depot/107202/chunk/52f888f7848537d725be13719ef9870f0b1da910). My regexp should match the url each time. The IP changes, but /depot/ and /chunk/ are always there.

What am I doing wrong? :F

Regards,
Joona

Tikimotel · Feb 3, 2013, 1:01 PM

Will we see an update of the squid3 package to the 3.2 branch?

Stable v3.2.7 released on the 1st of Feb.
http://www.squid-cache.org/Versions/v3/3.2/RELEASENOTES.html

Some bugs regarding ipv6 have been fixed in this latest version.
http://www.squid-cache.org/Versions/v3/3.2/changesets/SQUID_3_2_7.html

Still no freshports.org update though, that is still at v3.2.6

p.s. the package name is 3.1.20 and 3.1.22 is rolled out in PBI. (I'm running 2.1 beta)

marcelloc · Feb 3, 2013, 10:55 PM

I'm testing 3.2.6 and SSL filtering this weekend. :)

Tikimotel · Feb 6, 2013, 6:27 PM

I've checked just now, freshports.org has finished porting v3.2.7 ;)

marcelloc · Feb 6, 2013, 7:54 PM

@Tikimotel:

I've checked just now, freshports.org has finished porting v3.2.7 ;)

Compiling and testing

big changelog

Fri 2013-02-01 03:54:21 -0700 Amos Jeffries +23 -6 3.2.7
Thu 2013-01-31 21:58:59 -0700 Amos Jeffries +4 -0 Fix ipv6 enabled pinger.
Thu 2013-01-31 21:57:13 -0700 Amos Jeffries +11 -8 Bug #3687: unhandled exception: c when using interception and peers
Thu 2013-01-31 21:56:07 -0700 Alex Rousskov +2 -0 Bug #3111: Mid-term fix for the forward.cc "err" assertion.
Thu 2013-01-31 21:54:23 -0700 Sebastien Wenske +5 -0 Support OpenSSL NO_Compression option
Tue 2013-01-29 00:00:55 -0700 Amos Jeffries +2 -2 Fix WCCPv2 'comparison between signed and unsigned integer expressions'
Mon 2013-01-28 04:49:40 -0700 Amos Jeffries +1 -1 Bug #3567: Memory leak handling malformed requests
Mon 2013-01-28 04:48:06 -0700 Amos Jeffries +9 -0 Bug #3735: raw-IPv6 domain URLs crash if IPv6-disabled
Mon 2013-01-28 04:45:07 -0700 Amos Jeffries +2 -3 Bug #3732: Fix ConnOpener IPv6 awareness
Mon 2013-01-28 04:41:04 -0700 Amos Jeffries +6 -2 Initialize mem_node fully
Mon 2013-01-28 04:40:12 -0700 Tianyin Xu +6 -0 Bug #3736: Floating point exception due to divide by zero
Mon 2013-01-28 04:29:16 -0700 Alex Rousskov +5 -0 Fix "address.GetPort() != 0" assertion for helpers on FreeBSD (at least).
Mon 2013-01-28 04:25:34 -0700 Amos Jeffries +25 -18 WCCP: Fix memory leak in mask assignment, improve debuggsing.
Mon 2013-01-28 04:23:27 -0700 Amos Jeffries +32 -32 Polish quick_abort feature decision code
Mon 2013-01-28 04:22:25 -0700 Amos Jeffries +1 -0 Fix memory leak in IP address unit test
Mon 2013-01-28 02:59:18 -0700 Amos Jeffries +12 -2 Fix memory leaks in ICMP
Mon 2013-01-28 02:58:16 -0700 Tianyin Xu +1 -1 Bug #3729: 32-bit overflow in parsing 64-bit configuration values
Sun 2013-01-27 22:43:11 -0700 Tomas Hozza +36 -10 Fix various Disk I/O issues in all modules
Sun 2013-01-27 22:40:06 -0700 Francesco Chemolli +2 -2 Fix error in config parser which would mis-assign the sslcrlfile directive.
Sun 2013-01-27 22:39:19 -0700 Francesco Chemolli +54 -26 Plugged memory leaks in digest authentication module
Sun 2013-01-27 21:33:00 -0700 Tianyin Xu +4 -0 Bug #3728: Improve debug for cache_dir
Sun 2013-01-27 21:30:26 -0700 Tomas Hozza +3 -0 Make sure copied strings are properly terminated in snmplib and wccp2
Sun 2013-01-27 21:28:17 -0700 Tomas Hozza +32 -10 Fix various issues in snmplib
Sun 2013-01-27 21:27:32 -0700 Tomas Hozza +31 -11 Fix various issues in smblib
Fri 2013-01-25 02:59:54 -0700 Timo Teras +7 -2 Bug #3678: external acl grace period causes acl lookup failures

geijt · Feb 10, 2013, 10:02 AM

I've modified the squid-reverse package to support redirects.
This was something I missed after migrating from Microsoft TMG 2010 to pfsense.

You can redirect requests from mail.mydomain.com (protocol HTTP selected) to https://mail.mydomain.com
Or in case of Microsoft Exchange redirecting from mail.mydomain.com and webmail.mydomain.com (both protocols selected) to https://webmail.mydomain.com/owa is also supported.
You also can use it to support the Microsoft Exchange Autodiscover HTTP to HTTPS redirect function.

I think you can expect the updated package soon.

redflag237 · Feb 13, 2013, 8:51 AM

@Truster:

1st: You have to correct the typo as descripted in my post before…

I configured it like this:
reverse Proxy Interface: WAN
External FQDN webmail.domain.tld
Reset TCP connections if request is unauthorized: active
Enable HTTPS reverse proxy: Active
reverse HTTPS port: 443
reverse SSL certificate
Ignore internal Certificate validation: active (we use self signed certs)
OWA frontend IP address: IP Address four your Frontent OWA Server
Enable ActiveSync: yes
Enable Outlook Anywhere: yes
Enable Exchange WebServices: yes
Enable AutoDiscover:yes

and of course, you have to open that port in the firewall rules.

i'm using OWA/Exch2010...

hope, this helps.

Edit: if you correct the typo after enabling RPC-over-HTTPS you have to uncheck that box, save the configuration, recheck the box and save it again to apply the changes.

I configured it like this:
reverse Proxy Interface: WAN
External FQDN secret.no-ip.org
Reset TCP connections if request is unauthorized: active
Enable HTTP reverse proxy: Active
reverse HTTP port: 8080

Web Servers:
Enable this peer: active
Peer Alias: mynas
Peer Ip: 10.178.1.61
Peer Port: 8080
Peer Protocol: HTTP

Mappings:
Enable this URI: enabled
Group Name: NAS-MyNAS
Peers: mynas (selected)
URIs: *

Result: TCP Connection is being reset.
–---

When disabling the 'Reset if unauthorised' button, i'm getting an access denied Message of Squid.

Can anyone help me, please?

geijt · Feb 13, 2013, 5:56 PM

Redflag237,

Did you tried entering some actual URIs instead of * on you mapping?

The URIs field expects a regex, just * isn't a valid regex.
If you really want to accept everything enter ^.*$ as URI, I didn't checked or squid accept this but it's a valid regex. ;-)

dneuhaeuser · Feb 17, 2013, 3:27 PM

I think I found a small bug leading to this message in system log:
"Not calling package sync code for dependency squidreverse of squid3 because some include files are missing."

This is my patch suggestion:

–- /usr/local/pkg/squid_reverse.xml 2013-02-17 16:15:30.000000000 +0100
+++ /usr/local/pkg/squid_reverse.xml 2013-02-17 16:15:49.000000000 +0100
@@ -50,3 +50,3 @@
<title>Proxy server: Reverse Proxy</title>

<include_file>squid.inc</include_file>

<include_file>/usr/local/pkg/squid.inc</include_file>
<tabs>–Dennis</tabs>

fragged · Feb 18, 2013, 11:37 AM

@fragged:

I want to not cache Steam downloads. Shouldn't this work in "Custom settings" on the General-tab:
acl steamtest url_regex -i ^http://.*/depot/.*/chunk/.*
cache deny steamtest
Squid is still caching Steam downloads (url http://81.171.70.221/depot/107202/chunk/52f888f7848537d725be13719ef9870f0b1da910). My regexp should match the url each time. The IP changes, but /depot/ and /chunk/ are always there.

What am I doing wrong? :F

Regards,
Joona

Can anyone help me with this? The custom settings pops up into /usr/pbi/squid-amd64/etc/squid/squid.conf, but content that falls under the url_regex is still being cached. My cache keeps getting filled by one time Steam downloads.

marcelloc · Feb 22, 2013, 2:58 AM

Fragged, did you enabled cache dynamic content?
Can you check on squid conf for more cache options defined?

fragged · Feb 22, 2013, 6:09 AM

@marcelloc:

Fragged, did you enabled cache dynamic content?
Can you check on squid conf for more cache options defined?

Dynamic content is off. Here's my full config.


[2.1-BETA1][admin@pfsense.localdomain]/root(10): cat /usr/pbi/squid-amd64/etc/squid/squid.conf
# This file is automatically generated by pfSense
# Do not edit manually !
http_port 192.168.11.1:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first off
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/pbi/squid-amd64/etc/squid/icons
visible_hostname localhost
cache_mgr fragged@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 30
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  192.168.11.0/24
httpd_suppress_version_string on
uri_whitespace strip

acl dynamic urlpath_regex cgi-bin \?
cache deny dynamic
cache_mem 4096 MB
maximum_object_size_in_memory 524288 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap GDSF
cache_dir ufs /var/squid/cache 4096 8 256
minimum_object_size 0 KB
maximum_object_size 1024 KB
offline_mode offcache_swap_low 90
cache_swap_high 95

# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 1025-65535
acl sslports port 443 563
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

http_access allow manager localhost

# Allow external cache managers
acl ext_manager src 127.0.0.1
acl ext_manager src 192.168.11.1
acl ext_manager src
http_access allow manager ext_manager

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

# Custom options
acl steamtest url_regex -i http://.*/depot/.*/chunk/.*
cache deny steamtest

# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

jaredadams · Feb 25, 2013, 9:21 PM

I see you guys trying to disable the cache of steam downloads. But what about for LAN parties that would absolutely LOVE to do this? How well does it work? When ONE person on the LAN downloads a game from steam will everyone else then get it from the squid cache?

marcelloc · Feb 25, 2013, 9:37 PM

@jaredadams:

When ONE person on the LAN downloads a game from steam will everyone else then get it from the squid cache?

If you have storage to this and a good cache config, then yes :)

Mozart · Mar 11, 2013, 7:18 AM

I'm trying to create a reverse proxy for a Remote Desktop Gateway server, also a RPC over HTTP connection.
With all the different posts that are in this thread I was able to get the login prompt but now I stuck because it keeps giving me the prompt after entering the correct credentials.
I've read a post on a different forum from someone with a similar problem: http://www.mentby.com/Group/squid-users/ntlm-and-persistent-connections-reverse-proxy-3120-solved-patch.html

Has some been able to publish a RDGW with Squid3 already?
Here is my squid.conf.


# This file is automatically generated by pfSense
# Do not edit manually !
http_port 172.30.244.10:3128
icp_port 7
dns_v4_first off
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname proxy
cache_mgr admin@domain.intra
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 5
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  172.30.244.0/28
uri_whitespace strip
dns_nameservers 172.30.255.11 172.30.255.21 
acl dynamic urlpath_regex cgi-bin \?
cache deny dynamic
cache_mem 8 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA

minimum_object_size 0 KB
maximum_object_size 10 KB
offline_mode off
# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 1025-65535 
acl sslports port 443 563  2607
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

acl allowed_subnets src 172.30.252.0/22
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings
http_port 172.30.248.100:80 accel defaultsite=sv002.domain.intra vhost
https_port 172.30.248.100:443 accel cert=/usr/local/etc/squid/51399e473ace0.crt key=/usr/local/etc/squid/51399e473ace0.key defaultsite=rdgw.domain.intra vhost
#rdgw001
cache_peer 172.30.255.16 parent 443 0 proxy-only no-query no-digest originserver login=PASS ssl sslflags=DONT_VERIFY_PEER front-end-https=auto name=rvp_rdgw001

acl rvm_CAServers url_regex -i http://sv002.domain.intra/*
acl rvm_CAServers url_regex -i https://sv002.domain.intra/*
acl rvm_RDGWServers url_regex -i ^https://rdgw.domain.intra/.*$
cache_peer_access rvp_rdgw001 allow rvm_RDGWServers
cache_peer_access rvp_rdgw001 deny allsrc
never_direct allow rvm_CAServers
never_direct allow rvm_RDGWServers
http_access allow rvm_CAServers
http_access allow rvm_RDGWServers

deny_info TCP_RESET allsrc

# Custom options

# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow allowed_subnets
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

eldersouza · Mar 12, 2013, 3:55 PM

I'm trying to enable transparent mode, but after check Transparent HTTP Proxy (in LAN or Loopback - with NAT/rules ok) my client win7 receives an error message in all sites that I tried to open (print screen below).

When I opened squid.conf, I saw that SQUID GUI had put two lines http_port. One with "intercept" and another without this parameter. Then I removed the line without intercept and the squid worked fine.

And another crazy thing that I saw in this Squid3 GUI…

If I select loopback interface and check Transparent Mode... it puts in squid.conf:
http_port 127.0.0.1:3128
http_port 127.0.0.1:3128 intercept
If I select LAN (or WAN) interface and check Transparent Mode... it puts in squid.conf:
http_port 10.0.0.1:3128
http_port 127.0.0.1:3128 intercept

My configuration is:

pfSense: 2.0.2-Release (i386) - (Clean install and configuration. I didn't update the firmware and didn't import configuration from my old pfSense)
Squid: 3.1.20 pkg 2.0.6

Can anyone confirm if it is really a bug? Or I'm doing something wrong in the configuration Squid.

Thanks

Elder Souza

marcelloc · Mar 12, 2013, 9:53 PM

@eldersouza:

If I select LAN (or WAN) interface and check Transparent Mode… it puts in squid.conf:
http_port 10.0.0.1:3128
http_port 127.0.0.1:3128 intercept

This config is fine, users that connect via lan will not use intercept but transparent access will.

@eldersouza:

Can anyone confirm if it is really a bug? Or I'm doing something wrong in the configuration Squid.

Check your config again, transparent proxy works.

quetzalcoatl · Apr 9, 2013, 10:46 PM

With fine tuning i got squid working in a beautiful way. It caches a lot, i get a byte hit ration between 5 and 20%. Sometimes even up to 50% when i install lots of windows updates in many OEM computers. It is very fast, stable and reliable.
Thanks to marcelloc and all others working here.

But there is a big problem going on at least with my config.
After about 2 months of squid working really nice, suddently it stops working, and i cannot browse anymore.
I think that only https pages work but http stuff doesn't work any more. or the other way around…i don't remember exactly.

I tested many different settings in a hurry because it is a production environment but still nothing.

The only way to make pfsense allow users to browse the web properly is uninstalling completely the squid package and only then people are able to browse again.
The weird thing is that before that problem, squid works really nice for more than a month 24/7.
No settings change is being done before squid crashes.
This happened to me at lease 3 times so i had to live for a while without squid until i install a new pfsense snapshots and squid again (always hoping for a newer updated squid package).

I'm pretty sure that if i give you some logs it will help you understand where the problem is but i don't know where and how to look those logs to see what squid does just before it stops working.

Something makes me thing like some sort of log or cached stuff that becomes way too big after 1 or 2 months with my current config.
I just neet to know what file, dir, log, or whatever inflates so big after that 60 days.

Thanks for your great support!

podilarius · Apr 10, 2013, 2:24 AM

I had something like that happen as we'll. my question is to you remember that during the time it was not working, was there a lot of disk activity? Mine did like it was clearing out the cache. I had to stop squid and let traffic pass with out it and manually remove the cache directory. After I cleared it out, squid started working. I have since lowered the max cache size so cleanup happens more often and does not take as long. Is this something that can happen when the cache is large. I had it set to something like 100gb and it crashed around 50-60gb. I am guessing that if I let it go I am sure it would have cleared up and eventually started working.

quetzalcoatl · Apr 10, 2013, 3:55 AM

I didn't notice any disk activity, in fact it was completely quiet.

Mind that in a 500 GB hard drive i set squid to save into disk up to 350 GB of cached data.

Maybe that's too much and i should lower it to 50 Gigabytes.

Maybe there is a limit of how much cache squid is able to handle.

Also with RAM cache i noticed that if i set the maximum object size to 512kb or above it might crash.

Setting the maximum object size in RAM below 384 kb gives me no problems at all, but it takes days to fill half of the 20 GB of RAM that i have dedicated to Squid.

To take advantage of many gigabytes of RAM as cache, maximum object size should be at least 1024 KB but in some situations i had pfsense crashing over and over until i lowered that value below 512.

Maybe the original squid is able to handle more than than, but the ported freeBSD version is having a hard time handling big object sizes, too many objects, too much cached data into disk.

Hopefully someone can tell us the real squid limits inside pfsense, so we can stay within the limit for maximum performance and stability.