Squid3 - New GUI with sync, normal and reverse proxy

dneuhaeuser

I think I found a small bug leading to this message in system log:
"Not calling package sync code for dependency squidreverse of squid3 because some include files are missing."

This is my patch suggestion:

–- /usr/local/pkg/squid_reverse.xml 2013-02-17 16:15:30.000000000 +0100
+++ /usr/local/pkg/squid_reverse.xml 2013-02-17 16:15:49.000000000 +0100
@@ -50,3 +50,3 @@
<title>Proxy server: Reverse Proxy</title>

<include_file>squid.inc</include_file>

<include_file>/usr/local/pkg/squid.inc</include_file>
<tabs>–Dennis</tabs>

fragged

@fragged:

I want to not cache Steam downloads. Shouldn't this work in "Custom settings" on the General-tab:
acl steamtest url_regex -i ^http://.*/depot/.*/chunk/.*
cache deny steamtest
Squid is still caching Steam downloads (url http://81.171.70.221/depot/107202/chunk/52f888f7848537d725be13719ef9870f0b1da910). My regexp should match the url each time. The IP changes, but /depot/ and /chunk/ are always there.

What am I doing wrong? :F

Regards,
Joona

Can anyone help me with this? The custom settings pops up into /usr/pbi/squid-amd64/etc/squid/squid.conf, but content that falls under the url_regex is still being cached. My cache keeps getting filled by one time Steam downloads.

marcelloc

Fragged, did you enabled cache dynamic content?
Can you check on squid conf for more cache options defined?

fragged

@marcelloc:

Fragged, did you enabled cache dynamic content?
Can you check on squid conf for more cache options defined?

Dynamic content is off. Here's my full config.


[2.1-BETA1][admin@pfsense.localdomain]/root(10): cat /usr/pbi/squid-amd64/etc/squid/squid.conf
# This file is automatically generated by pfSense
# Do not edit manually !
http_port 192.168.11.1:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first off
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/pbi/squid-amd64/etc/squid/icons
visible_hostname localhost
cache_mgr fragged@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 30
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  192.168.11.0/24
httpd_suppress_version_string on
uri_whitespace strip

acl dynamic urlpath_regex cgi-bin \?
cache deny dynamic
cache_mem 4096 MB
maximum_object_size_in_memory 524288 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap GDSF
cache_dir ufs /var/squid/cache 4096 8 256
minimum_object_size 0 KB
maximum_object_size 1024 KB
offline_mode offcache_swap_low 90
cache_swap_high 95

# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 1025-65535
acl sslports port 443 563
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

http_access allow manager localhost

# Allow external cache managers
acl ext_manager src 127.0.0.1
acl ext_manager src 192.168.11.1
acl ext_manager src
http_access allow manager ext_manager

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

# Custom options
acl steamtest url_regex -i http://.*/depot/.*/chunk/.*
cache deny steamtest

# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

jaredadams

I see you guys trying to disable the cache of steam downloads. But what about for LAN parties that would absolutely LOVE to do this? How well does it work? When ONE person on the LAN downloads a game from steam will everyone else then get it from the squid cache?

marcelloc

@jaredadams:

When ONE person on the LAN downloads a game from steam will everyone else then get it from the squid cache?

If you have storage to this and a good cache config, then yes :)

Mozart

I'm trying to create a reverse proxy for a Remote Desktop Gateway server, also a RPC over HTTP connection.
With all the different posts that are in this thread I was able to get the login prompt but now I stuck because it keeps giving me the prompt after entering the correct credentials.
I've read a post on a different forum from someone with a similar problem: http://www.mentby.com/Group/squid-users/ntlm-and-persistent-connections-reverse-proxy-3120-solved-patch.html

Has some been able to publish a RDGW with Squid3 already?
Here is my squid.conf.


# This file is automatically generated by pfSense
# Do not edit manually !
http_port 172.30.244.10:3128
icp_port 7
dns_v4_first off
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/local/etc/squid/icons
visible_hostname proxy
cache_mgr admin@domain.intra
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 5
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  172.30.244.0/28
uri_whitespace strip
dns_nameservers 172.30.255.11 172.30.255.21 
acl dynamic urlpath_regex cgi-bin \?
cache deny dynamic
cache_mem 8 MB
maximum_object_size_in_memory 32 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA

minimum_object_size 0 KB
maximum_object_size 10 KB
offline_mode off
# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901  3128 1025-65535 
acl sslports port 443 563  2607
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

acl allowed_subnets src 172.30.252.0/22
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings
http_port 172.30.248.100:80 accel defaultsite=sv002.domain.intra vhost
https_port 172.30.248.100:443 accel cert=/usr/local/etc/squid/51399e473ace0.crt key=/usr/local/etc/squid/51399e473ace0.key defaultsite=rdgw.domain.intra vhost
#rdgw001
cache_peer 172.30.255.16 parent 443 0 proxy-only no-query no-digest originserver login=PASS ssl sslflags=DONT_VERIFY_PEER front-end-https=auto name=rvp_rdgw001

acl rvm_CAServers url_regex -i http://sv002.domain.intra/*
acl rvm_CAServers url_regex -i https://sv002.domain.intra/*
acl rvm_RDGWServers url_regex -i ^https://rdgw.domain.intra/.*$
cache_peer_access rvp_rdgw001 allow rvm_RDGWServers
cache_peer_access rvp_rdgw001 deny allsrc
never_direct allow rvm_CAServers
never_direct allow rvm_RDGWServers
http_access allow rvm_CAServers
http_access allow rvm_RDGWServers

deny_info TCP_RESET allsrc

# Custom options

# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow allowed_subnets
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

eldersouza

I'm trying to enable transparent mode, but after check Transparent HTTP Proxy (in LAN or Loopback - with NAT/rules ok) my client win7 receives an error message in all sites that I tried to open (print screen below).

When I opened squid.conf, I saw that SQUID GUI had put two lines http_port. One with "intercept" and another without this parameter. Then I removed the line without intercept and the squid worked fine.

And another crazy thing that I saw in this Squid3 GUI…

If I select loopback interface and check Transparent Mode... it puts in squid.conf:
http_port 127.0.0.1:3128
http_port 127.0.0.1:3128 intercept
If I select LAN (or WAN) interface and check Transparent Mode... it puts in squid.conf:
http_port 10.0.0.1:3128
http_port 127.0.0.1:3128 intercept

My configuration is:

pfSense: 2.0.2-Release (i386) - (Clean install and configuration. I didn't update the firmware and didn't import configuration from my old pfSense)
Squid: 3.1.20 pkg 2.0.6

Can anyone confirm if it is really a bug? Or I'm doing something wrong in the configuration Squid.

Thanks

Elder Souza

marcelloc

@eldersouza:

If I select LAN (or WAN) interface and check Transparent Mode… it puts in squid.conf:
http_port 10.0.0.1:3128
http_port 127.0.0.1:3128 intercept

This config is fine, users that connect via lan will not use intercept but transparent access will.

@eldersouza:

Can anyone confirm if it is really a bug? Or I'm doing something wrong in the configuration Squid.

Check your config again, transparent proxy works.

quetzalcoatl

With fine tuning i got squid working in a beautiful way. It caches a lot, i get a byte hit ration between 5 and 20%. Sometimes even up to 50% when i install lots of windows updates in many OEM computers. It is very fast, stable and reliable.
Thanks to marcelloc and all others working here.

But there is a big problem going on at least with my config.
After about 2 months of squid working really nice, suddently it stops working, and i cannot browse anymore.
I think that only https pages work but http stuff doesn't work any more. or the other way around…i don't remember exactly.

I tested many different settings in a hurry because it is a production environment but still nothing.

The only way to make pfsense allow users to browse the web properly is uninstalling completely the squid package and only then people are able to browse again.
The weird thing is that before that problem, squid works really nice for more than a month 24/7.
No settings change is being done before squid crashes.
This happened to me at lease 3 times so i had to live for a while without squid until i install a new pfsense snapshots and squid again (always hoping for a newer updated squid package).

I'm pretty sure that if i give you some logs it will help you understand where the problem is but i don't know where and how to look those logs to see what squid does just before it stops working.

Something makes me thing like some sort of log or cached stuff that becomes way too big after 1 or 2 months with my current config.
I just neet to know what file, dir, log, or whatever inflates so big after that 60 days.

Thanks for your great support!

podilarius

I had something like that happen as we'll. my question is to you remember that during the time it was not working, was there a lot of disk activity? Mine did like it was clearing out the cache. I had to stop squid and let traffic pass with out it and manually remove the cache directory. After I cleared it out, squid started working. I have since lowered the max cache size so cleanup happens more often and does not take as long. Is this something that can happen when the cache is large. I had it set to something like 100gb and it crashed around 50-60gb. I am guessing that if I let it go I am sure it would have cleared up and eventually started working.

quetzalcoatl

I didn't notice any disk activity, in fact it was completely quiet.

Mind that in a 500 GB hard drive i set squid to save into disk up to 350 GB of cached data.

Maybe that's too much and i should lower it to 50 Gigabytes.

Maybe there is a limit of how much cache squid is able to handle.

Also with RAM cache i noticed that if i set the maximum object size to 512kb or above it might crash.

Setting the maximum object size in RAM below 384 kb gives me no problems at all, but it takes days to fill half of the 20 GB of RAM that i have dedicated to Squid.

To take advantage of many gigabytes of RAM as cache, maximum object size should be at least 1024 KB but in some situations i had pfsense crashing over and over until i lowered that value below 512.

Maybe the original squid is able to handle more than than, but the ported freeBSD version is having a hard time handling big object sizes, too many objects, too much cached data into disk.

Hopefully someone can tell us the real squid limits inside pfsense, so we can stay within the limit for maximum performance and stability.

eldersouza

marcelloc,

My (simple) configuration is:

SQUID:

Proxy Interface: loopback
Proxy Port: 3128
Transparent HTTP Proxy: Enabled

Firewall: NAT

Proto: TCP
Src.Addr.: LAN (my LAN network address and mask)
Src.Ports: *
Dest.Addr: ! LAN (everything except my LAN
Dest.Ports: 80
NAT IP: 127.0.0.1
NAT Ports: 3128

Firewall: Rules

Rule created automatically

Squid.conf: This is what made the Proxy Server GUI (first lines)

This file is automatically generated by pfSense

Do not edit manually !
http_port 127.0.0.1:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on

And this is the error:

If I remove this line (http_port 127.0.0.1:3128) all works fine!

So, what I'm doing wrong?

Thanks

@marcelloc:

@eldersouza:

If I select LAN (or WAN) interface and check Transparent Mode… it puts in squid.conf:
http_port 10.0.0.1:3128
http_port 127.0.0.1:3128 intercept

This config is fine, users that connect via lan will not use intercept but transparent access will.

@eldersouza:

Can anyone confirm if it is really a bug? Or I'm doing something wrong in the configuration Squid.

Check your config again, transparent proxy works.

Nachtfalke

Hi,

I found that there is squid 3.2 available of FreeBSD ports. If I remember correct there were some bugs on 3.1 which were fixed on 3.2.
I am no expert - just found this here:
http://www.freebsd.org/cgi/cvsweb.cgi/ports/www/squid32/Makefile?rev=1.13

scotty562

Do I need to uninstall the squid package before installing squid3 or can I install squid3 with squid already installed?

marcelloc

Remove old squid package before installing squid3.

quetzalcoatl

I'm still waiting for squid 3.3 for freebsd.
we know that squid 3.2 is already available for freeBSD but it does not appear in the squid packages yet.
There is people that say that squid3.2 and squid3.3 are as efficient as the old 2.7.
According to them squid 3.1 and 3.0 are much slower than 3.3/3.2 and can manage less clients and connections.

I don'k know how true all that is but i did notice a bery big byte hit ratio comparing squid 3.1(the latest package for pfsense) with the lusca cache(based on squid 2.7)

I didn't want to use lusca cache anymore because i saw newer squid 3.x packages coming out for pfsense and the latest lusca was released on 2010.

lusca does not support IPv6 and it looks like it's not even updated anymore.

but what should i do? stick with the best performance with lusca or keep using squid 3.1 even if i know it's slower and anyways updates come too slowly.

I would like to be able to help marcelloc to port the latest squid 3.3 to pfsense but i don't know much about porting stuff, but i'm so desperate to get the latest squid that i'm seriously thinking about learning this stuff.

Your opinions are warmly welcome!

marcelloc

@quetzalcoatl:

I'm still waiting for squid 3.3 for freebsd.

It's almost done, http://forum.pfsense.org/index.php/topic,58368.msg312732.html#msg312732

quetzalcoatl

I found this stuff:

Squid-3.0 has been largely sponsored by the Web-Filtering user community. With features aimed at adapting and altering content in transit.
Squid-2.7 has been largely sponsored by high-performance user community. With features aimed at Caching extremely high traffic volumes in the order of Terabytes per day.
Source (http://wiki.squid-cache.org/Squid-2.7)

For what i can understand squid 2.7 should be able to have higher hit and byte hit ratio than squid 3.0/3.1

If squid 3.3 is out, to get the best hit ratio do you suggest me to upgrade to squid 3.3 or go to 2.7?
It would be great to see some benchmarks around.

can i upgrade or it is better to reinstall 3.3 from scratch? or even reinstall the whole latest pfsense snapshots with squid3.3dev?
what is the best way to do it to get the best results?

Also, I don't know if it's a squid or pfsense issue but i can never get more than 180 megabits of download speed from squid cache.
I have a gigabit network that is supposed to transfer stuff at 800/900 megabits without issues (in fact it does when i transfer files from computer to computer).
But when i download windows updates that are already cached into squid they download at 180 megabits/second. That is way waster than downloading directly from the Internet with no cache, but is there a setting that will allow me to use the full gigabit network speed, or at least half of it?

Is there a way i can donate workforce instead of money? since i don't have the money but i do have time and I'm willing to help this project.

quetzalcoatl

On pfsense2.1 snapshot from first week of maay 2013 I uninstalled squid 3.1 and installed squid 3.3.4

I noticed that the squid service was not starting anymore and there were 2 new services not starting as well.

the other 2 services are:
Icap inteface for squid and clamav integration
Clamav Antivirus

Also i noticed that after installing squid 3.3.4 my old config was still there…...handy but not sure if was a very clean installation.

so i installed in a new machine the latest 2.1 snapshot from today may 13th and installed the latest squid 3.3.4
and the very same issue is still there, squid service, c-icap and clamd not starting.

what should i do to make it work?