Squid3 - New GUI with sync, normal and reverse proxy

quetzalcoatl

With fine tuning i got squid working in a beautiful way. It caches a lot, i get a byte hit ration between 5 and 20%. Sometimes even up to 50% when i install lots of windows updates in many OEM computers. It is very fast, stable and reliable.
Thanks to marcelloc and all others working here.

But there is a big problem going on at least with my config.
After about 2 months of squid working really nice, suddently it stops working, and i cannot browse anymore.
I think that only https pages work but http stuff doesn't work any more. or the other way around…i don't remember exactly.

I tested many different settings in a hurry because it is a production environment but still nothing.

The only way to make pfsense allow users to browse the web properly is uninstalling completely the squid package and only then people are able to browse again.
The weird thing is that before that problem, squid works really nice for more than a month 24/7.
No settings change is being done before squid crashes.
This happened to me at lease 3 times so i had to live for a while without squid until i install a new pfsense snapshots and squid again (always hoping for a newer updated squid package).

I'm pretty sure that if i give you some logs it will help you understand where the problem is but i don't know where and how to look those logs to see what squid does just before it stops working.

Something makes me thing like some sort of log or cached stuff that becomes way too big after 1 or 2 months with my current config.
I just neet to know what file, dir, log, or whatever inflates so big after that 60 days.

Thanks for your great support!

podilarius

I had something like that happen as we'll. my question is to you remember that during the time it was not working, was there a lot of disk activity? Mine did like it was clearing out the cache. I had to stop squid and let traffic pass with out it and manually remove the cache directory. After I cleared it out, squid started working. I have since lowered the max cache size so cleanup happens more often and does not take as long. Is this something that can happen when the cache is large. I had it set to something like 100gb and it crashed around 50-60gb. I am guessing that if I let it go I am sure it would have cleared up and eventually started working.

quetzalcoatl

I didn't notice any disk activity, in fact it was completely quiet.

Mind that in a 500 GB hard drive i set squid to save into disk up to 350 GB of cached data.

Maybe that's too much and i should lower it to 50 Gigabytes.

Maybe there is a limit of how much cache squid is able to handle.

Also with RAM cache i noticed that if i set the maximum object size to 512kb or above it might crash.

Setting the maximum object size in RAM below 384 kb gives me no problems at all, but it takes days to fill half of the 20 GB of RAM that i have dedicated to Squid.

To take advantage of many gigabytes of RAM as cache, maximum object size should be at least 1024 KB but in some situations i had pfsense crashing over and over until i lowered that value below 512.

Maybe the original squid is able to handle more than than, but the ported freeBSD version is having a hard time handling big object sizes, too many objects, too much cached data into disk.

Hopefully someone can tell us the real squid limits inside pfsense, so we can stay within the limit for maximum performance and stability.

eldersouza

marcelloc,

My (simple) configuration is:

SQUID:

Proxy Interface: loopback
Proxy Port: 3128
Transparent HTTP Proxy: Enabled

Firewall: NAT

Proto: TCP
Src.Addr.: LAN (my LAN network address and mask)
Src.Ports: *
Dest.Addr: ! LAN (everything except my LAN
Dest.Ports: 80
NAT IP: 127.0.0.1
NAT Ports: 3128

Firewall: Rules

Rule created automatically

Squid.conf: This is what made the Proxy Server GUI (first lines)

This file is automatically generated by pfSense

Do not edit manually !
http_port 127.0.0.1:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on

And this is the error:

If I remove this line (http_port 127.0.0.1:3128) all works fine!

So, what I'm doing wrong?

Thanks

@marcelloc:

@eldersouza:

If I select LAN (or WAN) interface and check Transparent Mode… it puts in squid.conf:
http_port 10.0.0.1:3128
http_port 127.0.0.1:3128 intercept

This config is fine, users that connect via lan will not use intercept but transparent access will.

@eldersouza:

Can anyone confirm if it is really a bug? Or I'm doing something wrong in the configuration Squid.

Check your config again, transparent proxy works.

Nachtfalke

Hi,

I found that there is squid 3.2 available of FreeBSD ports. If I remember correct there were some bugs on 3.1 which were fixed on 3.2.
I am no expert - just found this here:
http://www.freebsd.org/cgi/cvsweb.cgi/ports/www/squid32/Makefile?rev=1.13

scotty562

Do I need to uninstall the squid package before installing squid3 or can I install squid3 with squid already installed?

marcelloc

Remove old squid package before installing squid3.

quetzalcoatl

I'm still waiting for squid 3.3 for freebsd.
we know that squid 3.2 is already available for freeBSD but it does not appear in the squid packages yet.
There is people that say that squid3.2 and squid3.3 are as efficient as the old 2.7.
According to them squid 3.1 and 3.0 are much slower than 3.3/3.2 and can manage less clients and connections.

I don'k know how true all that is but i did notice a bery big byte hit ratio comparing squid 3.1(the latest package for pfsense) with the lusca cache(based on squid 2.7)

I didn't want to use lusca cache anymore because i saw newer squid 3.x packages coming out for pfsense and the latest lusca was released on 2010.

lusca does not support IPv6 and it looks like it's not even updated anymore.

but what should i do? stick with the best performance with lusca or keep using squid 3.1 even if i know it's slower and anyways updates come too slowly.

I would like to be able to help marcelloc to port the latest squid 3.3 to pfsense but i don't know much about porting stuff, but i'm so desperate to get the latest squid that i'm seriously thinking about learning this stuff.

Your opinions are warmly welcome!

marcelloc

@quetzalcoatl:

I'm still waiting for squid 3.3 for freebsd.

It's almost done, http://forum.pfsense.org/index.php/topic,58368.msg312732.html#msg312732

quetzalcoatl

I found this stuff:

Squid-3.0 has been largely sponsored by the Web-Filtering user community. With features aimed at adapting and altering content in transit.
Squid-2.7 has been largely sponsored by high-performance user community. With features aimed at Caching extremely high traffic volumes in the order of Terabytes per day.
Source (http://wiki.squid-cache.org/Squid-2.7)

For what i can understand squid 2.7 should be able to have higher hit and byte hit ratio than squid 3.0/3.1

If squid 3.3 is out, to get the best hit ratio do you suggest me to upgrade to squid 3.3 or go to 2.7?
It would be great to see some benchmarks around.

can i upgrade or it is better to reinstall 3.3 from scratch? or even reinstall the whole latest pfsense snapshots with squid3.3dev?
what is the best way to do it to get the best results?

Also, I don't know if it's a squid or pfsense issue but i can never get more than 180 megabits of download speed from squid cache.
I have a gigabit network that is supposed to transfer stuff at 800/900 megabits without issues (in fact it does when i transfer files from computer to computer).
But when i download windows updates that are already cached into squid they download at 180 megabits/second. That is way waster than downloading directly from the Internet with no cache, but is there a setting that will allow me to use the full gigabit network speed, or at least half of it?

Is there a way i can donate workforce instead of money? since i don't have the money but i do have time and I'm willing to help this project.

quetzalcoatl

On pfsense2.1 snapshot from first week of maay 2013 I uninstalled squid 3.1 and installed squid 3.3.4

I noticed that the squid service was not starting anymore and there were 2 new services not starting as well.

the other 2 services are:
Icap inteface for squid and clamav integration
Clamav Antivirus

Also i noticed that after installing squid 3.3.4 my old config was still there…...handy but not sure if was a very clean installation.

so i installed in a new machine the latest 2.1 snapshot from today may 13th and installed the latest squid 3.3.4
and the very same issue is still there, squid service, c-icap and clamd not starting.

what should i do to make it work?

marcelloc

@quetzalcoatl:

what should i do to make it work?

Disable antivirus integration, it's causing crash on this first release as I've described on release post.

TheNetStriker

It would be very nice to be able to configure more than one http and one https port in reverse proxy settings. (I need different authentification types on my https sites) Would it be possible to replace the static http and https config section with a dynamic list, so multiple ports can be configured?

I did take a look at the package sources and think I could even add this config section myself. Would you add this to the official package if i submit the files here?

marcelloc

@TheNetStriker:

I did take a look at the package sources and think I could even add this config section myself. Would you add this to the official package if i submit the files here?

I'm doing changes on squid3-dev. I'll submit some changes today. take a look after this.

The best way to push patches is via github.

TheNetStriker

I've commited my changes on GitHub: https://github.com/TheNetStriker/pfsense-packages/commit/7d926f3d44cee817475c20bde44fe6471bab4ba7 (I hope it worked, it was my first commit on GitHub)

I've added a new page to configure additional ports and added a configuration for client certificates.

marcelloc

@TheNetStriker:

I've added a new page to configure additional ports and added a configuration for client certificates.

I'm doing it on web servers page but I'll take a look on your code and see how can I merge it with my current uncommitted code.

Thanks for your code contribution. :)

technical

squid access.log rotate problems

http://forum.pfsense.org/index.php/topic,59557.msg341017.html#msg341017

cron jobs not work
/bin/rm /var/squid/cache/swap.state; /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf

returns with error.

check it please marchelloc

/var/squid/logs/access.log
and my cache not works to

my squid .conf


# This file is automatically generated by pfSense
# Do not edit manually !
http_port 10.50.2.1:3949
http_port 10.50.3.1:3949
http_port 127.0.0.1:3949 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language tr
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 1
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  10.50.2.0/24 10.50.3.0/24
forwarded_for off
via off
httpd_suppress_version_string on
uri_whitespace strip

# Break HTTP standard for flash videos. Keep them in cache even if asked not to.
refresh_pattern -i \.flv$ 10080 90% 999999 ignore-no-cache override-expire ignore-private

# Let the clients favorite video site through with full caching
acl youtube dstdomain .youtube.com
cache allow youtube

# Windows Update refresh_pattern
range_offset_limit -1
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i my.windowsupdate.website.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
cache_mem 512 MB
maximum_object_size_in_memory 128 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir ufs /var/squid/cache 140000 16 256
minimum_object_size 0 KB
maximum_object_size 4500000 KB
offline_mode offcache_swap_low 90
cache_swap_high 95

# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
refresh_pattern .    0  20%  4320
# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 443 3128 1025-65535 
acl sslports port 443 563 443 
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

deny_info TCP_RESET allsrc

# Package Integration
redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass off
url_rewrite_children 5

# Custom options

# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

file is now 539 mb and become larger more than …

marcelloc

@TechnicaL:

/bin/rm /var/squid/cache/swap.state; /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf

the squid rotate cmd is the same from squid2

what errors do you get running it on console?

If you use sarg, you can disable squid rotate logs and enable it on sarg after report creation.

technical

@marcelloc:

the squid rotate cmd is the same from squid2

what errors do you get running it on console?

If you use sarg, you can disable squid rotate logs and enable it on sarg after report creation.

i cant using sarg.

command is working but access.log file not rotating i delete today manuel that bigger than 600 mb.

access.log file rotate at 00:00 dially is important for us.

i ll check tomorrow morning.

technical

Update :13-06-2013 09:00 PM access.log rotate is not working, still including yesterday log events.

1371044860.589 1479 10.50.2.115 TCP_MISS/200 39264 GET http://www.youtube.com/ - DIRECT/173.194.39.199 text/html

time stamp = GMT: Wed, 12 Jun 2013 13:47:40 GMT