Squid3 - New GUI with sync, normal and reverse proxy

marcelloc · May 13, 2013, 4:41 PM

I'm still waiting for squid 3.3 for freebsd.

It's almost done, http://forum.pfsense.org/index.php/topic,58368.msg312732.html#msg312732

quetzalcoatl · May 13, 2013, 5:18 PM

I found this stuff:

Squid-3.0 has been largely sponsored by the Web-Filtering user community. With features aimed at adapting and altering content in transit.
Squid-2.7 has been largely sponsored by high-performance user community. With features aimed at Caching extremely high traffic volumes in the order of Terabytes per day.
Source (http://wiki.squid-cache.org/Squid-2.7)

For what i can understand squid 2.7 should be able to have higher hit and byte hit ratio than squid 3.0/3.1

If squid 3.3 is out, to get the best hit ratio do you suggest me to upgrade to squid 3.3 or go to 2.7?
It would be great to see some benchmarks around.

can i upgrade or it is better to reinstall 3.3 from scratch? or even reinstall the whole latest pfsense snapshots with squid3.3dev?
what is the best way to do it to get the best results?

Also, I don't know if it's a squid or pfsense issue but i can never get more than 180 megabits of download speed from squid cache.
I have a gigabit network that is supposed to transfer stuff at 800/900 megabits without issues (in fact it does when i transfer files from computer to computer).
But when i download windows updates that are already cached into squid they download at 180 megabits/second. That is way waster than downloading directly from the Internet with no cache, but is there a setting that will allow me to use the full gigabit network speed, or at least half of it?

Is there a way i can donate workforce instead of money? since i don't have the money but i do have time and I'm willing to help this project.

quetzalcoatl · May 13, 2013, 8:06 PM

On pfsense2.1 snapshot from first week of maay 2013 I uninstalled squid 3.1 and installed squid 3.3.4

I noticed that the squid service was not starting anymore and there were 2 new services not starting as well.

the other 2 services are:
Icap inteface for squid and clamav integration
Clamav Antivirus

Also i noticed that after installing squid 3.3.4 my old config was still there…...handy but not sure if was a very clean installation.

so i installed in a new machine the latest 2.1 snapshot from today may 13th and installed the latest squid 3.3.4
and the very same issue is still there, squid service, c-icap and clamd not starting.

what should i do to make it work?

marcelloc · May 13, 2013, 8:19 PM

@quetzalcoatl:

what should i do to make it work?

Disable antivirus integration, it's causing crash on this first release as I've described on release post.

TheNetStriker · Jun 7, 2013, 3:10 PM

It would be very nice to be able to configure more than one http and one https port in reverse proxy settings. (I need different authentification types on my https sites) Would it be possible to replace the static http and https config section with a dynamic list, so multiple ports can be configured?

I did take a look at the package sources and think I could even add this config section myself. Would you add this to the official package if i submit the files here?

marcelloc · Jun 7, 2013, 5:42 PM

@TheNetStriker:

I did take a look at the package sources and think I could even add this config section myself. Would you add this to the official package if i submit the files here?

I'm doing changes on squid3-dev. I'll submit some changes today. take a look after this.

The best way to push patches is via github.

TheNetStriker · Jun 10, 2013, 9:22 PM

I've commited my changes on GitHub: https://github.com/TheNetStriker/pfsense-packages/commit/7d926f3d44cee817475c20bde44fe6471bab4ba7 (I hope it worked, it was my first commit on GitHub)

I've added a new page to configure additional ports and added a configuration for client certificates.

marcelloc · Jun 10, 2013, 10:29 PM

@TheNetStriker:

I've added a new page to configure additional ports and added a configuration for client certificates.

I'm doing it on web servers page but I'll take a look on your code and see how can I merge it with my current uncommitted code.

Thanks for your code contribution. :)

technical · Jun 12, 2013, 1:14 PM

squid access.log rotate problems

http://forum.pfsense.org/index.php/topic,59557.msg341017.html#msg341017

cron jobs not work
/bin/rm /var/squid/cache/swap.state; /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf

returns with error.

check it please marchelloc

/var/squid/logs/access.log
and my cache not works to

my squid .conf


# This file is automatically generated by pfSense
# Do not edit manually !
http_port 10.50.2.1:3949
http_port 10.50.3.1:3949
http_port 127.0.0.1:3949 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language tr
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 1
shutdown_lifetime 3 seconds
# Allow local network(s) on interface(s)
acl localnet src  10.50.2.0/24 10.50.3.0/24
forwarded_for off
via off
httpd_suppress_version_string on
uri_whitespace strip

# Break HTTP standard for flash videos. Keep them in cache even if asked not to.
refresh_pattern -i \.flv$ 10080 90% 999999 ignore-no-cache override-expire ignore-private

# Let the clients favorite video site through with full caching
acl youtube dstdomain .youtube.com
cache allow youtube

# Windows Update refresh_pattern
range_offset_limit -1
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i my.windowsupdate.website.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
cache_mem 512 MB
maximum_object_size_in_memory 128 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir ufs /var/squid/cache 140000 16 256
minimum_object_size 0 KB
maximum_object_size 4500000 KB
offline_mode offcache_swap_low 90
cache_swap_high 95

# Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp:    1440  20%  10080
refresh_pattern ^gopher:  1440  0%  1440
refresh_pattern -i (/cgi-bin/|\?) 0  0%  0
refresh_pattern .    0  20%  4320
# No redirector configured

#Remote proxies

# Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 443 3128 1025-65535 
acl sslports port 443 563 443 
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT

# Define protocols used for redirects
acl HTTP proto HTTP
acl HTTPS proto HTTPS

http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

# Always allow localhost connections
http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

# Reverse Proxy settings

deny_info TCP_RESET allsrc

# Package Integration
redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass off
url_rewrite_children 5

# Custom options

# Setup allowed acls
# Allow local network(s) on interface(s)
http_access allow localnet
# Default block all to be sure
http_access deny allsrc

file is now 539 mb and become larger more than …

marcelloc · Jun 12, 2013, 1:52 PM

@TechnicaL:

/bin/rm /var/squid/cache/swap.state; /usr/local/sbin/squid -k rotate -f /usr/local/etc/squid/squid.conf

the squid rotate cmd is the same from squid2

what errors do you get running it on console?

If you use sarg, you can disable squid rotate logs and enable it on sarg after report creation.

technical · Jun 12, 2013, 3:14 PM

@marcelloc:

the squid rotate cmd is the same from squid2

what errors do you get running it on console?

If you use sarg, you can disable squid rotate logs and enable it on sarg after report creation.

i cant using sarg.

command is working but access.log file not rotating i delete today manuel that bigger than 600 mb.

access.log file rotate at 00:00 dially is important for us.

i ll check tomorrow morning.

technical · Jun 13, 2013, 6:04 AM

Update :13-06-2013 09:00 PM access.log rotate is not working, still including yesterday log events.

1371044860.589 1479 10.50.2.115 TCP_MISS/200 39264 GET http://www.youtube.com/ - DIRECT/173.194.39.199 text/html

time stamp = GMT: Wed, 12 Jun 2013 13:47:40 GMT

marcelloc · Jun 13, 2013, 2:23 PM

If the script is on cron and if you can successful run it by hand, It's not related to squid.

Are you on nanobsd?

technical · Jun 13, 2013, 3:03 PM

@marcelloc:

If the script is on cron and if you can successful run it by hand, It's not related to squid.

Are you on nanobsd?

no not nanobsd, system running at P4 PC

how to create script without delete the access.log file
if im create rm -rf command log file will be deleted. how to clean the only inside access.log file.

marcelloc · Jun 13, 2013, 6:58 PM

@TechnicaL:

how to create script without delete the access.log file
if im create rm -rf command log file will be deleted. how to clean the only inside access.log file.

squid -k rotate -f /usr/local/etc/squid/squid.conf does it.

technical · Jun 14, 2013, 5:08 AM

@marcelloc:

@TechnicaL:

how to create script without delete the access.log file
if im create rm -rf command log file will be deleted. how to clean the only inside access.log file.

squid -k rotate -f /usr/local/etc/squid/squid.conf does it.

oke i ll try it and inform you.

arch113 · Jun 14, 2013, 9:00 PM

Does any version of Squidguard work with this -dev version of squid? I cant seem to get the Service to START.

marcelloc · Jun 14, 2013, 9:08 PM

@arch113:

Does any version of Squidguard work with this -dev version of squid? I cant seem to get the Service to START.

Since squid 3, it starts on demand. If you have access then squidguard will run.

technical · Jun 15, 2013, 5:50 AM

@marcelloc:

squid -k rotate -f /usr/local/etc/squid/squid.conf does it.

marchelloc it does not work. still access.log file include 12.06.2013 events.
installed packages

squid3 3.1.20 pkg 2.0.6
squidguard 1.4_4 pkg v.1.9.5
Lightsquid 1.8.2 pkg v.2.32

Tikimotel · Jun 15, 2013, 8:22 AM

@arch113:

Does any version of Squidguard work with this -dev version of squid? I cant seem to get the Service to START.

If you reinstalled the packages, sometimes squid is already started without "kids" or "children" active.
Somehow when squid is running it won't take the squidguard process as a child-process.
When that happens, just stop squid via the dashboard or services page and restart using the save button in the proxy server page.

I use squid3-dev with squidGuard-squid3.
With squid3 and onwards I always manually edit the "squidguard_configurator.inc" to use squid3 options for child-processes.


# ------------------------------------------------------------------------------
# squid config options
# ------------------------------------------------------------------------------
define('REDIRECTOR_OPTIONS_REM',   '# squidGuard options');
define('REDIRECTOR_PROGRAM_OPT',   'url_rewrite_program');
define('REDIRECT_BYPASS_OPT',      'url_rewrite_bypass');
define('REDIRECT_CHILDREN_OPT',    'url_rewrite_children');
define('REDIRECTOR_PROCESS_COUNT', '12 startup=8 idle=6 concurrency=0'); # redirector processes count will start with 8 do a max of 12 and idles at 6 (with v3.3+)