Squid 3.3.4 package for pfsense with ssl filtering

[quetzalcoatl]

Thank you Nachtfalke for the tip!
Actually thanks to this new "squid.conf" discover, from now on i will just copy and paste squid.conf content for every new installation instead of using the GUI.
So it's quicker to set squid up and not being afraid that i forgot some setting.

I wonder if there are other files than squid.conf that get changed when i setup squid from the GUI.
It says " Do not edit manually !" but it's so tempting and easy to edit manually. Can i go on and edit manually and save that file?
Hopefully someone can answer that.

Here i found squid.conf contents.
Seeing this config i hope you can tell me how to cache more stuff to get a decent hit ratio and avoid page updating issues:

squid.conf:

This file is automatically generated by pfSense

Do not edit manually !

http_port 10.0.0.10:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/pbi/squid-amd64/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none

logfile_rotate 0
shutdown_lifetime 3 seconds

Allow local network(s) on interface(s)

acl localnet src  10.0.0.0/24
httpd_suppress_version_string on
uri_whitespace strip

acl dynamic urlpath_regex cgi-bin ?
cache deny dynamic

cache_mem 6000 MB
maximum_object_size_in_memory 16384 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 50000 16 256
minimum_object_size 0 KB
maximum_object_size 900000 KB
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all

No redirector configured

#Remote proxies

Setup some default acls

From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.

acl localhost src 127.0.0.1/32

acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 4080 3128 3127 1025-65535
acl sslports port 443 563 4080

From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.

#acl manager proto cache_object

acl purge method PURGE
acl connect method CONNECT

Define protocols used for redirects

acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

Always allow localhost connections

From 3.2 further configuration cleanups have been done to make things easier and safer.

The manager, localhost, and to_localhost ACL definitions are now built-in.

http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

Reverse Proxy settings

Custom options

refresh_pattern -i .(3g2|3gp|asf|asx|avi|divx|flv|iff|ifo|m3u|m4a|m4v|mov|mpa|mpeg|mpe|qt|qtm|viv|mpg|ogg|rm|rmvb|scr|swf|vob|wmv|x-flv|xvid)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(aif|aiff|amr|cda|mid|wav|wma|midi|au|ram|ra|snd|mp2|mp3|mp4)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(3dm|ai|ani|art|bmp|cdr|cdt|cmf|cur|drw|dwg|dxf|eps|eps2|gif|icl|icm|ico|indd|jpeg|jpg|jpe|max|pct|pcx|png)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(ps|psd|psp|qxd|qxp|rels|svg|tga|thm|tif|tiff|wmf|wrl|xbm|xcf|xif|yuv|pnm|pbm|pgm|ppm|rgb|xpm|xwd|pic|pict)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(accdb|bfc|cbr|chm|csv|db|dbf|doc|docx|dot|hlp|kml|Kmz|lab|log|mdb|msg|odt|ost|pages|pdb|pdf|pps|txt|ppt|pptx|pst|pub|rtf|wpd|wps|wri|xlr|xls|xlsx|xlt)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(app|bat|cmd|com|exe|gadget|msi|pif|vb|wsf|torrent)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(8bi|bin|cat|cpl|dbx|dll|drv|gam|hex|hqx|lnk|nes|plugin|reg|rom|sav|sys|xll)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(arj|sit|zip|rar|rgz|psf|lzh|lha|cab|tar|tgz|gz|Z|wp|wp5|7z|pkg|rpm|sea|sitx|tar.gz|zipx|prn|srf|tex|latax|gpf|upd|jar|bz2|gzip|ace|kf|a[0-9][0-9]|r[0-9][0-9])$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(fnt|fon|otf|ttf)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(dmg|iso|toast|vcd)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(api|bas|c|cbl|class|cpp|cs|dtd|fla|java|m|pl|py|vbx)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(bak|bup|cdl|cfg|dat|deb|dss|dvf|efx|emf|eml|gho|gpx|ini|key|keychain|m4b|m4p|mcd|mim|mswmm|ori|prf|ptb|qbb|qbw|raw|sdf|ses|sql|ss|tmp|uue|uxx|vcf|xml|xsl|xtm)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(ht|htm|html|shtml|xhtml|css|js|jsp|asp|cer|cgi|csr|part|php|phtml|rss)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern ^gopher: 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern ^ftp: 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern . 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i (/cgi-bin/|?)$ 0 0% 0

Setup allowed acls

Allow local network(s) on interface(s)

http_access allow localnet

Default block all to be sure

http_access deny allsrc

[marcelloc]

@quetzalcoatl:

It says " Do not edit manually !" but it's so tempting and easy to edit manually. Can i go on and edit manually and save that file?
Hopefully someone can answer that.

You can do that if you do not install package via packages gui. remove squid package and add via console/ssh squid install via pkg_add.

[Nachtfalke]

As marcelloc said, if you do not use the package offered by the pfsense package manager you can do that.
But if you use squid with GUI then the squid.conf will be overwritten when pfsense reboot or if you click save on squid GUI or if squidguard or dansguardian restart squid.

Further you should have a look at this chapter when using refresh pattern:
http://doc.pfsense.org/index.php/Squid_Package_Tuning#Tweaking_Update_Caching_.2F_Squid_seems_to_download_on_its_own

Even if the titel is not exactly what you want you should inform about these options on the squid-cache.org wiki.

[quetzalcoatl]

I noticed that the semicolon ; at the end of the custom config is still used in that tutorial
but i saw somewhere that the ";" is not longer needed in the new pfsense/squid configs

Also i keep using the sign $ in custom config but the $ symbol does not appear in that tutorial…

Should i get rid of the $ symbol from all my custom lines?
Should i add the semicolon at the end of every custom config line?

[Nachtfalke]

The semicolon depends in what the GUI uses to separate different config lines.

In squid.conf it must be without semicolon.

The $ Defines that the regex must be on the end of the URL. You can use it but you do not need to.

[tester_02]

marcelloc.  I've been going through all the v3 threads for the last while and am trying to understand what the current status of the package is.

Seems like for 2.0.X we have to manually get some libs and then install dev package from front end.  Do we still have to manually fetch an update package from your personal repo?  Is there plans to move to a setup like squid where the updates are always published and we just reinstall the package from the gui?

For 2.1, we can do the same as above, but we just don't need to manually install the libs?

Maybe the dev's can sticky a thread and give marcelloc access to update post #1 in a thread and keep the latest status updated?  It's hard to follow the status of the package progress, unlike snort dev.

[marcelloc]

If you enable ipv6 on pfsense 2.0.3 or use 2.1, all you need is to install package from gui and missing libs from my repo.

Antivirus integration is still not working.

SSL interception is working fine.

cache issues can get better folowing this doc
http://doc.pfsense.org/index.php/Squid_Package_Tuning#Tweaking_Update_Caching_.2F_Squid_seems_to_download_on_its_own

[avp]

Hi, I had Squid and SquidGuard working for some time.  I recently noticed that it was no longer working, probably for months.  So I've uninstalled both packages.  Now I'm trying to use this Squid 3.3.4 pkg.  I appear to have Squid running correctly.  I've copied the libs from your repo.  I'm having a lot of trouble with SquidGuard.  I had it running, but it was blocking everything.  I had it running in the past, so I feel I am somewhat comfortable with the settings etc.  Now I can no longer get it running at all.  I get the msg 0/5 SG process started.  I know I haven't given much specifics here for your help, but I am wondering if there are any guidelines to try and get this pkg working with SG?

In the bigger picture, I'm looking for Squid + SG + AV functionality, I don't mind if I have to use different pkgs if that is the recommendation?

Also, it seems there are so many Squids, SGs in the pkg repo, and no foolproof instructions to get SG set up…

Any advice much appreciated. Thanks.

[marcelloc]

Use squid 3.3.5.

Since squid 3.x squidguard is started on demand, ifyou have no traffic, tthen no squid guard daemon will be running.

[avp]

Ok, I am using your 3.3.5.

I think I located some of my issue - Proxy server, Custom settings still had remnants from HAVP in there.  I think I have Squid + SG working correctly now.

Should it be possible to get HAVP working with this Squid + SG?  Or do you think you will get your AV integration working sometime soon?  Can I help with this?

Thanks very much…

[marcelloc]

@avp:

Should it be possible to get HAVP working with this Squid + SG?

Yes, havp is a proxy daemon, if you configure it as a parent for squid, it can work.

[ryan.low]

Good day, I had come across having problem with transparent proxy for both http and https to remote cache. It doesn't redirect to the other proxy server when I enabled both settings.

It works perfectly fine without enable ssl transparent proxy.

(opened a thread here. http://forum.pfsense.org/index.php/topic,64192.0.html)

thanks. :)

[Fehler20]

I'm not sure, if this is a bug or not:

if I enter wrong code into the custom options box and save those options squid stops working (this is ok :))
But if you remove the wrong options from the custom option field and hit save/restart squid manually, it still does not work. It seems, that if squid stops responding no changes are committed to the squid.conf file when you change some options in the webinterface. The system log still displays the wrong option line and says that there was no running copy found.

[workingman]

Hi marcelloc.

First thanks for all your work getting squid with ssl interception!  I currently have a hacked in squid 3.3.1 with it working on pfSense 2.1-BETA.

So I really would like to replace that as it prevents me from easily updating the system.

Trying the new 3.3.5 package on a virtual machine here I was able to get squid and squidguard installed and squid will run for me but refuses connections and netstat shows me:

tcp4      0      0 192.168.56.254.3128    .                    CLOSED

right IP:port but CLOSED..?  Let me know if you want to see cache.log or output from squid -NsXY or if you happen to know the fix?  ;)

[workingman]

Hi again.

I figured out how to get squid to start.  Disable pf :(

If I don't the squid cache.log stops at:

2013/07/24 13:29:21 kid1| WARNING: no_suid: setuid(0): (1) Operation not permitted
2013/07/24 13:29:21 kid1| sendto FD 25: (1) Operation not permitted
2013/07/24 13:29:21 kid1| ipcCreate: CHILD: hello write test failed

Once I run pfctl -d it starts up normally.

2013/07/24 13:32:34 kid1|  Completed Validation Procedure
2013/07/24 13:32:34 kid1|  Validated 325 Entries
2013/07/24 13:32:34 kid1|  store_swap_size = 5758.00 KB
2013/07/24 13:32:35 kid1| storeLateRelease: released 0 objects

As I mentioned this is running in a VM so that may be part of the problem but I have done similar setups in the past and did not have this issue.

[avp]

I had 3.3.5 working well with SG and HAVP.  i noticed the other day your pkg had been updated to 3.3.8.  i tried to upgrade to 3.3.8 by re-installing the pkg.  The re-install failed, and since then I can't get squid to work.  I've tried completely removing and re-installing the pkg, but no good.

here is the log:

Jul 25 14:51:37 squid[26589]: Squid Parent: will start 1 kids
Jul 25 14:51:37 squid[26589]: Squid Parent: (squid-1) process 26798 started
Jul 25 14:51:38 (squid-1): I don't handle this error well!
Jul 25 14:51:38 squid[26589]: Squid Parent: (squid-1) process 26798 exited with status 1
Jul 25 14:51:41 squid[26589]: Squid Parent: (squid-1) process 27792 started
Jul 25 14:51:43 (squid-1): I don't handle this error well!
Jul 25 14:51:43 squid[26589]: Squid Parent: (squid-1) process 27792 exited with status 1
Jul 25 14:51:46 squid[26589]: Squid Parent: (squid-1) process 32037 started
Jul 25 14:51:47 (squid-1): I don't handle this error well!
Jul 25 14:51:47 squid[26589]: Squid Parent: (squid-1) process 32037 exited with status 1
Jul 25 14:51:50 squid[26589]: Squid Parent: (squid-1) process 32672 started
Jul 25 14:51:51 Squid_Alarm[34792]: Squid has resumed. Reconfiguring filter.
Jul 25 14:51:51 (squid-1): I don't handle this error well!
Jul 25 14:51:51 squid[26589]: Squid Parent: (squid-1) process 32672 exited with status 1
Jul 25 14:51:51 check_reload_status: Reloading filter
Jul 25 14:51:54 squid[26589]: Squid Parent: (squid-1) process 35905 started
Jul 25 14:51:55 (squid-1): I don't handle this error well!
Jul 25 14:51:55 squid[26589]: Squid Parent: (squid-1) process 35905 exited with status 1
Jul 25 14:51:55 squid[26589]: Squid Parent: (squid-1) process 35905 will not be restarted due to repeated, frequent failures
Jul 25 14:51:55 squid[26589]: Exiting due to repeated, frequent failures
Jul 25 14:52:00 php: : SQUID is installed but not started. Not installing "nat" rules.
Jul 25 14:52:03 php: : SQUID is installed but not started. Not installing "pfearly" rules.

Any suggestions on how to proceed?

Thanks

[msi]

Hi, I have problems too with the "3.3.8" package on 2.1 amd64 see system.log:

Jul 25 22:10:20 <hostname>php: /status_services.php: The command '/usr/local/etc/rc.d/squid.sh stop' returned exit code '1', the output was '/libexec/ld-elf.so.1: Shared object "libheimntlm.so.10" not found, required by "squid"'

Seems the PBI is missing this library yet to launch?</hostname>

[workingman]

Grab the libs from the first post and copy to /usr/local/lib

squid should run.. I'm just having weird issues where it looks like pf is blocking my squid port.

[msi]

Thanks @workingman, the thread just got a bit long (aka TL;DR) ;-)

So squid >3.3 is yet quite of a moving target. Anyhow thanks to the packager(s) for all their time put into this fine proxy.

Update:

• Since I'm on 2.1 (I have due to H/W support) with PBIs  I put the libs under /usr/pbi/squid-amd64/lib

• Although the libs work, the build dates suggest they are from FreeBSD 8.1 (base of 2.0.x), I consider

getting those libs from a patched 8.3 for my 2.1

[stanthewizard]

Hello

Since 2.1 RC1
Latest Squid doesn't works anymore:

Aug 6 08:54:33 (squid-1): I don't handle this error well!
Aug 6 08:54:33 squid[64384]: Squid Parent: (squid-1) process 71825 exited with status 1
Aug 6 08:54:36 squid[64384]: Squid Parent: (squid-1) process 76944 started
Aug 6 08:54:38 (squid-1): I don't handle this error well!
Aug 6 08:54:38 squid[64384]: Squid Parent: (squid-1) process 76944 exited with status 1
Aug 6 08:54:38 squid[64384]: Squid Parent: (squid-1) process 76944 will not be restarted due to repeated, frequent failures
Aug 6 08:54:38 squid[64384]: Exiting due to repeated, frequent failures

Is there a turnaround ?