Squid 3.3.4 package for pfsense with ssl filtering

[stanthewizard]

At last It works

Thanks

BTW
with squid 3.3.5 there is a bug in 2.1 RC0 … endless loop updating package

[marcelloc]

@stanthewizard:

At last It works

Thanks

BTW
with squid 3.3.5 there is a bug in 2.1 RC0 … endless loop updating package

During firmware upgrade?

[stanthewizard]

At reboot

Warning about package updating …. endlessly

[quetzalcoatl]

I always use the latest snapshot, install new 64 bit snapshot every week during the night when no client of mine browses the internet.

I also install squid 3.3.5 package or whatever is the latest.

I never install squidguard or any other package. The absolutely only package i have installed is squid 3.3.x

Following marcello instructions i got some caching, but it's a ridiculous 1% byte hit ratio. Most of the windows updates are not cache anymore.(maybe they are stored into cache but not server, or not even stored at all).

Also there are some pages that get updated daily and squid does not detect the update.

www.filehippo.com  and  www.jalopnik.com  are 2 of the pages that i noticed that squid does not detect that they have been updated…..
Sometimes even a week later squid is showing me the week old page, and both filehippo and jalopnik update daily.

The only way to see those updated pages is turning the squid service off.

There is an option with a checkbox that says "Enable offline mode" and i keep that off otherwise squid will never check tons of page updates.
Enable offline mode is off in my squid.

I always do have some custom refresh options in the "custom options" field.....but even if i leave that field empty, save and restart the whole pfsense machine, i never get a decent hit ratio or byte hit ratio.

I don't even ask for youtube caching because i know it's hard and complicated, but at least some caching like before with ms windows updates, ms office updates, and exe files caching.

I used to be able to download the winrar, winamp, VLC player and more files at 180 mb/s using older squids (when it was already downloaded once).

Now with 3.3.x i never get the cached file, it always starts downloading again at 24kb/s........

Am I the only one with this issue? what am i doing wrong?

[marcelloc]

@quetzalcoatl:

Am I the only one with this issue? what am i doing wrong?

Did you disabled dynamic content? This behavior/issue happens when its enabled.

[Nachtfalke]

Hi,

are the "known bugs" on the very first post of this thread still persistent?

[marcelloc]

@Nachtfalke:

Hi,

Are the "known bugs" on the very first post of this thread still persistent?

Only antivirus integration, missing libs and enable ipv6 on 2.0.x.

I can't  edit the first topic after one or two days, Maybe only moderators.

[Deadringers]

Hi can anyone help with my issue detailed here:http://forum.pfsense.org/index.php/topic,63618.0.html

Seems that when I try to use either proxy or transparent proxy it just doesn't pass any traffic? :(

[quetzalcoatl]

I am keeping dynamic content caching always disabled because if I enable it i don't even get 1% byte hit ratio, it just goes to 0% hit ratio.

Maybe i can post here the full squid config so you can see it and tell me where the issue is.

I just don't know where and what squid config file i need to post.

Since you told me to disable dynamic content caching i do get some hits but is ridiculous. It caches almost nothing and when it does, it will not update the page if changed serving always an offline like page.

[Nachtfalke]

Squid.conf is the important file.

Fürther check the pfsense docs for squid caching options like Windows updates. It should hell you to cache other things.

[quetzalcoatl]

Thank you Nachtfalke for the tip!
Actually thanks to this new "squid.conf" discover, from now on i will just copy and paste squid.conf content for every new installation instead of using the GUI.
So it's quicker to set squid up and not being afraid that i forgot some setting.

I wonder if there are other files than squid.conf that get changed when i setup squid from the GUI.
It says " Do not edit manually !" but it's so tempting and easy to edit manually. Can i go on and edit manually and save that file?
Hopefully someone can answer that.

Here i found squid.conf contents.
Seeing this config i hope you can tell me how to cache more stuff to get a decent hit ratio and avoid page updating issues:

squid.conf:

This file is automatically generated by pfSense

Do not edit manually !

http_port 10.0.0.10:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/pbi/squid-amd64/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none

logfile_rotate 0
shutdown_lifetime 3 seconds

Allow local network(s) on interface(s)

acl localnet src  10.0.0.0/24
httpd_suppress_version_string on
uri_whitespace strip

acl dynamic urlpath_regex cgi-bin ?
cache deny dynamic

cache_mem 6000 MB
maximum_object_size_in_memory 16384 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 50000 16 256
minimum_object_size 0 KB
maximum_object_size 900000 KB
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all

No redirector configured

#Remote proxies

Setup some default acls

From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.

acl localhost src 127.0.0.1/32

acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 4080 3128 3127 1025-65535
acl sslports port 443 563 4080

From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.

#acl manager proto cache_object

acl purge method PURGE
acl connect method CONNECT

Define protocols used for redirects

acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

Always allow localhost connections

From 3.2 further configuration cleanups have been done to make things easier and safer.

The manager, localhost, and to_localhost ACL definitions are now built-in.

http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

Reverse Proxy settings

Custom options

refresh_pattern -i .(3g2|3gp|asf|asx|avi|divx|flv|iff|ifo|m3u|m4a|m4v|mov|mpa|mpeg|mpe|qt|qtm|viv|mpg|ogg|rm|rmvb|scr|swf|vob|wmv|x-flv|xvid)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(aif|aiff|amr|cda|mid|wav|wma|midi|au|ram|ra|snd|mp2|mp3|mp4)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(3dm|ai|ani|art|bmp|cdr|cdt|cmf|cur|drw|dwg|dxf|eps|eps2|gif|icl|icm|ico|indd|jpeg|jpg|jpe|max|pct|pcx|png)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(ps|psd|psp|qxd|qxp|rels|svg|tga|thm|tif|tiff|wmf|wrl|xbm|xcf|xif|yuv|pnm|pbm|pgm|ppm|rgb|xpm|xwd|pic|pict)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(accdb|bfc|cbr|chm|csv|db|dbf|doc|docx|dot|hlp|kml|Kmz|lab|log|mdb|msg|odt|ost|pages|pdb|pdf|pps|txt|ppt|pptx|pst|pub|rtf|wpd|wps|wri|xlr|xls|xlsx|xlt)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(app|bat|cmd|com|exe|gadget|msi|pif|vb|wsf|torrent)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(8bi|bin|cat|cpl|dbx|dll|drv|gam|hex|hqx|lnk|nes|plugin|reg|rom|sav|sys|xll)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(arj|sit|zip|rar|rgz|psf|lzh|lha|cab|tar|tgz|gz|Z|wp|wp5|7z|pkg|rpm|sea|sitx|tar.gz|zipx|prn|srf|tex|latax|gpf|upd|jar|bz2|gzip|ace|kf|a[0-9][0-9]|r[0-9][0-9])$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(fnt|fon|otf|ttf)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(dmg|iso|toast|vcd)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(api|bas|c|cbl|class|cpp|cs|dtd|fla|java|m|pl|py|vbx)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(bak|bup|cdl|cfg|dat|deb|dss|dvf|efx|emf|eml|gho|gpx|ini|key|keychain|m4b|m4p|mcd|mim|mswmm|ori|prf|ptb|qbb|qbw|raw|sdf|ses|sql|ss|tmp|uue|uxx|vcf|xml|xsl|xtm)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(ht|htm|html|shtml|xhtml|css|js|jsp|asp|cer|cgi|csr|part|php|phtml|rss)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern ^gopher: 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern ^ftp: 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern . 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i (/cgi-bin/|?)$ 0 0% 0

Setup allowed acls

Allow local network(s) on interface(s)

http_access allow localnet

Default block all to be sure

http_access deny allsrc

[marcelloc]

@quetzalcoatl:

It says " Do not edit manually !" but it's so tempting and easy to edit manually. Can i go on and edit manually and save that file?
Hopefully someone can answer that.

You can do that if you do not install package via packages gui. remove squid package and add via console/ssh squid install via pkg_add.

[Nachtfalke]

As marcelloc said, if you do not use the package offered by the pfsense package manager you can do that.
But if you use squid with GUI then the squid.conf will be overwritten when pfsense reboot or if you click save on squid GUI or if squidguard or dansguardian restart squid.

Further you should have a look at this chapter when using refresh pattern:
http://doc.pfsense.org/index.php/Squid_Package_Tuning#Tweaking_Update_Caching_.2F_Squid_seems_to_download_on_its_own

Even if the titel is not exactly what you want you should inform about these options on the squid-cache.org wiki.

[quetzalcoatl]

I noticed that the semicolon ; at the end of the custom config is still used in that tutorial
but i saw somewhere that the ";" is not longer needed in the new pfsense/squid configs

Also i keep using the sign $ in custom config but the $ symbol does not appear in that tutorial…

Should i get rid of the $ symbol from all my custom lines?
Should i add the semicolon at the end of every custom config line?

[Nachtfalke]

The semicolon depends in what the GUI uses to separate different config lines.

In squid.conf it must be without semicolon.

The $ Defines that the regex must be on the end of the URL. You can use it but you do not need to.

[tester_02]

marcelloc.  I've been going through all the v3 threads for the last while and am trying to understand what the current status of the package is.

Seems like for 2.0.X we have to manually get some libs and then install dev package from front end.  Do we still have to manually fetch an update package from your personal repo?  Is there plans to move to a setup like squid where the updates are always published and we just reinstall the package from the gui?

For 2.1, we can do the same as above, but we just don't need to manually install the libs?

Maybe the dev's can sticky a thread and give marcelloc access to update post #1 in a thread and keep the latest status updated?  It's hard to follow the status of the package progress, unlike snort dev.

[marcelloc]

If you enable ipv6 on pfsense 2.0.3 or use 2.1, all you need is to install package from gui and missing libs from my repo.

Antivirus integration is still not working.

SSL interception is working fine.

cache issues can get better folowing this doc
http://doc.pfsense.org/index.php/Squid_Package_Tuning#Tweaking_Update_Caching_.2F_Squid_seems_to_download_on_its_own

[avp]

Hi, I had Squid and SquidGuard working for some time.  I recently noticed that it was no longer working, probably for months.  So I've uninstalled both packages.  Now I'm trying to use this Squid 3.3.4 pkg.  I appear to have Squid running correctly.  I've copied the libs from your repo.  I'm having a lot of trouble with SquidGuard.  I had it running, but it was blocking everything.  I had it running in the past, so I feel I am somewhat comfortable with the settings etc.  Now I can no longer get it running at all.  I get the msg 0/5 SG process started.  I know I haven't given much specifics here for your help, but I am wondering if there are any guidelines to try and get this pkg working with SG?

In the bigger picture, I'm looking for Squid + SG + AV functionality, I don't mind if I have to use different pkgs if that is the recommendation?

Also, it seems there are so many Squids, SGs in the pkg repo, and no foolproof instructions to get SG set up…

Any advice much appreciated. Thanks.

[marcelloc]

Use squid 3.3.5.

Since squid 3.x squidguard is started on demand, ifyou have no traffic, tthen no squid guard daemon will be running.

[avp]

Ok, I am using your 3.3.5.

I think I located some of my issue - Proxy server, Custom settings still had remnants from HAVP in there.  I think I have Squid + SG working correctly now.

Should it be possible to get HAVP working with this Squid + SG?  Or do you think you will get your AV integration working sometime soon?  Can I help with this?

Thanks very much…