Squid 3.3.4 package for pfsense with ssl filtering

stanthewizard

sorry I don't understand

I think I have no specific url

marcelloc

@stanthewizard:

sorry I don't understand

On console/ssh use pkg_delete to remove squid-3.3.4 and pkg_add to get squid 3.3.5

i386 systems
pkg_add -r http://e-sac.siteseguro.ws/packages/8/All/squid-3.3.5.tbz

amd64 systems
pkg_add -r http://e-sac.siteseguro.ws/packages/amd64/8/All/squid-3.3.5.tbz

stanthewizard

OK

I installed it … but nothing in the GUI packages ...

Did I do something wrong ?

marcelloc

@stanthewizard:

Did I do something wrong ?

You forgot to install squid3-dev package before doing pkgdell/add on console.

stanthewizard

At last It works

Thanks

BTW
with squid 3.3.5 there is a bug in 2.1 RC0 … endless loop updating package

marcelloc

@stanthewizard:

At last It works

Thanks

BTW
with squid 3.3.5 there is a bug in 2.1 RC0 … endless loop updating package

During firmware upgrade?

stanthewizard

At reboot

Warning about package updating …. endlessly

quetzalcoatl

I always use the latest snapshot, install new 64 bit snapshot every week during the night when no client of mine browses the internet.

I also install squid 3.3.5 package or whatever is the latest.

I never install squidguard or any other package. The absolutely only package i have installed is squid 3.3.x

Following marcello instructions i got some caching, but it's a ridiculous 1% byte hit ratio. Most of the windows updates are not cache anymore.(maybe they are stored into cache but not server, or not even stored at all).

Also there are some pages that get updated daily and squid does not detect the update.

www.filehippo.com and www.jalopnik.com are 2 of the pages that i noticed that squid does not detect that they have been updated…..
Sometimes even a week later squid is showing me the week old page, and both filehippo and jalopnik update daily.

The only way to see those updated pages is turning the squid service off.

There is an option with a checkbox that says "Enable offline mode" and i keep that off otherwise squid will never check tons of page updates.
Enable offline mode is off in my squid.

I always do have some custom refresh options in the "custom options" field.....but even if i leave that field empty, save and restart the whole pfsense machine, i never get a decent hit ratio or byte hit ratio.

I don't even ask for youtube caching because i know it's hard and complicated, but at least some caching like before with ms windows updates, ms office updates, and exe files caching.

I used to be able to download the winrar, winamp, VLC player and more files at 180 mb/s using older squids (when it was already downloaded once).

Now with 3.3.x i never get the cached file, it always starts downloading again at 24kb/s........

Am I the only one with this issue? what am i doing wrong?

marcelloc

@quetzalcoatl:

Am I the only one with this issue? what am i doing wrong?

Did you disabled dynamic content? This behavior/issue happens when its enabled.

Nachtfalke

Hi,

are the "known bugs" on the very first post of this thread still persistent?

marcelloc

@Nachtfalke:

Hi,

Are the "known bugs" on the very first post of this thread still persistent?

Only antivirus integration, missing libs and enable ipv6 on 2.0.x.

I can't edit the first topic after one or two days, Maybe only moderators.

Deadringers

Hi can anyone help with my issue detailed here:http://forum.pfsense.org/index.php/topic,63618.0.html

Seems that when I try to use either proxy or transparent proxy it just doesn't pass any traffic? :(

quetzalcoatl

I am keeping dynamic content caching always disabled because if I enable it i don't even get 1% byte hit ratio, it just goes to 0% hit ratio.

Maybe i can post here the full squid config so you can see it and tell me where the issue is.

I just don't know where and what squid config file i need to post.

Since you told me to disable dynamic content caching i do get some hits but is ridiculous. It caches almost nothing and when it does, it will not update the page if changed serving always an offline like page.

Nachtfalke

Squid.conf is the important file.

Fürther check the pfsense docs for squid caching options like Windows updates. It should hell you to cache other things.

quetzalcoatl

Thank you Nachtfalke for the tip!
Actually thanks to this new "squid.conf" discover, from now on i will just copy and paste squid.conf content for every new installation instead of using the GUI.
So it's quicker to set squid up and not being afraid that i forgot some setting.

I wonder if there are other files than squid.conf that get changed when i setup squid from the GUI.
It says " Do not edit manually !" but it's so tempting and easy to edit manually. Can i go on and edit manually and save that file?
Hopefully someone can answer that.

Here i found squid.conf contents.
Seeing this config i hope you can tell me how to cache more stuff to get a decent hit ratio and avoid page updating issues:

squid.conf:

This file is automatically generated by pfSense

Do not edit manually !

http_port 10.0.0.10:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language en
icon_directory /usr/pbi/squid-amd64/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none

logfile_rotate 0
shutdown_lifetime 3 seconds

Allow local network(s) on interface(s)

acl localnet src 10.0.0.0/24
httpd_suppress_version_string on
uri_whitespace strip

acl dynamic urlpath_regex cgi-bin ?
cache deny dynamic

cache_mem 6000 MB
maximum_object_size_in_memory 16384 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 50000 16 256
minimum_object_size 0 KB
maximum_object_size 900000 KB
offline_mode off
cache_swap_low 90
cache_swap_high 95
cache allow all

No redirector configured

#Remote proxies

Setup some default acls

From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.

acl localhost src 127.0.0.1/32

acl allsrc src all
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 4080 3128 3127 1025-65535
acl sslports port 443 563 4080

From 3.2 further configuration cleanups have been done to make things easier and safer. The manager, localhost, and to_localhost ACL definitions are now built-in.

#acl manager proto cache_object

acl purge method PURGE
acl connect method CONNECT

Define protocols used for redirects

acl HTTP proto HTTP
acl HTTPS proto HTTPS
http_access allow manager localhost

http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslports

Always allow localhost connections

From 3.2 further configuration cleanups have been done to make things easier and safer.

The manager, localhost, and to_localhost ACL definitions are now built-in.

http_access allow localhost

request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrc

Reverse Proxy settings

Custom options

refresh_pattern -i .(3g2|3gp|asf|asx|avi|divx|flv|iff|ifo|m3u|m4a|m4v|mov|mpa|mpeg|mpe|qt|qtm|viv|mpg|ogg|rm|rmvb|scr|swf|vob|wmv|x-flv|xvid)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(aif|aiff|amr|cda|mid|wav|wma|midi|au|ram|ra|snd|mp2|mp3|mp4)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(3dm|ai|ani|art|bmp|cdr|cdt|cmf|cur|drw|dwg|dxf|eps|eps2|gif|icl|icm|ico|indd|jpeg|jpg|jpe|max|pct|pcx|png)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(ps|psd|psp|qxd|qxp|rels|svg|tga|thm|tif|tiff|wmf|wrl|xbm|xcf|xif|yuv|pnm|pbm|pgm|ppm|rgb|xpm|xwd|pic|pict)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(accdb|bfc|cbr|chm|csv|db|dbf|doc|docx|dot|hlp|kml|Kmz|lab|log|mdb|msg|odt|ost|pages|pdb|pdf|pps|txt|ppt|pptx|pst|pub|rtf|wpd|wps|wri|xlr|xls|xlsx|xlt)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(app|bat|cmd|com|exe|gadget|msi|pif|vb|wsf|torrent)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(8bi|bin|cat|cpl|dbx|dll|drv|gam|hex|hqx|lnk|nes|plugin|reg|rom|sav|sys|xll)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(arj|sit|zip|rar|rgz|psf|lzh|lha|cab|tar|tgz|gz|Z|wp|wp5|7z|pkg|rpm|sea|sitx|tar.gz|zipx|prn|srf|tex|latax|gpf|upd|jar|bz2|gzip|ace|kf|a[0-9][0-9]|r[0-9][0-9])$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(fnt|fon|otf|ttf)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(dmg|iso|toast|vcd)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(api|bas|c|cbl|class|cpp|cs|dtd|fla|java|m|pl|py|vbx)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(bak|bup|cdl|cfg|dat|deb|dss|dvf|efx|emf|eml|gho|gpx|ini|key|keychain|m4b|m4p|mcd|mim|mswmm|ori|prf|ptb|qbb|qbw|raw|sdf|ses|sql|ss|tmp|uue|uxx|vcf|xml|xsl|xtm)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i .(ht|htm|html|shtml|xhtml|css|js|jsp|asp|cer|cgi|csr|part|php|phtml|rss)$ 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern ^gopher: 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern ^ftp: 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern . 99999 100% 9999999 override-expire override-lastmod ignore-no-store ignore-reload ignore-private reload-into-ims refresh-ims store-stale
refresh_pattern -i (/cgi-bin/|?)$ 0 0% 0

Setup allowed acls

Allow local network(s) on interface(s)

http_access allow localnet

Default block all to be sure

http_access deny allsrc

marcelloc

@quetzalcoatl:

It says " Do not edit manually !" but it's so tempting and easy to edit manually. Can i go on and edit manually and save that file?
Hopefully someone can answer that.

You can do that if you do not install package via packages gui. remove squid package and add via console/ssh squid install via pkg_add.

Nachtfalke

As marcelloc said, if you do not use the package offered by the pfsense package manager you can do that.
But if you use squid with GUI then the squid.conf will be overwritten when pfsense reboot or if you click save on squid GUI or if squidguard or dansguardian restart squid.

Further you should have a look at this chapter when using refresh pattern:
http://doc.pfsense.org/index.php/Squid_Package_Tuning#Tweaking_Update_Caching_.2F_Squid_seems_to_download_on_its_own

Even if the titel is not exactly what you want you should inform about these options on the squid-cache.org wiki.

quetzalcoatl

I noticed that the semicolon ; at the end of the custom config is still used in that tutorial
but i saw somewhere that the ";" is not longer needed in the new pfsense/squid configs

Also i keep using the sign $ in custom config but the $ symbol does not appear in that tutorial…

Should i get rid of the $ symbol from all my custom lines?
Should i add the semicolon at the end of every custom config line?

Nachtfalke

The semicolon depends in what the GUI uses to separate different config lines.

In squid.conf it must be without semicolon.

The $ Defines that the regex must be on the end of the URL. You can use it but you do not need to.

tester_02

marcelloc. I've been going through all the v3 threads for the last while and am trying to understand what the current status of the package is.

Seems like for 2.0.X we have to manually get some libs and then install dev package from front end. Do we still have to manually fetch an update package from your personal repo? Is there plans to move to a setup like squid where the updates are always published and we just reinstall the package from the gui?

For 2.1, we can do the same as above, but we just don't need to manually install the libs?

Maybe the dev's can sticky a thread and give marcelloc access to update post #1 in a thread and keep the latest status updated? It's hard to follow the status of the package progress, unlike snort dev.