Squid 3.3.4 package for pfsense with ssl filtering

marcelloc · May 20, 2013, 2:36 PM

I'll try to push the fix I've applied to 2.0.x to freebsd ports.

Since it's merged, I'll ask core team for another compile run. :)

att,
Marcello Coutinho

quetzalcoatl · May 24, 2013, 1:48 PM

Squid no caching and getting more data in from WAN than what i get out of LAN (in interface statistics)

I download the missing files…
Then i install squid 3.3.4 dev
Then all in the GUI i just set resolve ipv4 first, enable transparent proxy, and set max object size in RAM 8192KB, Max RAM for squid 6000MB, max HDD 50GB max object size in disk 900MB, enable caching for dynamic stuff selecting youtube and windows updates and leave everything else as it is.
I don't even add any custom option.

and as soon as i start the squid service i start getting more data downloaded from WAN than what is served to LAN.

in fact:

interface statistics before starting squid service: WAN in: 2.12GB - LAN OUT 2.12GB
interface statistics after starting squid service: WAN in: 2.93GB - LAN OUT 2.59GB

I always got more LAN out than WAN in when Squid is caching properly.

I remember those manual settings that made me get more WAN in than LAN out:
quick_abort_min
quick_abort_max
quick_abort_pct

but even if i don't use them i get more WAN in than LAN out.

I even set them as:
quick_abort_min 0 KB
quick_abort_max 0 KB
quick_abort_pct 0

but nothing improves.

I can roughly say that with squid 2.7(lusca) i was getting a 15% byte hit ratio, with squid 2.1 a 5% byte hit ratio, and with 3.3.4 0% hit ratio...or should i say -3% hit ratio? (note the negative value)

it looks that the newer the squid is, the less caching it does.........
i miss the good days when 100% of windows and office updates were downloaded from the squid cache at 100 megabit speeds!
at that time it took more to install updates than to download 1Gb of updates from pfsense. even with internet speeds of just 1 megabit!!!!!
i wonder where is going all this effort to update squid with these results..........or am i just doing massive configuration errors?

marcelloc · May 24, 2013, 2:01 PM

Disabling dynamic content option on gui and check if on log files you get only TCP_MISS or you start seeing some TCP_HIT.

quetzalcoatl · May 24, 2013, 7:42 PM

at last getting some hits.

I don't even know how to see squid logs, except than going to diagnostics, edit file, /var/squid/logs/cache.log but i don't see any miss/hit statistics there.

anyways as soon as i disabled dynamic caching the "LAN out" number is growing faster than "WAN in" in interface statistics.

So at last my squid cache is working.

Maybe i got confused because the previous squid 3.3.4 release was not caching for some reason even when dynamic content caching was off.

Thanks marcelloc

marcelloc · May 24, 2013, 11:04 PM

@quetzalcoatl:

I don't even know how to see squid logs

There is a realtime tab on gui.

You can also go via console/ssh and do a tail -f /var/squid/logs/access.log

quetzalcoatl · May 25, 2013, 12:55 AM

The only file i have in the "/var/squid/logs" folder is "cache.log"

Also if i go to the "real time" tab in the GUI i see this stuff:

Max lines: Max. lines to be displayed.
String filter: Enter a grep like string/pattern to filterlog.
eg. username, ip addr, url.
Use ! to invert the sense of matching, to select non-matching lines.

Squid Logs
Date IP Status Address User Destination

SquidGuard Logs
Date-Time ACL Address Host User

but everything is empty and i see no data or statistics, just the section titles i just pasted here.

marcelloc · May 26, 2013, 12:52 PM

Did you enabled logging on squid GUI configuration?

Fehler20 · May 26, 2013, 7:22 PM

I have to report another bug and a problem:

1.) If you enable transparent proxy an disable SSL-interception, transparent mode does not work. It seems that there is a problem with the intercept command with transparent connections at the config file. If you change from "192.168.x.x:8080 intercept" to "192.168.x.x:8080 transparent" everything is ok.

2.) If you enable the proxy for more than one interface only the first gets access to the internet. For every other interface, access is denied, regardless wether you define an acl or check the option "Allow users on interface".

packeteer · May 26, 2013, 11:23 PM

I have a strange issue with this version of squid. Squid service is running but unable to connect on the designated port (3128).

Reverted back to the older version.

marcelloc · May 27, 2013, 2:23 AM

@packeteer:

I have a strange issue with this version of squid. Squid service is running but unable to connect on the designated port (3128).

If you are on 2.0.x, you need to enable ivp6 or update package via pkg_delete and pkg_add from my repo.

marcelloc · May 27, 2013, 3:01 PM

Util freebsd ports are updated, I've pushed squid-3.3.5 pbi files to my repo.

This way you can test/use squid3 package on latest 2.1-rc0

pfsense 2.1 amd64
pbi_delete squid-3.3.4-amd64
fetch http://e-sac.siteseguro.ws/packages/amd64/8/All/squid-3.3.5-amd64.pbi
fetch http://e-sac.siteseguro.ws/packages/amd64/8/All/squid-3.3.5-amd64.pbi.sha256
pbi_add –no-checksig squid-3.3.5-amd64.pbi
rehash

pfsense 2.1 i386
pbi_delete squid-3.3.4-i386
fetch http://e-sac.siteseguro.ws/packages/8/All/squid-3.3.5-i386.pbi
fetch http://e-sac.siteseguro.ws/packages/8/All/squid-3.3.5-i386.pbi.sha256
pbi_add –no-checksig squid-3.3.5-i386.pbi
rehash

quetzalcoatl · May 27, 2013, 4:48 PM

Fehler20 says that if he keeps SSL-interception disabled the transparent proxy doesn't work.

That is the same issue that i believe is happening to me.

I will try to turn on SSL-interception, but i don't know if i have to configure something to make it work properly with the transparent proxy.

And yes, i did forget to turn squid logging on!
Sorry!

wheelz · May 27, 2013, 4:55 PM

I set up my 2.0.3 version to try the SSL filtering. I got the HTTP traffic going through squid so that works. I generated a Test CA cert in Cert Manager and installed in on my test workstation. However when I go to https://www.google.com it just spins at connecting… I can telnet to port 3129 and something answers so I think the traffic is getting through the firewall. Squid service appears to be running but I don't see anything in the logs about the request. Any help?

marcelloc · May 27, 2013, 5:18 PM

@wheelz:

Any help?

Did you replaced squid to 3.3.5 from my repo?

wheelz · May 27, 2013, 6:04 PM

@marcelloc:

Did you replaced squid to 3.3.5 from my repo?

I have now but I'm not sure if I did it right. I'm getting this when I'm try to run squid:

/libexec/ld-elf.so.1: /usr/lib/librt.so.1: unsupported file layout

I wasn't sure if I needed to do that pbi command either since I'm on 2.0.3. I tried to see if it would execute but it wasn't there.

marcelloc · May 27, 2013, 8:44 PM

@wheelz:

/libexec/ld-elf.so.1: /usr/lib/librt.so.1: unsupported file layout

on 2.0.3, use pkg_add and pkg_delete to get squid 3.3.5

i386
pkg_add -r http://e-sac.siteseguro.ws/packages/8/All/squid-3.3.5.tbz

amd64
pkg_add -r http://e-sac.siteseguro.ws/packages/amd64/8/All/squid-3.3.5.tbz

wheelz · May 27, 2013, 8:51 PM

@marcelloc:

@wheelz:

/libexec/ld-elf.so.1: /usr/lib/librt.so.1: unsupported file layout

on 2.0.3, use pkg_add and pkg_delete to get squid 3.3.5

i386
pkg_add -r http://e-sac.siteseguro.ws/packages/8/All/squid-3.3.5.tbz

amd64
pkg_add -r http://e-sac.siteseguro.ws/packages/amd64/8/All/squid-3.3.5.tbz

Hmm… that's what I did. It complained about perl and openssl (I think) so I deleted them and let the pkg_add -r install the versions it wanted. After that it was successfull, however squid -v won't run with the error above.

marcelloc · May 27, 2013, 9:06 PM

you need libs and package with same arch as you pfsense version

if you installed pfsense 32 bits, you need i386 libs and packages
if you installed pfsense 64 bits, you need amd64 libs and packages

wheelz · May 28, 2013, 2:22 AM

ok, my fault. I had copy pasted the wrong PACKAGESITE path. I have that fixed now and squid starts fine, but still the same as before. HTTP goes fine but HTTPS just sits trying to connect. I just have HTTPS/SSL interception enabled on the loopback adapter, port 3129 (NAT rule to forward 3129 and firewall rule allowing it in). My TestCA certificate is selected and I left the rest as the defaults. My client has the TestCA certificate installed in the trusted CAs and I configured the SSL proxy to my pfsense on port 3129.

Here is what squid says:

squid -v
Squid Cache: Version 3.3.5
configure options:  '--with-default-user=squid' '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/var' '--sysconfdir=/usr/local/etc/squid' '--with-logdir=/var/log/squid' '--with-pidfile=/var/run/squid/squid.pid' '--with-swapdir=/var/squid/cache/squid' '--enable-auth' '--enable-build-info' '--enable-loadable-modules' '--enable-removal-policies=lru heap' '--disable-epoll' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-translation' '--enable-auth-basic=DB MSNT MSNT-multi-domain NCSA PAM POP3 RADIUS  fake getpwnam LDAP SASL NIS' '--enable-auth-digest=file' '--enable-external-acl-helpers=file_userip time_quota unix_group LDAP_group kerberos_ldap_group' '--enable-auth-negotiate=kerberos wrapper' '--enable-auth-ntlm=fake smb_lm' '--enable-storeio=diskd rock ufs aufs' '--enable-disk-io=AIO Blocking DiskDaemon IpcIo Mmapped DiskThreads' '--enable-log-daemon-helpers=file' '--enable-url-rewrite-helpers=fake' '--disable-ipv6' '--enable-delay-pools' '--enable-ssl' '--with-openssl=/usr' '--enable-ssl-crtd' '--enable-icmp' '--enable-htcp' '--disable-forw-via-db' '--enable-cache-digests' '--enable-wccp' '--enable-wccpv2' '--enable-eui' '--disable-ipfw-transparent' '--enable-pf-transparent' '--disable-ipf-transparent' '--enable-follow-x-forwarded-for' '--disable-ecap' '--enable-icap-client' '--disable-esi' '--enable-kqueue' '--with-large-files' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=amd64-portbld-freebsd8.3' 'build_alias=amd64-portbld-freebsd8.3' 'CC=cc' 'CFLAGS=-O2 -pipe -I/usr/local/include -I/usr/local/include -I/usr/include -DLDAP_DEPRECATED -fno-strict-aliasing' 'LDFLAGS= -L/usr/local/lib -L/usr/local/lib -pthread -Wl,-rpath=/usr/lib:/usr/local/lib -L/usr/lib' 'CPPFLAGS=-I/usr/local/include' 'CXX=c++' 'CXXFLAGS=-O2 -pipe -I/usr/local/include -I/usr/local/include -I/usr/include -DLDAP_DEPRECATED -fno-strict-aliasing' 'CPP=cpp'

Here is pkg_info:

pkg_info
arc-5.21p           Create & extract files from DOS .ARC files
arj-3.10.22_4       Open-source ARJ
bsdinstaller-2.0.2013.0412 BSD Installer mega-package
ca_root_nss-3.14.1  The root certificate bundle from the Mozilla Project
clamav-0.97.6       Command line virus scanner written entirely in C
compat6x-amd64-6.4.604000.200810_3 A convenience package to install the compat6x libraries
cyrus-sasl-2.1.26_2 RFC 2222 SASL (Simple Authentication and Security Layer)
dansguardian-2.12.0.3 A fast, feature-rich web content filter for Squid proxy ser
db41-4.1.25_4       The Berkeley DB package, revision 4.1
gettext-0.18.1.1    GNU gettext package
lha-1.14i_6         Archive files using LZSS and Huffman compression (.lzh file
libecap-0.2.0_1     Library for module based network content analysis
libiconv-1.14       A character set conversion library
libltdl-2.4.2       System independent dlopen wrapper
libwww-5.4.0_4      The W3C Reference Library
nano-2.2.6          Nano's ANOther editor, an enhanced free Pico clone
openldap-sasl-client-2.4.35 Open source LDAP client implementation with SASL2 support
pcre-8.32           Perl Compatible Regular Expressions library
perl-5.14.2_3       Practical Extraction and Report Language
squid-3.3.5         HTTP Caching Proxy
unzoo-4.4_2         A zoo archive extractor

Port 3129 connect via telnet but HTTPS connections time out without anything mentioned in the sys logs or access.log. Am I doing something wrong?

marcelloc · May 28, 2013, 2:35 AM

@wheelz:

I just have HTTPS/SSL interception enabled on the loopback adapter, port 3129 (NAT rule to forward 3129 and firewall rule allowing it in).

No need to create a nat rule.

Remove your nat, enable SSL on interface other then loopback and let squid do the config for you.