Squid3 - New GUI with sync, normal and reverse proxy
-
Hi,
i'm using without any probs the squid3 (3.1.20 pkg 2.0.5_3) package on pfsense 2.0.1. thanks for this.
is it possible to use squid on 2 diffent ports on the same ip. for now it's '8080', additionally i want to use '3128'. 8080 should use the wan1 and 3128 should use the wan2 connection. is this in any way possible?
greets
-
you can try this config on custom options.
find the listening args you need and place there.
-
I don't understand what the point is of having XMLRPC sync without being able to listen on a CARP IP address.
- can anyone tell me if there is a way to get this pkg to listen on a CARP IP (or even just listen on all interfaces!), and
- what is an example use case for synchronized configs otherwise?
(Also see http://redmine.pfsense.org/issues/2591 and http://redmine.pfsense.org/issues/2592.)
Thanks,
-Adam -
Unless the cache and connection state are shared between squid instances, you don't want it binding on a CARP VIP. It makes little sense to have the traffic leave from a shared IP and shared pf states when the actual connections in squid are independent, even if the config is synchronized.
The point of syncing the config is so the slave can take over, and it can, you just can't seamlessly fail over 100% in squid. Even if the states were to sync over it doesn't matter, the squid process itself would have no knowledge of the connection.
-
- can anyone tell me if there is a way to get this pkg to listen on a CARP IP (or even just listen on all interfaces!), and
Listen squid on loopback(lo0) and create nat rules from carp address to 127.0.0.1
2.1 gui framework has the listen all as well listen on virtual ips, so squid3 on pfsense 2.1 will have this option and no need to workaround with nats.
-
Unless the cache and connection state are shared between squid instances, you don't want it binding on a CARP VIP. It makes little sense to have the traffic leave from a shared IP and shared pf states when the actual connections in squid are independent, even if the config is synchronized.
The point of syncing the config is so the slave can take over, and it can, you just can't seamlessly fail over 100% in squid. Even if the states were to sync over it doesn't matter, the squid process itself would have no knowledge of the connection.I don't particularly care about seamless failover, I just want to avoid using PAC. If I have a redundant pair of firewalls, I use the CARP VIP as my default gateway from the inside. The way it's set up right now, I have to point all the clients at one firewall or the other's physical (LAN) IP address. So if that firewall dies, I lose the ability to browse the web without manually reconfiguring each client…? Whereas if squid was bound to the CARP VIP, any in-flight HTTP transactions would fail, but at least I wouldn't have to go around and reconfigure every client. Is there some other mechanism I don't know about?
Listen squid on loopback(lo0) and create nat rules from carp address to 127.0.0.1
2.1 gui framework has the listen all as well listen on virtual ips, so squid3 on pfsense 2.1 will have this option and no need to workaround with nats.That approach hadn't occurred to me - trying it now to see if it works, but I don't expect problems.
However, as to the second point: I am using 2.1-BETA, and I do not see any such option. ("squid3" package, version "beta 3.1.20 pkg 2.0.5_3")-Adam
-
NAT is indeed the easy solution there.
-
However, as to the second point: I am using 2.1-BETA, and I do not see any such option. ("squid3" package, version "beta 3.1.20 pkg 2.0.5_3")
I said it will, I did not included the code to squid package yet ;)
- 10 days later
-
Hi, my squid cannot start :'(, here is the message in cache.log:
2012/08/28 23:09:44| Accepting HTTP connections at [::]:3128, FD 29. 2012/08/28 23:09:44| HTCP Disabled. 2012/08/28 23:09:44| Ready to serve requests. 2012/08/28 23:09:44| WARNING: ssl_crtd #1 (FD 19) exited 2012/08/28 23:09:44| WARNING: ssl_crtd #2 (FD 21) exited 2012/08/28 23:09:44| WARNING: ssl_crtd #3 (FD 23) exited (ssl_crtd): Uninitialized SSL certificate database directory: /var/squid/lib/ssl_db. To initialize, r un "ssl_crtd -c -s /var/squid/lib/ssl_db". 2012/08/28 23:09:44| WARNING: ssl_crtd #4 (FD 25) exited 2012/08/28 23:09:44| Too few ssl_crtd processes are running 2012/08/28 23:09:44| storeDirWriteCleanLogs: Starting... 2012/08/28 23:09:44| Finished. Wrote 0 entries. 2012/08/28 23:09:44| Took 0.00 seconds ( 0.00 entries/sec). FATAL: The ssl_crtd helpers are crashing too rapidly, need help! Squid Cache (Version 3.1.20): Terminated abnormally. CPU Usage: 0.092 seconds = 0.061 user + 0.031 sys Maximum Resident Size: 10532 KB Page faults with physical i/o: 0 (ssl_crtd): Uninitialized SSL certificate database directory: /var/squid/lib/ssl_db. To initialize, r un "ssl_crtd -c -s /var/squid/lib/ssl_db".
I have tried both squid 2 & 3, both got the same message ::)
Also it seems there is no "ssl_crtd" exist in my system…so I cannot try to run the command :o
Do you have any suggestion..? :'(Thanks
-
I have this dir on my squid3 install.
What are you trying to configure?
ls -la /var/squid/lib/ total 6 drwxr-xr-x 3 proxy proxy 512 Apr 14 02:17 . drwxrwxr-x 7 proxy proxy 512 Apr 14 02:17 .. drwxr-xr-x 2 proxy proxy 512 Apr 14 02:17 ssl_db
- 16 days later
-
Hey guys,
I have a strange problem regarding reverse proxy and authentification with Certificates… The package itself is working as expected and without certs needed to authenticate I can open all services (owa, msas, rpc).
Now here it comes: (all IP's, ports and url's are renamed for security reasons ;) )
if i use this line in my squid.conf:
https_port xxx.xxx.xxx.xxx:Port cert=/website.crt key=/website.key defaultsite=this.website.me vhost
everything works as expected. I get my login page… but if I use this one:
https_port xxx.xxx.xxx.xxx:Port cert=/website.crt key=/website.key defaultsite=this.website.me clientca=/CA.crt cafile=/CA.crt capath=/ sslcontext=id vhost
I get asked for my client cert and then "page cannot be found" with this in squid logs:
clientNegotiateSSL: Error negotiating SSL connection on FD 17: error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm (1/-1)
My CA is created with SHA512… So the clients are too ;)
When I doopenssl list-message-digest-commands
on my pfsense shell, I get
md2 md4 md5 mdc2 rmd160 sha sha1
But with openssl speed, sha512 is listed!
What I found so far is, that somewhere in the squid source this line should be added
OpenSSL_add_all_algorithms()
somwhere like
src/ssl_support.* function: ssl_initialize(void)
Due to the lack of time, I wasn't able to get deeper inside… Does someone has an idea?!
@marcelloc:
If this needs to be posted somewhere else, please let mo know. If I understand this package right, you are compiling squid on your own?! So this could be solved easily...Kind regards
EDIT:
found it in src/ssl_support.cc, line 593SSL_load_error_strings(); SSLeay_add_ssl_algorithms();
what i found by google, there should also be called: OpenSSL_add_all_algorithms()
@marcelloc:
possible? -
@marcelloc:
If this needs to be posted somewhere else, please let mo know. If I understand this package right, you are compiling squid on your own?! So this could be solved easily…No more :(
Packages now must be only compiled and provided by official repo, you can compile it on 8.1 freebsd, create the package and then install on your system.
Core team are really busy these days. I'm waiting some packages to be recompiled too. After this, I can continue package development and testing.
these are current squid3 compile options available on ports
SQUID_KERB_AUTH Install Kerberos authentication helper
SQUID_LDAP_AUTH Install LDAP authentication helpers
SQUID_NIS_AUTH Install NIS/YP authentication helpers
SQUID_SASL_AUTH Install SASL authentication helpers
SQUID_IPV6 Enable IPv6 support
SQUID_DELAY_POOLS Enable delay pools
SQUID_SNMP Enable SNMP support
SQUID_SSL Enable SSL support for reverse proxies
SQUID_SSL_CRTD Enable SSL certificate daemon
SQUID_PINGER Install the icmp helper
SQUID_DNS_HELPER Use the old 'dnsserver' helper
SQUID_HTCP Enable HTCP support
SQUID_VIA_DB Enable forward/via database
SQUID_CACHE_DIGESTS Enable cache digests
SQUID_WCCP Enable Web Cache Coordination Prot. v1
SQUID_WCCPV2 Enable Web Cache Coordination Prot. v2
SQUID_STRICT_HTTP Be strictly HTTP compliant
SQUID_IDENT Enable ident (RFC 931) lookups
SQUID_REFERER_LOG Enable Referer-header logging
SQUID_USERAGENT_LOG Enable User-Agent-header logging
SQUID_ARP_ACL Enable ACLs based on ethernet address
SQUID_IPFW Enable transparent proxying with IPFW
SQUID_PF Enable transparent proxying with PF
SQUID_IPFILTER Enable transp. proxying with IPFilter
SQUID_FOLLOW_XFF Follow X-Forwarded-For headers
SQUID_ECAP En. loadable content adaptation modules
SQUID_ICAP Enable ICAP client functionality
SQUID_ESI Enable ESI support (experimental)
SQUID_AUFS Enable the aufs storage scheme
SQUID_COSS Enable COSS (currently not available)
SQUID_KQUEUE Use kqueue(2) (experimental)
SQUID_LARGEFILE Support log and cache files >2GB
SQUID_STACKTRACES Create backtraces on fatal errors
SQUID_DEBUG Enable debugging options -
Packages now must be only compiled and provided by official repo, you can compile it on 8.1 freebsd, create the package and then install on your system.
Core team are really busy these days. I'm waiting some packages to be recompiled too. After this, I can continue package development and testing.
these are current squid3 compile options available on ports
So if I would have a FreeBSD 8.1, download squid sources and build it with all the options marked in your post, we could run the binaries on pfsense?
EDIT:
could I use squid 3.2.1 source? or won't this one run on pfsense?EDIT #2:
downloading freebsd 8.1 i386. will try to get things work.@marcelloc:
please let me know, which version of squid I should compile. 3.1.19, 3.1.20 or 3.2.1? -
The best way is to use freebsd ports to compile.
Portsnap fetch
Portsnap extractCD /usr/ports/www/squid31
make package -
why don't we use version 3.2.1?
-
It's not on ports yet. To create the package for 3.2.1 we need first a port for it.
We can try to create it based on 3.1.20 dir -
did it.
here we go:
compiled squid 3.1.20 with ports on FreeBSD 8.1. I modified the source and now squid is able to authenticate users with certs (all digests) in reverse mode… i try to find some place to upload the binaries.EDIT:
forgot to ask… are you interested in including "my" squid version? I just added the mentioned one line in the source.... -
Post what you did. Maybe I can send it as a feature request to squid port maintained @ freebsd
-
this is what i did:
source file: ssl_support.cc, around line 593 (in squid 3.2.1 the file is located in "ssl" subdir named "support.cc")
function: ssl_initialize(void)
original:
SSL_load_error_strings(); SSLeay_add_ssl_algorithms();
to:
SSL_load_error_strings(); OpenSSL_add_all_algorithms(); SSLeay_add_ssl_algorithms();
would you like to include my binaries in your package?
-
@namek:
I am using this mainly for content filtering/caching/logging…
squid3 has some new features on it's gui too, check it on a test machine first.
-
@namek:
Oh yes, one the features I found was 'patch captive portal' - which I believe that if the users are on a proxy they will not be able to bypass captive portal?
yes. I did it based on some topics on this forum with problems on using captive and squid.
@namek:
And ofcourse there are more features..are many users using this in production? as you have marked it as "Beta"
another striking feature are the realtime logs integrated in the GUI itself. So i believe that somewhat eliminates the need of using the 'realtime' feature of both sarg and lightsquid…The realtime tab on squid3 package is to help debug access while using squid and squidguard.
-
Hi all, i have install squid3 package and just use standard config, squid reverse not config yet. After install i have problem to access banking website and got error msg from squid Unable to determine IP address from host name "www.cimbclicks.com.my". How can i solve this problem? Below my squid config.
# This file is automatically generated by pfSense # Do not edit manually ! http_port 192.168.10.1:3128 http_port 127.0.0.1:3128 intercept icp_port 7 dns_v4_first on pid_filename /var/run/squid.pid cache_effective_user proxy cache_effective_group proxy error_default_language English icon_directory /usr/local/etc/squid/icons visible_hostname localhost cache_mgr ikhwans@kbc.com.my access_log /var/squid/logs/access.log cache_log /var/squid/logs/cache.log cache_store_log none sslcrtd_children 0 logfile_rotate 0 shutdown_lifetime 3 seconds # Allow local network(s) on interface(s) acl localnet src 192.168.10.0/24 uri_whitespace strip acl dynamic urlpath_regex cgi-bin ? cache deny dynamic cache_mem 32 MB maximum_object_size_in_memory 64 KB memory_replacement_policy heap GDSF cache_replacement_policy heap LFUDA cache_dir ufs /var/squid/cache 1000 16 256 minimum_object_size 0 KB maximum_object_size 4 KB offline_mode offcache_swap_low 90 cache_swap_high 95 # No redirector configured #Remote proxies # Setup some default acls acl allsrc src all acl localhost src 127.0.0.1/32 acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 8888 3128 1025-65535 acl sslports port 443 563 8888 acl manager proto cache_object acl purge method PURGE acl connect method CONNECT http_access allow manager localhost # Allow external cache managers acl ext_manager src 127.0.0.1 acl ext_manager src 192.168.10.1 acl ext_manager src http_access allow manager ext_manager http_access deny manager http_access allow purge localhost http_access deny purge http_access deny !safeports http_access deny CONNECT !sslports # Always allow localhost connections http_access allow localhost request_body_max_size 0 KB delay_pools 1 delay_class 1 2 delay_parameters 1 -1/-1 -1/-1 delay_initial_bucket_level 100 delay_access 1 allow allsrc # Reverse Proxy settings # Package Integration redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf redirector_bypass on redirect_children 3 # Custom options # Setup allowed acls # Allow local network(s) on interface(s) http_access allow localnet # Default block all to be sure http_access deny allsrc
-
Dns v4 first turned on was the latest workaround for this https issue.
Did you tried to stop and start squid after this change?
-
It's working now, i have restart the pfsense. Thanks for the help ;D
-
Hi,
short question:
Is squid3 completly ready fpr pfsense 2.1 ?
It installs and it starts on pfsense 2.1 but there are some errors on systemlog because of the wrong path "/usr/local/etc/".Further in the squid.inc there is no code to differentiate between pfsense 2.0.x and 2.1.
Did I miss something or does the squid3 package need some more work to be ready for 2.1?Thank you :-)
-
It installs and it starts on pfsense 2.1 but there are some errors on systemlog because of the wrong path "/usr/local/etc/".
I've committed a folder check right now, check and feedback :)
-
It installs and it starts on pfsense 2.1 but there are some errors on systemlog because of the wrong path "/usr/local/etc/".
I've committed a folder check right now, check and feedback :)
Your commit fixed squid3 on pfsense 2.1 and it broke squidguard on 2.1 ;-)
SquigdGuard does not make any decision on pfsense version. But it seems that only "squidguard_configurator.inc" need to be changed but not "squidguard.inc".Hmm, perhaps I should post this on this thread:
http://forum.pfsense.org/index.php/topic,50603.0.htmlFor curiosity squid3 and squidguard are marked as "working"…
Anyway - thanks for that fix on your package :)
-
Hi, I've been trying to get this package to work since it first appeared, I always get the same problem, it doesn't seem to cache any content, not in memory nor in HDD, I'm using NanoBSD, I always get TCP_MISS/200 on content that should be cached, is anyone facing this too? please help, its the only thing stopping me from using this great package, same settings work fine with squid 2.7x. I've posted about this here a couple of times but never got a reply :(
-
I didn't think you could run squid in the Nano builds. Do you have to have an extra drive for the cache?
-
yes i use a separated CF for caching
-
try to uncheck dynamic content like YouTube cache option and see if it starts caching
-
Yes! disabling dynamic content caching fixed it. Thank you marcelloc, but honestly its the only reason I want squid 3 ;D
-
It needs some improvements, current dynamic cache rules are based on squid wiki.
-
there is an unofficial package called lusca for pfsense that worked fine with caching dynamic content, but it hasn't been updated for a while, maybe taking examples from it can help? http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca Lusca is a squid 2 fork though. thanks again!
- 12 days later
-
there is an unofficial package called lusca for pfsense that worked fine with caching dynamic content, but it hasn't been updated for a while, maybe taking examples from it can help? http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca Lusca is a squid 2 fork though. thanks again!
nesense! i love you!
Thanks for reminding them about lusca!!!!!I did notice that (very sadly) lusca did not get any updates for a very long time.
In fact it's still based on an older 2.7 branch of Squid.I believe that Squid is easily the best web cache but it´s never as aggressive as Lusca which is almost the same.
It is true that the latest versions of Squid are getting more aggressive but i don´t think it´s reaching the Lusca performance.Another problem with both Squid and Lusca is that it doesn't support partial file caching so big files need to be downloaded over and over and are not cached if the is a network disconnection, something really common in unreliable wireless connections.
I noticed that worse web caches already have partial file caching while squid, being among the best, if not the best, still doesn't have this feature.
Also i don´t like how squid messes up with load balancing. I have to use 2 machines, 1 with a load balancing squid and another with a caching squid because if i do load balancing and web caching in the same machine, it will end using only 1 wan even if i set the other wan with the maximum weight. Squid always uses the default gateway. And if you have no default gateway or both are default gateways, squid begins doing weird stuff….
-
Something else that i´m not sure of is maximum object size in RAM.
Since i have a 32 Gigabyte RAM system, I dedicate at least 25 Gigs for Squid only and the rest of RAM for the PFSense system
To make sure that RAM is filled with useful objects i set the maximum object size in ram as 4 Megabytes.I´m not sure about the limits because in squid documents they always talk about minimum (0) and default which is a ridiculous (32kb) if i'm not wrong.
So i wonder if squid can properly work with a maximum object size in RAM of 4096kb.
Also, when the system is restarted, are those smaller than 4096kb objects reloaded from the Disk? or no objects is ever saved in the disk if smaller than maximum RAM object size?
I would like to have squid cache saving absolutely everything cacheable inside the disk and then after a system restart reload all cached objects smaller than maximum RAM object size back into the RAM at each restart.
Maybe squid already does something like that but if not, i think that COSS (cyclic object storage system) already does it.
Lusca can optionally support COSS but they say it's not very reliable even if sometimes it's even faster than squid's "aufs"
-
1.) To use squid in LoadBalancing /MultiWAN you just have to do additional configuration. There are many forum posts about that and how to configure that.
RAM is more a kind of "cache" so that squid does not have to access the HDD too much. After reboot the RAM will be empty. The RAM will be filled again automatically with files which are most requested by users of the proxy.
My setting an intention is as following:
The minimum file size must be 128kb. I do not want to cache smaller files because getting them from cache will ber perhaps not much faster than downloading them (depending on your WAN connection).
The maximum object size in RAM is 512kb. Caching many small files on HDD will cause many read/write accesses on HDD which cost time so I will cache small files in my RAM which is very fast.
The maximum object size is 1GB because I want to cache windows updates and so on but I do not want to have a ServicePack in my RAM which wasts much space and will not be needed so often.As memory replacement policy I use:
Heap GDSF: Greedy-Dual Size Frequency - The Heap GDSF policy optimizes object-hit rate by keeping smaller, popular objects in cache. It achieves a lower byte hit rate than LFUDA though, since it evicts larger (possibly popular) objects.
As disk cache replacement I use:
Heap LFUDA: Least Frequently Used with Dynamic Aging - The Heap LFUDA policy keeps popular objects in cache regardless of their size and thus optimizes byte hit rate at the expense of hit rate since one large, popular object will prevent many smaller, slightly less popular objects from being cached.
-
Nachtfalke
Thanks for your reply
Those settings that you are using are very efficient and are very similar to what i use. But it think they can be improved a little:
RAM is still faster than any WAN even for very small files, so i see no reason to limit the minimum object size in RAM setting it above 0KB.If you have a big hard drive you can have a maximum object size in Disk bigger than 1 GB.
I actually keep the maximum object size to 9999 Megabytes so if a user downloads a double layer dvd, it will still be there for anyone to re-download it at huge LAN speeds.
This is good when you are using a Terabyte or bigger hard drive dedicated to squid.I did notice that RAM gets filled slowly after the system is restarted, but it would be great to have it filled quickly with a synchronous cache backup done in the hard drive.
But that may be too complicated. Some other caches actually do this.Also, i saw in other forum posts here about the load balancing with squid but many are still complaining about having only one WAN used when squid service is active.
I found a post that had a way to make squid work with both WANS and it did work but it crashed squid several times a week.That is why i´m using two machines, one with pfsense with pppoe wan connections(to make wan connection faster than router), load balancing and failover but without squid,
and a second machine with pfsense and just squid cache service. - 7 days later
-
Hi,
can someone help me with that:
http://www.squid-cache.org/Doc/config/accept_filter/2012/10/30 17:27:44| Installing accept filter 'httpready' on FD 359 2012/10/30 17:27:44| SO_ACCEPTFILTER 'httpready': (2) No such file or directory
I enabled this on my squid.conf
accept_filter httpready
Is this possible with pfsense (kernel) and squid2 or squid3 ?
Thank you!
-
Another question to the squid users.
The description for http://www.squid-cache.org/Doc/config/minimum_object_size/ is:
Objects smaller than this size will NOT be saved on disk. The value is specified in kilobytes, and the default is 0 KB, which means there is no minimum.
Will the objects smaller than 128KB be cache in mem or not if I set the value to 128 ? I want to prevent that to many very small files will be cached on disk because this causes so much I/O on disk. These small files should be placed in RAM which is much faster.