Squid3 - New GUI with sync, normal and reverse proxy

Nachtfalke

@Nachtfalke:

It installs and it starts on pfsense 2.1 but there are some errors on systemlog because of the wrong path "/usr/local/etc/".

I've committed a folder check right now, check and feedback :)

Your commit fixed squid3 on pfsense 2.1 and it broke squidguard on 2.1 ;-)
SquigdGuard does not make any decision on pfsense version. But it seems that only "squidguard_configurator.inc" need to be changed but not "squidguard.inc".

Hmm, perhaps I should post this on this thread:
http://forum.pfsense.org/index.php/topic,50603.0.html

For curiosity squid3 and squidguard are marked as "working"…

Anyway - thanks for that fix on your package :)

nesense

Hi, I've been trying to get this package to work since it first appeared, I always get the same problem, it doesn't seem to cache any content, not in memory nor in HDD, I'm using NanoBSD, I always get TCP_MISS/200 on content that should be cached, is anyone facing this too? please help, its the only thing stopping me from using this great package, same settings work fine with squid 2.7x. I've posted about this here a couple of times but never got a reply :(

podilarius

I didn't think you could run squid in the Nano builds. Do you have to have an extra drive for the cache?

nesense

yes i use a separated CF for caching

marcelloc

try to uncheck dynamic content like YouTube cache option and see if it starts caching

nesense

Yes! disabling dynamic content caching fixed it. Thank you marcelloc, but honestly its the only reason I want squid 3 ;D

marcelloc

It needs some improvements, current dynamic cache rules are based on squid wiki.

nesense

there is an unofficial package called lusca for pfsense that worked fine with caching dynamic content, but it hasn't been updated for a while, maybe taking examples from it can help? http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca Lusca is a squid 2 fork though. thanks again!

quetzalcoatl

@nesense:

there is an unofficial package called lusca for pfsense that worked fine with caching dynamic content, but it hasn't been updated for a while, maybe taking examples from it can help? http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca Lusca is a squid 2 fork though. thanks again!

nesense! i love you!
Thanks for reminding them about lusca!!!!!

I did notice that (very sadly) lusca did not get any updates for a very long time.
In fact it's still based on an older 2.7 branch of Squid.

I believe that Squid is easily the best web cache but it´s never as aggressive as Lusca which is almost the same.
It is true that the latest versions of Squid are getting more aggressive but i don´t think it´s reaching the Lusca performance.

Another problem with both Squid and Lusca is that it doesn't support partial file caching so big files need to be downloaded over and over and are not cached if the is a network disconnection, something really common in unreliable wireless connections.

I noticed that worse web caches already have partial file caching while squid, being among the best, if not the best, still doesn't have this feature.

Also i don´t like how squid messes up with load balancing. I have to use 2 machines, 1 with a load balancing squid and another with a caching squid because if i do load balancing and web caching in the same machine, it will end using only 1 wan even if i set the other wan with the maximum weight. Squid always uses the default gateway. And if you have no default gateway or both are default gateways, squid begins doing weird stuff….

quetzalcoatl

Something else that i´m not sure of is maximum object size in RAM.
Since i have a 32 Gigabyte RAM system, I dedicate at least 25 Gigs for Squid only and the rest of RAM for the PFSense system
To make sure that RAM is filled with useful objects i set the maximum object size in ram as 4 Megabytes.

I´m not sure about the limits because in squid documents they always talk about minimum (0) and default which is a ridiculous (32kb) if i'm not wrong.

So i wonder if squid can properly work with a maximum object size in RAM of 4096kb.

Also, when the system is restarted, are those smaller than 4096kb objects reloaded from the Disk? or no objects is ever saved in the disk if smaller than maximum RAM object size?

I would like to have squid cache saving absolutely everything cacheable inside the disk and then after a system restart reload all cached objects smaller than maximum RAM object size back into the RAM at each restart.

Maybe squid already does something like that but if not, i think that COSS (cyclic object storage system) already does it.

Lusca can optionally support COSS but they say it's not very reliable even if sometimes it's even faster than squid's "aufs"

Nachtfalke

@quetzalcoatl

1.) To use squid in LoadBalancing /MultiWAN you just have to do additional configuration. There are many forum posts about that and how to configure that.

RAM is more a kind of "cache" so that squid does not have to access the HDD too much. After reboot the RAM will be empty. The RAM will be filled again automatically with files which are most requested by users of the proxy.

My setting an intention is as following:
The minimum file size must be 128kb. I do not want to cache smaller files because getting them from cache will ber perhaps not much faster than downloading them (depending on your WAN connection).
The maximum object size in RAM is 512kb. Caching many small files on HDD will cause many read/write accesses on HDD which cost time so I will cache small files in my RAM which is very fast.
The maximum object size is 1GB because I want to cache windows updates and so on but I do not want to have a ServicePack in my RAM which wasts much space and will not be needed so often.

As memory replacement policy I use:


Heap GDSF: Greedy-Dual Size Frequency - The Heap GDSF policy optimizes object-hit rate by keeping smaller, popular objects in cache. It achieves a lower byte hit rate than LFUDA though, since it evicts larger (possibly popular) objects.

As disk cache replacement I use:


Heap LFUDA: Least Frequently Used with Dynamic Aging - The Heap LFUDA policy keeps popular objects in cache regardless of their size and thus optimizes byte hit rate at the expense of hit rate since one large, popular object will prevent many smaller, slightly less popular objects from being cached.

quetzalcoatl

Nachtfalke

Thanks for your reply

Those settings that you are using are very efficient and are very similar to what i use. But it think they can be improved a little:
RAM is still faster than any WAN even for very small files, so i see no reason to limit the minimum object size in RAM setting it above 0KB.

If you have a big hard drive you can have a maximum object size in Disk bigger than 1 GB.
I actually keep the maximum object size to 9999 Megabytes so if a user downloads a double layer dvd, it will still be there for anyone to re-download it at huge LAN speeds.
This is good when you are using a Terabyte or bigger hard drive dedicated to squid.

I did notice that RAM gets filled slowly after the system is restarted, but it would be great to have it filled quickly with a synchronous cache backup done in the hard drive.
But that may be too complicated. Some other caches actually do this.

Also, i saw in other forum posts here about the load balancing with squid but many are still complaining about having only one WAN used when squid service is active.
I found a post that had a way to make squid work with both WANS and it did work but it crashed squid several times a week.

That is why i´m using two machines, one with pfsense with pppoe wan connections(to make wan connection faster than router), load balancing and failover but without squid,
and a second machine with pfsense and just squid cache service.

Nachtfalke

Hi,

can someone help me with that:
http://www.squid-cache.org/Doc/config/accept_filter/


2012/10/30 17:27:44| Installing accept filter 'httpready' on FD 359
2012/10/30 17:27:44| SO_ACCEPTFILTER 'httpready': (2) No such file or directory

I enabled this on my squid.conf


accept_filter httpready

Is this possible with pfsense (kernel) and squid2 or squid3 ?

Thank you!

Nachtfalke

Another question to the squid users.

The description for http://www.squid-cache.org/Doc/config/minimum_object_size/ is:


Objects smaller than this size will NOT be saved on disk.  The
value is specified in kilobytes, and the default is 0 KB, which
means there is no minimum.

Will the objects smaller than 128KB be cache in mem or not if I set the value to 128 ? I want to prevent that to many very small files will be cached on disk because this causes so much I/O on disk. These small files should be placed in RAM which is much faster.

griffin

Is there a guide somewhere to get this Reverse Proxy working in pfsense.. ?
I can't get the right combination of NAT, firewall, ports and interfaces.

Maybe an example?
Has Anyone really got it working?

thanks!

-Griffin

Truster

Hi,

i found a bug/typo in /usr/local/pkg/squid_reverse.inc:136

RPC-over-HTTPS doen't work correct.

    136                         if($settngs['reverse_owa_rpchttp'])
    137                                         array_push($owa_dirs,'rpc/rpcproxy.dll','rpcwithcert/rpcproxy.dll');

correct is:

    136                         if($settings['reverse_owa_rpchttp'])
    137                                         array_push($owa_dirs,'rpc/rpcproxy.dll','rpcwithcert/rpcproxy.dll');

tooks lot of time to get Outlook RPC-over-HTTPS working :P

marcelloc

@Truster:

i found a bug in /usr/local/pkg/squid_reverse.inc:136

I've pushed the fix for this typo, thank's for the feedback! :)

moh10ly

Could you please post your configuration ? I would like to host my webserver through squid reverse proxy and it didn't work for me..

Truster

1st: You have to correct the typo as descripted in my post before…

I configured it like this:
reverse Proxy Interface: WAN
External FQDN webmail.domain.tld
Reset TCP connections if request is unauthorized: active
Enable HTTPS reverse proxy: Active
reverse HTTPS port: 443
reverse SSL certificate
Ignore internal Certificate validation: active (we use self signed certs)
OWA frontend IP address: IP Address four your Frontent OWA Server
Enable ActiveSync: yes
Enable Outlook Anywhere: yes
Enable Exchange WebServices: yes
Enable AutoDiscover:yes

and of course, you have to open that port in the firewall rules.

i'm using OWA/Exch2010...

hope, this helps.

Edit: if you correct the typo after enabling RPC-over-HTTPS you have to uncheck that box, save the configuration, recheck the box and save it again to apply the changes.

marcelloc

@Truster:

1st: You have to correct the typo as descripted in my post before…

Or just install latest package version (2.0.5_6) ;)