Squid3 - New GUI with sync, normal and reverse proxy

[al_reidy]

@phil.davis:

Apologies to all - my little "fix" for the default language had a bit of extra cut-and-paste somehow. I noticed it soon after it got committed, but unfortunately it took quite a while for anyone with the necessary privs to commit the better version. I'll stare a lot harder at seemingly mindless fixes in future!

Its working now. Thank you all for your help.  ;D

[nutt318]

I would like to report a bug in the Squid Reverse Proxy. Under the HTTPS settings the default port used is 443 if you leave the feild empty. However if you look at squid.conf in /usr/local/etc/squid the listening port is still 80. So you have to manually put in 443 for the listening port.

Even though I found this bug I'm still having issues with getting the HTTPS reverse proxy working correctly.

I'm getting a squid error page saying Access Denied. Access control configuration prevents your request from being allowed at this time.

Not sure of the problem but it looks like others are having problems with the reverse proxy on HTTPS as well.
http://forum.pfsense.org/index.php/topic,51945.0.html

[dhatz]

Folks, is there any reason why in proxy_monitor.sh you're using a series of pipes (btw both awk seem redundant)

NUM_PROCS=`ps auxw | grep "[s]quid -D"|awk '{print $2}'| wc -l | awk '{ print $1 }'`

rather than a simpler pgrep -f "process" | wc -l ?[/s]

[podilarius]

Please re-install as I have modified the code using the pgrep a couple of days ago.

[dhatz]

@podilarius:

Please re-install as I have modified the code using the pgrep a couple of days ago.

Hi, I'm not running squid on pfsense, but I just happened to notice the commits at:

https://github.com/bsdperimeter/pfsense-packages/blob/master/config/squid3/proxy_monitor.sh
(which is the latest code afaik)

[podilarius]

sorry .. looking at the wrong line of code. That package (squid3) is for 1.2.3. The one that is on 2.1 and I think 2.0 is under the directory squid-reverse. So, you are talking about making it:

NUM_PROCS=`pgrep -f "squid -f" | wc -l | awk '{ print $1 }'`

The awk has to be in there to format the number returned properly as output without the awk yields spaces or tabs before the number returned.
or even

NUM_PROCS=`ps auxw | grep -c "[s]quid -f"'

This has only 1 pipe like your example.[/s]

[phil.davis]

@marcelloc - I corrected the conf_mount_rw and conf_mount_ro calls so that after a squid3 installation on nanobsd-like systems the filesystem ends up read-only. (Previously it was accidentally left read-write at the end of the install).

Now this message appears in /tmp/PHP_errors.log

[03-Aug-2012 04:16:47 UTC] PHP Warning:  copy(/root/2.1-BETA0.captiveportal.inc.backup): failed to open stream: Read-only file system in /usr/local/pkg/squid.inc on line 1640

This is the code:

	if (!file_exists('/root/'.$pfsense_version.'.captiveportal.inc.backup')) {
		copy ($cp_file,'/root/'.$pfsense_version.'.captiveportal.inc.backup');
		}
	if($found_rule > 0){
		file_put_contents($cp_file,$new_cp_inc, LOCK_EX);
		}

I am not sure what you want to do, perhaps put the backup in /tmp? (lost after a reboot on nanobsd), perhaps also only do the backup if a rule is found? Maybe something like this:

	if($found_rule > 0){
		if (!file_exists('/tmp/'.$pfsense_version.'.captiveportal.inc.backup')) {
			copy ($cp_file,'/tmp/'.$pfsense_version.'.captiveportal.inc.backup');
			}

		file_put_contents($cp_file,$new_cp_inc, LOCK_EX);
		}

This code is not in the older squid(2), so this problem is just in squid3.

[marcelloc]

@phil.davis:

I am not sure what you want to do, perhaps put the backup in /tmp? (lost after a reboot on nanobsd), perhaps also only do the backup if a rule is found? Maybe something like this:

well, it will need a mount_rw before this test and a mount_ro after.

[mavnezz]

Hi,

i'm using without any probs the squid3 (3.1.20 pkg 2.0.5_3) package on pfsense 2.0.1. thanks for this.

is it possible to use squid on 2 diffent ports on the same ip. for now it's '8080', additionally i want to use '3128'. 8080 should use the wan1 and 3128 should use the wan2 connection. is this in any way possible?

greets

[marcelloc]

you can try this config on custom options.

find the listening args you need and place there.

[athompso]

I don't understand what the point is of having XMLRPC sync without being able to listen on a CARP IP address.

1. can anyone tell me if there is a way to get this pkg to listen on a CARP IP (or even just listen on all interfaces!), and
2. what is an example use case for synchronized configs otherwise?

(Also see http://redmine.pfsense.org/issues/2591 and http://redmine.pfsense.org/issues/2592.)

Thanks,
-Adam

[jimp]

Unless the cache and connection state are shared between squid instances, you don't want it binding on a CARP VIP. It makes little sense to have the traffic leave from a shared IP and shared pf states when the actual connections in squid are independent, even if the config is synchronized.

The point of syncing the config is so the slave can take over, and it can, you just can't seamlessly fail over 100% in squid. Even if the states were to sync over it doesn't matter, the squid process itself would have no knowledge of the connection.

[marcelloc]

@athompso:

1. can anyone tell me if there is a way to get this pkg to listen on a CARP IP (or even just listen on all interfaces!), and

Listen squid on loopback(lo0) and create nat rules from carp address to 127.0.0.1

2.1 gui framework has the listen all as well listen on virtual ips, so squid3 on pfsense 2.1 will have this option and no need to workaround with nats.

[athompso]

@jimp:

Unless the cache and connection state are shared between squid instances, you don't want it binding on a CARP VIP. It makes little sense to have the traffic leave from a shared IP and shared pf states when the actual connections in squid are independent, even if the config is synchronized.
The point of syncing the config is so the slave can take over, and it can, you just can't seamlessly fail over 100% in squid. Even if the states were to sync over it doesn't matter, the squid process itself would have no knowledge of the connection.

I don't particularly care about seamless failover, I just want to avoid using PAC.  If I have a redundant pair of firewalls, I use the CARP VIP as my default gateway from the inside.  The way it's set up right now, I have to point all the clients at one firewall or the other's physical (LAN) IP address.  So if that firewall dies, I lose the ability to browse the web without manually reconfiguring each client…?  Whereas if squid was bound to the CARP VIP, any in-flight HTTP transactions would fail, but at least I wouldn't have to go around and reconfigure every client.  Is there some other mechanism I don't know about?

@marcelloc:

Listen squid on loopback(lo0) and create nat rules from carp address to 127.0.0.1
2.1 gui framework has the listen all as well listen on virtual ips, so squid3 on pfsense 2.1 will have this option and no need to workaround with nats.

That approach hadn't occurred to me - trying it now to see if it works, but I don't expect problems.
However, as to the second point: I am using 2.1-BETA, and I do not see any such option.  ("squid3" package, version "beta 3.1.20 pkg 2.0.5_3")

-Adam

[jimp]

NAT is indeed the easy solution there.

[marcelloc]

@athompso:

However, as to the second point: I am using 2.1-BETA, and I do not see any such option.  ("squid3" package, version "beta 3.1.20 pkg 2.0.5_3")

I said it will, I did not included the code to squid package yet  ;)

[hisoka01]

Hi, my squid cannot start :'(, here is the message in cache.log:

2012/08/28 23:09:44| Accepting  HTTP connections at [::]:3128, FD 29.
2012/08/28 23:09:44| HTCP Disabled.
2012/08/28 23:09:44| Ready to serve requests.
2012/08/28 23:09:44| WARNING: ssl_crtd #1 (FD 19) exited
2012/08/28 23:09:44| WARNING: ssl_crtd #2 (FD 21) exited
2012/08/28 23:09:44| WARNING: ssl_crtd #3 (FD 23) exited
(ssl_crtd): Uninitialized SSL certificate database directory: /var/squid/lib/ssl_db. To initialize, r
un "ssl_crtd -c -s /var/squid/lib/ssl_db".
2012/08/28 23:09:44| WARNING: ssl_crtd #4 (FD 25) exited
2012/08/28 23:09:44| Too few ssl_crtd processes are running
2012/08/28 23:09:44| storeDirWriteCleanLogs: Starting...
2012/08/28 23:09:44|   Finished.  Wrote 0 entries.
2012/08/28 23:09:44|   Took 0.00 seconds (  0.00 entries/sec).
FATAL: The ssl_crtd helpers are crashing too rapidly, need help!

Squid Cache (Version 3.1.20): Terminated abnormally.
CPU Usage: 0.092 seconds = 0.061 user + 0.031 sys
Maximum Resident Size: 10532 KB
Page faults with physical i/o: 0
(ssl_crtd): Uninitialized SSL certificate database directory: /var/squid/lib/ssl_db. To initialize, r
un "ssl_crtd -c -s /var/squid/lib/ssl_db".

I have tried both squid 2 & 3, both got the same message ::)
Also it seems there is no "ssl_crtd" exist in my system…so I cannot try to run the command :o
Do you have any suggestion..? :'(

Thanks

[marcelloc]

I have this dir on my squid3 install.

What are you trying to configure?

ls -la /var/squid/lib/
total 6
drwxr-xr-x  3 proxy  proxy  512 Apr 14 02:17 .
drwxrwxr-x  7 proxy  proxy  512 Apr 14 02:17 ..
drwxr-xr-x  2 proxy  proxy  512 Apr 14 02:17 ssl_db

[Hobby-Student]

Hey guys,

I have a strange problem regarding reverse proxy and authentification with Certificates… The package itself is working as expected and without certs needed to authenticate I can open all services (owa, msas, rpc).

Now here it comes: (all IP's, ports and url's are renamed for security reasons  ;) )

if i use this line in my squid.conf:

https_port xxx.xxx.xxx.xxx:Port cert=/website.crt key=/website.key defaultsite=this.website.me vhost

everything works as expected. I get my login page… but if I use this one:

https_port xxx.xxx.xxx.xxx:Port cert=/website.crt key=/website.key defaultsite=this.website.me clientca=/CA.crt cafile=/CA.crt capath=/ sslcontext=id vhost

I get asked for my client cert and then "page cannot be found" with this in squid logs:

clientNegotiateSSL: Error negotiating SSL connection on FD 17: error:0D0C50A1:asn1 encoding routines:ASN1_item_verify:unknown message digest algorithm (1/-1)

My CA is created with SHA512… So the clients are too ;)
When I do

openssl list-message-digest-commands

on my pfsense shell, I get

md2
md4
md5
mdc2
rmd160
sha
sha1

But with openssl speed, sha512 is listed!

What I found so far is, that somewhere in the squid source this line should be added

OpenSSL_add_all_algorithms()

somwhere like

src/ssl_support.*
function: ssl_initialize(void)

Due to the lack of time, I wasn't able to get deeper inside… Does someone has an idea?!

@marcelloc:
If this needs to be posted somewhere else, please let mo know. If I understand this package right, you are compiling squid on your own?! So this could be solved easily...

Kind regards

EDIT:
found it in src/ssl_support.cc, line 593

SSL_load_error_strings();
SSLeay_add_ssl_algorithms();

what i found by google, there should also be called: OpenSSL_add_all_algorithms()

@marcelloc:
possible?

[marcelloc]

@Hobby-Student:

@marcelloc:
If this needs to be posted somewhere else, please let mo know. If I understand this package right, you are compiling squid on your own?! So this could be solved easily…

No more  :(

Packages now must be only compiled and provided by official repo, you can compile it on 8.1 freebsd, create the package and then install on your system.

Core team are really busy these days. I'm waiting some packages to be recompiled too. After this, I can continue package development and testing.

these are current squid3 compile options available on ports

SQUID_KERB_AUTH      Install Kerberos authentication helper

SQUID_LDAP_AUTH      Install LDAP authentication helpers 

SQUID_NIS_AUTH      Install NIS/YP authentication helpers

SQUID_SASL_AUTH      Install SASL authentication helpers 

SQUID_IPV6          Enable IPv6 support                                 

SQUID_DELAY_POOLS    Enable delay pools                   

SQUID_SNMP          Enable SNMP support                   

SQUID_SSL            Enable SSL support for reverse proxies

SQUID_SSL_CRTD      Enable SSL certificate daemon         

SQUID_PINGER        Install the icmp helper               

SQUID_DNS_HELPER    Use the old 'dnsserver' helper       

SQUID_HTCP          Enable HTCP support                   

SQUID_VIA_DB        Enable forward/via database           

SQUID_CACHE_DIGESTS  Enable cache digests                 

SQUID_WCCP          Enable Web Cache Coordination Prot. v1

SQUID_WCCPV2        Enable Web Cache Coordination Prot. v2

SQUID_STRICT_HTTP    Be strictly HTTP compliant           

SQUID_IDENT          Enable ident (RFC 931) lookups       

SQUID_REFERER_LOG    Enable Referer-header logging         

SQUID_USERAGENT_LOG  Enable User-Agent-header logging     

SQUID_ARP_ACL        Enable ACLs based on ethernet address 

SQUID_IPFW          Enable transparent proxying with IPFW 

SQUID_PF            Enable transparent proxying with PF   

SQUID_IPFILTER      Enable transp. proxying with IPFilter 

SQUID_FOLLOW_XFF    Follow X-Forwarded-For headers       

SQUID_ECAP          En. loadable content adaptation modules

SQUID_ICAP          Enable ICAP client functionality     

SQUID_ESI            Enable ESI support (experimental)     

SQUID_AUFS          Enable the aufs storage scheme       

SQUID_COSS          Enable COSS (currently not available) 

SQUID_KQUEUE        Use kqueue(2) (experimental)         

SQUID_LARGEFILE      Support log and cache files >2GB     

SQUID_STACKTRACES    Create backtraces on fatal errors     

SQUID_DEBUG          Enable debugging options