Squid3 - New GUI with sync, normal and reverse proxy

marcelloc

Hi all,

I've merged squid-rever and squid3 in only one package for pfsense 2.0 with reverse options in a brand new service-> reverse proxy menu as well XMLRPC sync options.

Before package install/reinstall/upgrade, backup you config(just in case) especially reverse proxy config.

I've tested 02 days without issues. :)

att,
Marcello Coutinho

Donny

Hello, I have some question. The screenshot above are reference to use for Web Server. For environment network without web server, Can I use its for regular normal with Sarg, Squid proxy and Dansguardian?
(at Reverse Proxy server: General > General tab) I am a little bit confused.

Thank u

marcelloc

For normal proxy use proxy server menu. Reverse proxy is just to publish your web servers to internet.

Matthias

Crashes if I try to access either the Services > Proxy Server or Services > Reverse Proxy with this error:

Warning: dir(/usr/local/etc/squid/errors/): failed to open dir: No such file or directory in /etc/inc/pfsense-utils.inc on line 432 Fatal error: Call to a member function read() on a non-object in /etc/inc/pfsense-utils.inc on line 433

marcelloc

What pfsense version are you using?

Pfsense util calls are the same from squid2

Matthias

2.1-DEVELOPMENT (i386)
built on Fri Apr 13 21:32:08 EDT 2012
FreeBSD 8.3-RELEASE

marcelloc

@Matthias:

2.1-DEVELOPMENT (i386)
built on Fri Apr 13 21:32:08 EDT 2012
FreeBSD 8.3-RELEASE

I've tested right now with

2.1-DEVELOPMENT (amd64)
built on Fri Apr 13 16:24:04 EDT 2012
FreeBSD 8.3-RELEASE

with no issues.

Squid 3 do not have pfsense 2.1 pbi packages yet, so you need to manual install squid3.

i386
pkg_add -rf http://files.pfsense.org/packages/8/All/squid-3.1.19.tbz

amd64
pkg_add -rf http://files.pfsense.org/packages/amd64/8/All/squid-3.1.19.tbz

Matthias

Seems to be working so far.

Nachtfalke

hi marcelloc,

nice to see that there is now just only one package and not two like it was with squid2 and squid-reverse.
If I am not completely wrong - you made some changes on the GUI (re-order some options), right ? But you didn't add any relevant new options?

Another question is:

I am using squid2 at the moment with squidguard and many custom options.
If I update to squid3 - should I pay attention on the custom options ? In the new GUI there are two text boxes - one for custom options - and another one for squidguard / havp options. Will they be "imported" correctly when updating?

PS: Not really related to this topic - but will there be a dansguardian version which uses squid3 or better not forces any version of squid ?

Thank you for the very hard work - on all the many different packages :-)

Donny

Hello all, I just clean install pfSense (i386) and also I have installed Squid3 and Dansguardian. After that I reboot pfsense system. I got some warning on pfSense console like this:

Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 103
Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 146

There is some bug in Squid3 (squid_reverse.inc) on the line 103 and 146.
Again for Dansguardian does not appear on services menu (Services > …....),after I installed. I have to reinstall it and then appear on the services menu.

Also Squid and Perl have two version installed. Uninstall and reinstall, it is the same.

Any idea.

harish

me too having this issue

Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 103
Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 146

and squidguard service does not startup in my case

marcelloc

@Nachtfalke:

nice to see that there is now just only one package and not two like it was with squid2 and squid-reverse.
If I am not completely wrong - you made some changes on the GUI (re-order some options), right ? But you didn't add any relevant new options?

There are new options just on reverse menu. Instead of text config, squid-reverse has config screens for peers and mappings.

@Nachtfalke:

I am using squid2 at the moment with squidguard and many custom options.
If I update to squid3 - should I pay attention on the custom options ?In the new GUI there are two text boxes - one for custom options - and another one for squidguard / havp options. Will they be "imported" correctly when updating?

They will stay all on Integration field.
After squid3 install, you can move your options from integration field to custom fields using a better viewing one per line option.
example:
integration field:```
auth_param ntlm program /usr/lib/squid/ntlm_auth --helper-protocol=squid-2.5-ntlmssp;auth_param basic program /usr/lib/squid/ntlm_auth --helper-protocol=squid-2.5-basic;auth_param basic children 5;auth_param basic realm Squid;proxy-caching web server;auth_param basic credentialsttl 2 hours


can be moved to:

#ntlm auth
auth_param ntlm program /usr/lib/squid/ntlm_auth --helper-protocol=squid-2.5-ntlmssp
auth_param basic program /usr/lib/squid/ntlm_auth --helper-protocol=squid-2.5-basic
auth_param basic children 5
auth_param basic realm Squid proxy-caching web server
auth_param basic credentialsttl 2 hours

on custom field

@Nachtfalke:

> PS: Not really related to this topic - but will there be a dansguardian version which uses squid3 or better not forces any version of squid ?

You can use this if you remeber to install squid3 after dansguardian or squidguard.

marcelloc

@harish:

Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 103
Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 146

and squidguard service does not startup in my case

~~I'll check it today.~~

~~Try to apply squidguard config again and then re-apply squid config.

To workaround squid-reverse error, just select a interface on in and fill up host fqdn. It will not enable reverse proxy but will create xml config that stops inc errors at line 103 and 146.~~

I've included some checks on squid-reverse.inc file. I'm just doing some tests before publishing this patch.

Thanks for your feedback.

marcelloc

@Donny:

Also Squid and Perl have two version installed. Uninstall and reinstall, it is the same.

Squidguard as well dansguardian force squid2 install.

To avoid squid3 overwrite, install squid3 package after squidguard or dansguardian.

marcelloc

I've just pushed squid_reverse.inc fix.

Upgrade to squid3 pkg v 2.0.1 and see if it fixes inc errors.

Nachtfalke

I installed squid2 package
after that squidguard
and then squid3

when click on "save" on squidguard page this line appears in squid3 integration box:

çbç-¦º ©¿ºÊÿ–‡—öâŸû*º'F¹ªÝsû¬¯ùhq©z×?²«¢tkšßìªèæ«uÊ'~·Š·œ¶ŠÛÊ–¬²‰ëyØ«yË\†)]é÷

Tikimotel

To "temporary" circumvent the integration gibberish, manually edit the custom options.
Integrations

(empty the edit box)

Custom
Options

quick_abort_pct 70
range_offset_limit 0
redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass on
redirect_children 8

(press save)

Squid should restart and activate 8 squidquard redirectors (temporary fix, because changing any setting in the proxy filter menu's will result in gibberish again)
Might also change at midnight because of squidguard crontab stuff.

Nachtfalke

@Tikimotel:

This helped and squidguard started (service). Couldn't test more.

redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass on
redirect_children 8

Donny

@marcelloc:

I've just pushed squid_reverse.inc fix.

Upgrade to squid3 pkg v 2.0.1 and see if it fixes inc errors.

SOLVED!, The Squid3 pkg v 2.0.1 has fixed this Waring: Invalid argument supplied for foreach() in /usr/local/pkg/squid_reverse.inc on line 103 and 146.

Next step I just want to be sure, I will try to clean install pfSense again in my testing machine, after that > First install: Dansguardian > Second install: Squid 3.

I will inform you later, Thank u Marcelloc.

marcelloc

@Nachtfalke:

I installed squid2 package
after that squidguard
and then squid3

when click on "save" on squidguard page this line appears in squid3 integration box:
çbç-¦º ©¿ºÊÿ–‡—öâŸû*º'F¹ªÝsû¬¯ùhq©z×?²«¢tkšßìªèæ«uÊ'~·Š·œ¶ŠÛÊ–¬²‰ëyØ«yË\†)]é÷

Check if I forgot to remove base64 info from custom_option on squid.XML

Custom_option should not have it but custom_option_squid3 should have.

You do not need squid2 package before squidguard.

I'm not at home right now so I could check this only tonight.

marcelloc

@Donny:

Next step I just want to be sure, I will try to clean install pfSense again in my testing machine, after that > First install: Dansguardian > Second install: Squid 3.

Yes :)

Nachtfalke

@marcelloc:

@Nachtfalke:
I installed squid2 package
after that squidguard
and then squid3

when click on "save" on squidguard page this line appears in squid3 integration box:
çbç-¦º ©¿ºÊÿ–‡—öâŸû*º'F¹ªÝsû¬¯ùhq©z×?²«¢tkšßìªèæ«uÊ'~·Š·œ¶ŠÛÊ–¬²‰ëyØ«yË\†)]é÷
Check if I forgot to remove base64 info from custom_option on squid.XML

Custom_option should not have it but custom_option_squid3 should have.

You do not need squid2 package before squidguard.

I'm not at home right now so I could check this only tonight.

Removing the "encode base64" from squid.xml worked. Now the command is visible in the text box BUT the command from this box is not copied into squid.conf file. So it does not take effect.

Don't hurry up and don't stress with that fact. It is sunday and you should have a free day and a nice weekend, too :-)

ccesario

Hi guys,

I'm testing new squid3 package, and after install it, I'm having a lot errors in http connections, squid show me a lot 'TCP_MISS/503'. This happen often in forms posts, so I need re-send form ou press F5.
I tested exhaustively the squid-2.7.9_1 + squidGuard and problem no happen. So I too tested exhaustively the squid3 + SquidGuard, and I give this problem.

All squid versions have the same config. And this problem only occurs in 'Transparent Mode'

Somebdoy can please test it and report the results?!

Thanks

mhab12

Just in case others were seeing performance issues, I saw my bandwidth drop to <5mbps after installing Squid3, however changing from AUFS to diskd brought the bandwidth backup up to approximately 60mbps where it should be.

Donny

@marcelloc:

@Donny:

Next step I just want to be sure, I will try to clean install pfSense again in my testing machine, after that > First install: Dansguardian > Second install: Squid 3.

Yes :)

After clean install pfSense, I try first to install Dansguardian. I got the same result as I told you before. Dansguardian does not appear on services menu. So I wait a few minute and then try to refresh pfSense WebGUI and not thing changed. The last final "fantasy" I reboot pfSense and it does not appear again. (The final "fantasy" I just only make a joke because today is Sunday, you should be relax.). Then the way I have to do before I am going to install Squid3 is reinstall Dansguardian and finally Dansquardian is appear.

The next step I am going to install Squid3

Just let you know, Marcelloc.

marcelloc

Before using disk cache, I suggest you to enable softupdates on /usr and /var. The performance difference is huge.

Donny

Now Squid3 and Dansguardian is working. I don't find any error yet. The next step I will trying to configure firewall, NAT with HTTP and HTTPS for how Squid3 and Dansguardian work together.

harish

error is gone but could not start squidguard, i rechecked with reinstalling the squidguard, but fails to start.

Nachtfalke

@harish:

error is gone but could not start squidguard, i rechecked with reinstalling the squidguard, but fails to start.

As far as I can say that at the moment the "Integrations" box isn't working. So put the commands squidguard creates manually in "custom options":

redirect_program /usr/local/bin/squidGuard -c /usr/local/etc/squidGuard/squidGuard.conf
redirector_bypass on
redirect_children 8

marcelloc

Just updated squid3 package to version 2.0.2 to fix integration erros.

Please update,test and feedback :)

harish

yes now its working after custom option.

marcelloc

Hi all,

After looking for some options o squid-wiki, I've included dynamic update options to cache tab on pkg v 2.0.3

Nachtfalke

@marcelloc:

Hi all,

After looking for some options o squid-wiki, I've included dynamic update options to cache tab on pkg v 2.0.3

Setting the refresh_pattern to -1 is not a really good solution because it always downloads the file even if the user aborted it. This causes that squid downloads most of the time on its own which causes more traffic usage for squid as it saves. it is better to set some values according to the update size:


Finish transfer if less than x KB remaining: 102400
Abort transfer if more than x KB remaining: 102400
Finish transfer if more than x % finished: 60

These are the same values you can set in squid - traffic mangt.
What is happening if I enable squid windows update and set different values on the mngt tab ?

What do you use as refresh pattern for the windows updates ? I am using these for squid2


refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;

Thanks :-)

Cino

@Marcelloc Nice work man!!! I do have a request/wish for this… Would it be possible to setup the GUI to have squid-reserve run as a separate process? This would allow it to have its own options and the log file could be separate. I created a separate conf file and added some code to the squid.inc so it would start with squid processes on my box. Basically where it starts/stop the service and creates the squid.sh file, i added another like to include my squid-reverse.conf.

just a thought when you have "free" time...

marcelloc

@Nachtfalke:

Setting the refresh_pattern to -1 is not a really good solution because it always downloads the file even if the user aborted it. This causes that squid downloads most of the time on its own which causes more traffic usage for squid as it saves. it is better to set some values according to the update size:
Finish transfer if less than x KB remaining: 102400
Abort transfer if more than x KB remaining: 102400
Finish transfer if more than x % finished: 60
These are the same values you can set in squid - traffic mangt.
What is happening if I enable squid windows update and set different values on the mngt tab ?

Nothing, I just force range_offset_limit -1 when updates are set, all traffic mgmt are configured by users.

@Nachtfalke:

What do you use as refresh pattern for the windows updates ? I am using these for squid2


refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;

Just the suggested by wiki

refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i my.windowsupdate.website.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims

marcelloc

@Cino:

@Marcelloc Nice work man!!! I do have a request/wish for this… Would it be possible to setup the GUI to have squid-reserve run as a separate process? This would allow it to have its own options and the log file could be separate. I created a separate conf file and added some code to the squid.inc so it would start with squid processes on my box. Basically where it starts/stop the service and creates the squid.sh file, i added another like to include my squid-reverse.conf.

just a thought when you have "free" time...

Hi cino,

I'ts a good idea but I have no idea how services tab could identify these two squid processes?

Cino

@marcelloc:

Hi cino,

I'ts a good idea but I have no idea how services tab could identify these two squid processes?

Good point! here is the output of mine… Keep in mind when I have squid.inc, i put the full path for path conf files... if there is a shutdown, reconfigure; i included the full path to the conf in the syantx


[2.1-DEVELOPMENT][root@]/root(1): ps -aux | grep squid
root    7806  0.0  0.2 10420  7120  ??  Is    7:48AM   0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid-reverse.conf
proxy   7895  0.0  0.4 17596 11036  ??  S     7:48AM   0:02.72 (squid) -f /usr/local/etc/squid/squid-reverse.conf (squid)
root    7953  0.0  0.2 10420  7136  ??  Is    7:48AM   0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf
proxy   8397  0.0  0.8 35376 24892  ??  S     7:48AM   3:52.19 (squid) -f /usr/local/etc/squid/squid.conf (squid)
proxy  46782  0.0  0.3 54556  8496  ??  S     7:48AM   0:03.85 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  47028  0.0  0.3 54556  8496  ??  I     7:48AM   0:00.84 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
proxy  47362  0.0  0.3 54556  8496  ??  I     7:48AM   0:00.39 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard)
root   28706  0.0  0.0  3524  1256   0  S+   10:49AM   0:00.01 grep squid

marcelloc

@Cino:

I'ts a good idea but I have no idea how services tab could identify these two squid processes?
Good point! here is the output of mine… Keep in mind when I have squid.inc, i put the full path for path conf files... if there is a shutdown, reconfigure; i included the full path to the conf in the syantx

Ok. Let's try to config it.

I did a lot of changes on squid.inc for this package. Can you try to reapply you patch on current config or show me what you did?

asterix

Getting this error. Did a clean pfSense install. SquidGuard won't start either.. as Squid fails to start.

Apr 16 11:22:56 php: /pkg_mgr_install.php: The command '/usr/local/sbin/squid' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept Squid Cache (Version 2.7.STABLE9): Terminated abnormally.'
Apr 16 11:22:56 squid[34066]: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept

marcelloc

@asterix:

Getting this error. Did a clean pfSense install. SquidGuard won't start either.. as Squid fails to start.

Apr 16 11:22:56 php: /pkg_mgr_install.php: The command '/usr/local/sbin/squid' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept Squid Cache (Version 2.7.STABLE9): Terminated abnormally.'
Apr 16 11:22:56 squid[34066]: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept

You running squid is Version 2.7.STABLE9.

What version of pfsense are you using?

Take a look on first posts of this thread to see package install sequence.

att,
Marcello Coutinho