Squid3 - New GUI with sync, normal and reverse proxy
-
@Marcelloc Nice work man!!! I do have a request/wish for this… Would it be possible to setup the GUI to have squid-reserve run as a separate process? This would allow it to have its own options and the log file could be separate. I created a separate conf file and added some code to the squid.inc so it would start with squid processes on my box. Basically where it starts/stop the service and creates the squid.sh file, i added another like to include my squid-reverse.conf.
just a thought when you have "free" time...
-
Setting the refresh_pattern to -1 is not a really good solution because it always downloads the file even if the user aborted it. This causes that squid downloads most of the time on its own which causes more traffic usage for squid as it saves. it is better to set some values according to the update size:
Finish transfer if less than x KB remaining: 102400 Abort transfer if more than x KB remaining: 102400 Finish transfer if more than x % finished: 60
These are the same values you can set in squid - traffic mangt.
What is happening if I enable squid windows update and set different values on the mngt tab ?Nothing, I just force range_offset_limit -1 when updates are set, all traffic mgmt are configured by users.
What do you use as refresh pattern for the windows updates ? I am using these for squid2
refresh_pattern -i .*microsoft\.com/.*\.(cab|exe|msi|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private; refresh_pattern -i .*windowsupdate\.com/.*\.(cab|exe|msi|msp) 259200 100% 259200 override-expire override-lastmod reload-into-ims ignore-reload ignore-no-cache ignore-private;
Just the suggested by wiki
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i my.windowsupdate.website.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
-
@Marcelloc Nice work man!!! I do have a request/wish for this… Would it be possible to setup the GUI to have squid-reserve run as a separate process? This would allow it to have its own options and the log file could be separate. I created a separate conf file and added some code to the squid.inc so it would start with squid processes on my box. Basically where it starts/stop the service and creates the squid.sh file, i added another like to include my squid-reverse.conf.
just a thought when you have "free" time...
Hi cino,
I'ts a good idea but I have no idea how services tab could identify these two squid processes?
-
Hi cino,
I'ts a good idea but I have no idea how services tab could identify these two squid processes?
Good point! here is the output of mine… Keep in mind when I have squid.inc, i put the full path for path conf files... if there is a shutdown, reconfigure; i included the full path to the conf in the syantx
[2.1-DEVELOPMENT][root@]/root(1): ps -aux | grep squid root 7806 0.0 0.2 10420 7120 ?? Is 7:48AM 0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid-reverse.conf proxy 7895 0.0 0.4 17596 11036 ?? S 7:48AM 0:02.72 (squid) -f /usr/local/etc/squid/squid-reverse.conf (squid) root 7953 0.0 0.2 10420 7136 ?? Is 7:48AM 0:00.00 /usr/local/sbin/squid -f /usr/local/etc/squid/squid.conf proxy 8397 0.0 0.8 35376 24892 ?? S 7:48AM 3:52.19 (squid) -f /usr/local/etc/squid/squid.conf (squid) proxy 46782 0.0 0.3 54556 8496 ?? S 7:48AM 0:03.85 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) proxy 47028 0.0 0.3 54556 8496 ?? I 7:48AM 0:00.84 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) proxy 47362 0.0 0.3 54556 8496 ?? I 7:48AM 0:00.39 (squidGuard) -c /usr/local/etc/squidGuard/squidGuard.conf (squidGuard) root 28706 0.0 0.0 3524 1256 0 S+ 10:49AM 0:00.01 grep squid
-
I'ts a good idea but I have no idea how services tab could identify these two squid processes?
Good point! here is the output of mine… Keep in mind when I have squid.inc, i put the full path for path conf files... if there is a shutdown, reconfigure; i included the full path to the conf in the syantxOk. Let's try to config it.
I did a lot of changes on squid.inc for this package. Can you try to reapply you patch on current config or show me what you did?
-
Getting this error. Did a clean pfSense install. SquidGuard won't start either.. as Squid fails to start.
Apr 16 11:22:56 php: /pkg_mgr_install.php: The command '/usr/local/sbin/squid' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept Squid Cache (Version 2.7.STABLE9): Terminated abnormally.'
Apr 16 11:22:56 squid[34066]: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept -
Getting this error. Did a clean pfSense install. SquidGuard won't start either.. as Squid fails to start.
Apr 16 11:22:56 php: /pkg_mgr_install.php: The command '/usr/local/sbin/squid' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept Squid Cache (Version 2.7.STABLE9): Terminated abnormally.'
Apr 16 11:22:56 squid[34066]: Bungled squid.conf line 7: http_port 127.0.0.1:3128 interceptYou running squid is Version 2.7.STABLE9.
What version of pfsense are you using?
Take a look on first posts of this thread to see package install sequence.
att,
Marcello Coutinho -
I'ts a good idea but I have no idea how services tab could identify these two squid processes?
Good point! here is the output of mine… Keep in mind when I have squid.inc, i put the full path for path conf files... if there is a shutdown, reconfigure; i included the full path to the conf in the syantxOk. Let's try to config it.
I did a lot of changes on squid.inc for this package. Can you try to reapply you patch on current config or show me what you did?
i sent you a pm
-
Getting this error. Did a clean pfSense install. SquidGuard won't start either.. as Squid fails to start.
Apr 16 11:22:56 php: /pkg_mgr_install.php: The command '/usr/local/sbin/squid' returned exit code '1', the output was 'FATAL: Bungled squid.conf line 7: http_port 127.0.0.1:3128 intercept Squid Cache (Version 2.7.STABLE9): Terminated abnormally.'
Apr 16 11:22:56 squid[34066]: Bungled squid.conf line 7: http_port 127.0.0.1:3128 interceptYou running squid is Version 2.7.STABLE9.
What version of pfsense are you using?
Take a look on first posts of this thread to see package install sequence.
att,
Marcello CoutinhoI clean installed this version
2.0.1-RELEASE (amd64)
built on Mon Dec 12 18:16:13 EST 2011Then went into packages and installed squid3 first. Same settings I have been using for over a year. Nothing has changed. I reinstalled pfSense again and again tried with your latest package.. same issue.
-
Hi guys,
I'm testing new squid3 package, and after install it, I'm having a lot errors in http connections, squid show me a lot 'TCP_MISS/503'. This happen often in forms posts, so I need re-send form ou press F5.
I tested exhaustively the squid-2.7.9_1 + squidGuard and problem no happen. So I too tested exhaustively the squid3 + SquidGuard, and I give this problem.All squid versions have the same config. And this problem only occurs in 'Transparent Mode'
Somebdoy can please test it and report the results?!
Thanks
Hi,
so I post what I did and while I am doing this it will take more than one minute. (Remember your pm to me).
I installed squid3 package and sent myself personal messages. It took all times very long till they get sent - but that's probably a forum issue. Nothing uncommon in access.log.
After that installed squidguard - it break squid3 and squidguard so I uninstalled squid3 and reinstalled squid3. after that both were running. I created a target in squidguard to block google.de and it is working. Other pages can be visited. Nothing uncommon and not TCP_MISS/503 in access.log
I sent some personal messages myself and no problem.
Now I am writing this post and we will see what happens.
PS: I did not enable any additional options on squid - just basic settings on a VM to test.
–-- EDIT ----
Got the same error as ccesario:
This is after writing the post:1334604903.140 56 192.168.0.112 TCP_MISS/503 4769 POST http://forum.pfsense.org/index.php? - DIRECT/forum.pfsense.org text/html 1334604903.969 659 192.168.0.112 TCP_MISS/200 13148 GET http://www.squid-cache.org/Artwork/SN.png - DIRECT/209.169.10.131 image/png
My brwoser showed the attached screenshot.
After that I pressed F5 and re-sent:
1334605018.876 60599 192.168.0.112 TCP_MISS/302 580 POST http://forum.pfsense.org/index.php? - DIRECT/69.64.6.7 text/html 1334605019.308 428 192.168.0.112 TCP_MISS/200 12060 GET http://forum.pfsense.org/index.php/board,15.0.html - DIRECT/69.64.6.7 text/html 1334605019.409 154 192.168.0.112 TCP_MISS/304 260 GET http://www.google-analytics.com/urchin.js - DIRECT/173.194.35.39 - 1334605019.530 307 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/style.css? - DIRECT/69.64.6.7 - 1334605019.542 158 192.168.0.112 TCP_MISS/304 258 GET http://pagead2.googlesyndication.com/pagead/show_ads.js - DIRECT/209.85.148.157 - 1334605019.546 319 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/default/print.css? - DIRECT/69.64.6.7 - 1334605019.561 332 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/transparency.gif - DIRECT/69.64.6.7 - 1334605019.581 352 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/folder_open.gif - DIRECT/69.64.6.7 - 1334605019.600 370 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/linktree_side.gif - DIRECT/69.64.6.7 - 1334605019.612 396 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/default/script.js? - DIRECT/69.64.6.7 - 1334605019.693 162 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/default/xml_board.js - DIRECT/69.64.6.7 - 1334605019.710 162 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/rss.gif - DIRECT/69.64.6.7 - 1334605019.729 166 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/filter.gif - DIRECT/69.64.6.7 - 1334605019.747 163 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/pfsense_banner_applianceshop.png - DIRECT/69.64.6.7 - 1334605019.765 163 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/sort_down.gif - DIRECT/69.64.6.7 - 1334605019.781 168 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/veryhot_post.gif - DIRECT/69.64.6.7 - 1334605019.858 164 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/post/xx.gif - DIRECT/69.64.6.7 - 1334605019.874 163 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/show_sticky.gif - DIRECT/69.64.6.7 - 1334605019.894 163 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/last_post.gif - DIRECT/69.64.6.7 - 1334605019.917 169 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/quick_lock.gif - DIRECT/69.64.6.7 - 1334605019.930 164 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/hot_post.gif - DIRECT/69.64.6.7 - 1334605019.947 165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/normal_post.gif - DIRECT/69.64.6.7 - 1334605019.968 68 192.168.0.112 TCP_MISS/200 500 GET http://www.google-analytics.com/__utm.gif? - DIRECT/173.194.35.39 image/gif 1334605020.024 165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/post/wink.gif - DIRECT/69.64.6.7 - 1334605020.037 163 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/my_veryhot_post.gif - DIRECT/69.64.6.7 - 1334605020.059 165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/post/thumbup.gif - DIRECT/69.64.6.7 - 1334605020.086 169 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/normal_poll.gif - DIRECT/69.64.6.7 - 1334605020.102 169 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/my_normal_post.gif - DIRECT/69.64.6.7 - 1334605020.115 167 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/post/question.gif - DIRECT/69.64.6.7 - 1334605020.191 167 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/quick_sticky.gif - DIRECT/69.64.6.7 - 1334605020.204 167 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/bg_body.gif - DIRECT/69.64.6.7 - 1334605020.225 164 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/images/logo.jpg - DIRECT/69.64.6.7 - 1334605020.251 164 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/coltitle_bg.gif - DIRECT/69.64.6.7 - 1334605020.269 166 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/mirrortab_first.gif - DIRECT/69.64.6.7 - 1334605020.357 164 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/mirrortab_last.gif - DIRECT/69.64.6.7 - 1334605020.374 168 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/images/catbg.jpg - DIRECT/69.64.6.7 - 1334605020.389 164 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/maintab_first.gif - DIRECT/69.64.6.7 - 1334605020.417 164 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/maintab_back.gif - DIRECT/69.64.6.7 - 1334605020.436 165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/maintab_last.gif - DIRECT/69.64.6.7 - 1334605020.523 164 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/images/titlebg.jpg - DIRECT/69.64.6.7 - 1334605020.553 303 192.168.0.112 TCP_MISS/200 2672 GET http://googleads.g.doubleclick.net/pagead/ads? - DIRECT/209.85.148.155 text/html 1334605020.806 690 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/mirrortab_back.gif - DIRECT/69.64.6.7 -
-
When enabling all cache options (window supdates and so on) the squid.conf is not correctly formatted and needs some new lines before "range offset limit":
range_offset_limit -1 refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims refresh_pattern -i my.windowsupdate.website.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-imsrange_offset_limit -1 refresh_pattern liveupdate.symantecliveupdate.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims refresh_pattern symantecliveupdate.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 reload-into-imsrange_offset_limit -1 refresh_pattern avast.com/.*\.(vpu|cab|stamp|exe) 10080 100% 43200 reload-into-imsrange_offset_limit -1 refresh_pattern personal.avira-update.com/.*\.(cab|exe|dll|msi|gz) 10080 100% 43200 reload-into-imscache_mem 64 MB maximum_object_size_in_memory 256 KB
Further I would make the other pattern case insensitive, too ( -i )
An what about the subdomains of microsoft.com ? Are they covered with this regex ?
Or better put .* in front like:refresh_pattern -i .*\.microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) refresh_pattern -i .*\.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip)
Further I didn't have any luck with a short test on caching youtube.com videos.
access.log shows "x-flv". Perhaps add this format to the config:refresh_pattern -i .*\.(x-flv|flv) 10080 90% 999999 ignore-no-cache override-expire ignore-private
-
Thanks, I'll fix it.
-
On squid -> cache this:
set Maximum download size on 'traffic mgmt' squid tab to a value that fits patters your are applying. Microsoft may need 200Mb and youtube 4GB.
should be probably renamed to:
set Maximum object size on 'cache' squid tab to a value that fits pattern your are applying. Microsoft may need 200Mb and youtube 4GB.
Question:
Could you add an option to change the time an object should be in cache ?
At the moment it is 4320 80% 43200. Perhaps someone likes to increase that.But probably if someone needs this he should create his custom options itself and the "click and save" GUI ist just for people who do not want to do to much work on squid and refresh_pattern :-)
-
On squid -> cache this:
set Maximum download size on 'traffic mgmt' squid tab to a value that fits patters your are applying. Microsoft may need 200Mb and youtube 4GB.
should be probably renamed to:
set Maximum object size on 'cache' squid tab to a value that fits pattern your are applying. Microsoft may need 200Mb and youtube 4GB.
The Maximum download size is on 'traffic mgmt' tab
But probably if someone needs this he should create his custom options itself and the "click and save" GUI ist just for people who do not want to do to much work on squid and refresh_pattern :-)
I think the same way :)
-
The Maximum download size is on 'traffic mgmt' tab
This will limit all downloads through squid or am I completly wrong !?! So if I set 200MB there and will try to download an 3GB ISO it will cut my download, isn't it ?
Damn…squid has so many options it is sometime really hard to understand when to use what ;)
-
Damn…squid has so many options it is sometime really hard to understand when to use what ;)
I second that :)
-
PS: I did not enable any additional options on squid - just basic settings on a VM to test.
My brwoser showed the attached screenshot.
After that I pressed F5 and re-sent:
Hi Nachtfalke, thank you by feedback!
This is the problem that happen! Exactly as your screenshot.
I have this screen in others sites too. I mean to you pfsense forum only to test/reproduce.
But in squid-2.7.9 this not happen.
PS: I too enable basic settings in squid.
Welll…. this can be considered a bug/error ?
-
PS: I did not enable any additional options on squid - just basic settings on a VM to test.
My brwoser showed the attached screenshot.
After that I pressed F5 and re-sent:
Hi Nachtfalke, thank you by feedback!
This is the problem that happen! Exactly as your screenshot.
I have this screen in others sites too. I mean to you pfsense forum only to test/reproduce.
But in squid-2.7.9 this not happen.
PS: I too enable basic settings in squid.
Welll…. this can be considered a bug/error ?
So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.
Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)
-
So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.
Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)
Hehehehh no, I don't have URL to can "spam" posts. But using pfsense forum its possible.
Edit your posts and save-it :) … I my tests I usage this to reproduce many times the error :)
Thanks
-
So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.
Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)
Hehehehh no, I don't have URL to can "spam" posts. But using pfsense forum its possible.
Edit your posts and save-it :) … I my tests I usage this to reproduce many times the error :)
Thanks
Will do this perhaps this afternoon/night when I am at home. Perhaps we can tweak something if it's not a bug. :-)