Squid3 - New GUI with sync, normal and reverse proxy

Nachtfalke

Hi guys,

I'm testing new squid3 package, and after install it, I'm having a lot errors in http connections, squid show me a lot 'TCP_MISS/503'. This happen often in forms posts, so I need re-send form ou press F5.
I tested exhaustively the squid-2.7.9_1 + squidGuard and problem no happen. So I too tested exhaustively the squid3 + SquidGuard, and I give this problem.

All squid versions have the same config. And this problem only occurs in 'Transparent Mode'

Somebdoy can please test it and report the results?!

Thanks

Hi,

so I post what I did and while I am doing this it will take more than one minute. (Remember your pm to me).

I installed squid3 package and sent myself personal messages. It took all times very long till they get sent - but that's probably a forum issue. Nothing uncommon in access.log.

After that installed squidguard - it break squid3 and squidguard so I uninstalled squid3 and reinstalled squid3. after that both were running. I created a target in squidguard to block google.de and it is working. Other pages can be visited. Nothing uncommon and not TCP_MISS/503 in access.log

I sent some personal messages myself and no problem.

Now I am writing this post and we will see what happens.

PS: I did not enable any additional options on squid - just basic settings on a VM to test.

–-- EDIT ----
Got the same error as ccesario:
This is after writing the post:

1334604903.140     56 192.168.0.112 TCP_MISS/503 4769 POST http://forum.pfsense.org/index.php? - DIRECT/forum.pfsense.org text/html
1334604903.969    659 192.168.0.112 TCP_MISS/200 13148 GET http://www.squid-cache.org/Artwork/SN.png - DIRECT/209.169.10.131 image/png

My brwoser showed the attached screenshot.

After that I pressed F5 and re-sent:


1334605018.876  60599 192.168.0.112 TCP_MISS/302 580 POST http://forum.pfsense.org/index.php? - DIRECT/69.64.6.7 text/html
1334605019.308    428 192.168.0.112 TCP_MISS/200 12060 GET http://forum.pfsense.org/index.php/board,15.0.html - DIRECT/69.64.6.7 text/html
1334605019.409    154 192.168.0.112 TCP_MISS/304 260 GET http://www.google-analytics.com/urchin.js - DIRECT/173.194.35.39 -
1334605019.530    307 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/style.css? - DIRECT/69.64.6.7 -
1334605019.542    158 192.168.0.112 TCP_MISS/304 258 GET http://pagead2.googlesyndication.com/pagead/show_ads.js - DIRECT/209.85.148.157 -
1334605019.546    319 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/default/print.css? - DIRECT/69.64.6.7 -
1334605019.561    332 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/transparency.gif - DIRECT/69.64.6.7 -
1334605019.581    352 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/folder_open.gif - DIRECT/69.64.6.7 -
1334605019.600    370 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/linktree_side.gif - DIRECT/69.64.6.7 -
1334605019.612    396 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/default/script.js? - DIRECT/69.64.6.7 -
1334605019.693    162 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/default/xml_board.js - DIRECT/69.64.6.7 -
1334605019.710    162 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/rss.gif - DIRECT/69.64.6.7 -
1334605019.729    166 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/filter.gif - DIRECT/69.64.6.7 -
1334605019.747    163 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/pfsense_banner_applianceshop.png - DIRECT/69.64.6.7 -
1334605019.765    163 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/sort_down.gif - DIRECT/69.64.6.7 -
1334605019.781    168 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/veryhot_post.gif - DIRECT/69.64.6.7 -
1334605019.858    164 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/post/xx.gif - DIRECT/69.64.6.7 -
1334605019.874    163 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/show_sticky.gif - DIRECT/69.64.6.7 -
1334605019.894    163 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/last_post.gif - DIRECT/69.64.6.7 -
1334605019.917    169 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/quick_lock.gif - DIRECT/69.64.6.7 -
1334605019.930    164 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/hot_post.gif - DIRECT/69.64.6.7 -
1334605019.947    165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/normal_post.gif - DIRECT/69.64.6.7 -
1334605019.968     68 192.168.0.112 TCP_MISS/200 500 GET http://www.google-analytics.com/__utm.gif? - DIRECT/173.194.35.39 image/gif
1334605020.024    165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/post/wink.gif - DIRECT/69.64.6.7 -
1334605020.037    163 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/my_veryhot_post.gif - DIRECT/69.64.6.7 -
1334605020.059    165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/post/thumbup.gif - DIRECT/69.64.6.7 -
1334605020.086    169 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/normal_poll.gif - DIRECT/69.64.6.7 -
1334605020.102    169 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/topic/my_normal_post.gif - DIRECT/69.64.6.7 -
1334605020.115    167 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/post/question.gif - DIRECT/69.64.6.7 -
1334605020.191    167 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/icons/quick_sticky.gif - DIRECT/69.64.6.7 -
1334605020.204    167 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/bg_body.gif - DIRECT/69.64.6.7 -
1334605020.225    164 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/images/logo.jpg - DIRECT/69.64.6.7 -
1334605020.251    164 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/coltitle_bg.gif - DIRECT/69.64.6.7 -
1334605020.269    166 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/mirrortab_first.gif - DIRECT/69.64.6.7 -
1334605020.357    164 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/mirrortab_last.gif - DIRECT/69.64.6.7 -
1334605020.374    168 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/images/catbg.jpg - DIRECT/69.64.6.7 -
1334605020.389    164 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/maintab_first.gif - DIRECT/69.64.6.7 -
1334605020.417    164 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/maintab_back.gif - DIRECT/69.64.6.7 -
1334605020.436    165 192.168.0.112 TCP_MISS/304 322 GET http://forum.pfsense.org/Themes/slickprographite/images/maintab_last.gif - DIRECT/69.64.6.7 -
1334605020.523    164 192.168.0.112 TCP_MISS/304 323 GET http://forum.pfsense.org/Themes/slickprographite/images/titlebg.jpg - DIRECT/69.64.6.7 -
1334605020.553    303 192.168.0.112 TCP_MISS/200 2672 GET http://googleads.g.doubleclick.net/pagead/ads? - DIRECT/209.85.148.155 text/html
1334605020.806    690 192.168.0.112 TCP_MISS/304 321 GET http://forum.pfsense.org/Themes/slickprographite/images/mirrortab_back.gif - DIRECT/69.64.6.7 -

First_post.JPG_thumb

Nachtfalke

When enabling all cache options (window supdates and so on) the squid.conf is not correctly formatted and needs some new lines before "range offset limit":

range_offset_limit -1
refresh_pattern -i microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i my.windowsupdate.website.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-imsrange_offset_limit -1
refresh_pattern liveupdate.symantecliveupdate.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims
refresh_pattern symantecliveupdate.com/.*\.(cab|exe|dll|msi) 10080 100% 43200 reload-into-imsrange_offset_limit -1
refresh_pattern avast.com/.*\.(vpu|cab|stamp|exe) 10080 100% 43200 reload-into-imsrange_offset_limit -1
refresh_pattern personal.avira-update.com/.*\.(cab|exe|dll|msi|gz) 10080 100% 43200 reload-into-imscache_mem 64 MB
maximum_object_size_in_memory 256 KB

Further I would make the other pattern case insensitive, too ( -i )

An what about the subdomains of microsoft.com ? Are they covered with this regex ?
Or better put .* in front like:


refresh_pattern -i .*\.microsoft.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip)
refresh_pattern -i .*\.windowsupdate.com/.*\.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip)

Further I didn't have any luck with a short test on caching youtube.com videos.
access.log shows "x-flv". Perhaps add this format to the config:

refresh_pattern -i .*\.(x-flv|flv) 10080 90% 999999 ignore-no-cache override-expire ignore-private

marcelloc

Thanks, I'll fix it.

Nachtfalke

On squid -> cache this:

set Maximum download size on 'traffic mgmt' squid tab to a value that fits patters your are applying.
Microsoft may need 200Mb and youtube 4GB.

should be probably renamed to:


set Maximum object size on 'cache' squid tab to a value that fits pattern your are applying.
Microsoft may need 200Mb and youtube 4GB.

Question:
Could you add an option to change the time an object should be in cache ?
At the moment it is 4320 80% 43200. Perhaps someone likes to increase that.

But probably if someone needs this he should create his custom options itself and the "click and save" GUI ist just for people who do not want to do to much work on squid and refresh_pattern :-)

marcelloc

@Nachtfalke:

On squid -> cache this:

set Maximum download size on 'traffic mgmt' squid tab to a value that fits patters your are applying.
Microsoft may need 200Mb and youtube 4GB.

should be probably renamed to:


set Maximum object size on 'cache' squid tab to a value that fits pattern your are applying.
Microsoft may need 200Mb and youtube 4GB.

The Maximum download size is on 'traffic mgmt' tab

@Nachtfalke:

But probably if someone needs this he should create his custom options itself and the "click and save" GUI ist just for people who do not want to do to much work on squid and refresh_pattern :-)

I think the same way :)

Nachtfalke

The Maximum download size is on 'traffic mgmt' tab

This will limit all downloads through squid or am I completly wrong !?! So if I set 200MB there and will try to download an 3GB ISO it will cut my download, isn't it ?

Damn…squid has so many options it is sometime really hard to understand when to use what ;)

marcelloc

@Nachtfalke:

Damn…squid has so many options it is sometime really hard to understand when to use what ;)

I second that :)

ccesario

@Nachtfalke:

PS: I did not enable any additional options on squid - just basic settings on a VM to test.

My brwoser showed the attached screenshot.

After that I pressed F5 and re-sent:

Hi Nachtfalke, thank you by feedback!

This is the problem that happen! Exactly as your screenshot.

I have this screen in others sites too. I mean to you pfsense forum only to test/reproduce.

But in squid-2.7.9 this not happen.

PS: I too enable basic settings in squid.

Welll…. this can be considered a bug/error ?

Nachtfalke

@ccesario:

@Nachtfalke:

PS: I did not enable any additional options on squid - just basic settings on a VM to test.

My brwoser showed the attached screenshot.

After that I pressed F5 and re-sent:

Hi Nachtfalke, thank you by feedback!

This is the problem that happen! Exactly as your screenshot.

I have this screen in others sites too. I mean to you pfsense forum only to test/reproduce.

But in squid-2.7.9 this not happen.

PS: I too enable basic settings in squid.

Welll…. this can be considered a bug/error ?

So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.

Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)

ccesario

@Nachtfalke:

So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.

Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)

Hehehehh no, I don't have URL to can "spam" posts. But using pfsense forum its possible.

Edit your posts and save-it :) … I my tests I usage this to reproduce many times the error :)

Thanks

Nachtfalke

@ccesario:

@Nachtfalke:

So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.

Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)

Hehehehh no, I don't have URL to can "spam" posts. But using pfsense forum its possible.

Edit your posts and save-it :) … I my tests I usage this to reproduce many times the error :)

Thanks

Will do this perhaps this afternoon/night when I am at home. Perhaps we can tweak something if it's not a bug. :-)

marcelloc

@Nachtfalke:

Will do this perhaps this afternoon/night when I am at home. Perhaps we can tweak something if it's not a bug. :-)

Maybe a compile option like 'Be strictly HTTP compliant'

ccesario

@Nachtfalke:

@ccesario:

@Nachtfalke:

So I am using squid2.7 and squidguard here on work and posting many times on the forum and there is not that "bug". Perhaps some parameters on squid3 which causes this problems. Perhaps POST HEADER size or something like that.

Do you have an URL where we can "spam" posts to test this ? Probably it is not the best to do with pfsense forum ;o)

Hehehehh no, I don't have URL to can "spam" posts. But using pfsense forum its possible.

Edit your posts and save-it :) … I my tests I usage this to reproduce many times the error :)

Thanks

Will do this perhaps this afternoon/night when I am at home. Perhaps we can tweak something if it's not a bug. :-)

Thanks…. if possible report your tests!

Nachtfalke

Hmm, no luck till now.

tried with different browsers (IE8 and FF11)
tried with (re)moving some refresh_patterns
tried with different DNS servers for squid (8.8.8.8 and 127.0.0.1)
added this to custom options to get more information from access.log

strip_query_terms off

This is a difference I found on squid access.log


540 192.168.0.112 TCP_MISS/302 601 POST http://forum.pfsense.org/index.php?action=post2;start=45;msg=255851;sesc=b98e34206a1c8d9eb69521c441186ad3;board=15 - DIRECT/69.64.6.7 text/html
71 192.168.0.112 TCP_MISS/503 5000 POST http://forum.pfsense.org/index.php?action=post2;start=45;msg=255851;sesc=b98e34206a1c8d9eb69521c441186ad3;board=15 - DIRECT/forum.pfsense.org text/html

The 503 line uses DNS and the 302 uses an IP address…

If I click on the URL posted in the error page I get returned to the post edit page and got an error message from pfsense forum:


Your session timed out while posting. Please try to re-submit your message. No subject was filled in. The message body was left empty.

hmmm…

canefield

Marcello and others,

I've still got problems configuring Squid 3 as a reverse proxy. Somehow I can't manage it to work properly.
As you illustrated in the forst postings I did exactly the same and added NAT and Firewall rules. I'm using port 8080 and 8443.

How come…?!?!

Thanks a lot,
Canefield

ccesario

@Nachtfalke:

This is a difference I found on squid access.log


540 192.168.0.112 TCP_MISS/302 601 POST http://forum.pfsense.org/index.php?action=post2;start=45;msg=255851;sesc=b98e34206a1c8d9eb69521c441186ad3;board=15 - DIRECT/69.64.6.7 text/html
71 192.168.0.112 TCP_MISS/503 5000 POST http://forum.pfsense.org/index.php?action=post2;start=45;msg=255851;sesc=b98e34206a1c8d9eb69521c441186ad3;board=15 - DIRECT/forum.pfsense.org text/html

The 503 line uses DNS and the 302 uses an IP address…

Hmmmm this can be a hint.. O_o

marcelloc

@canefield:

Marcello and others,

I've still got problems configuring Squid 3 as a reverse proxy. Somehow I can't manage it to work properly.
As you illustrated in the forst postings I did exactly the same and added NAT and Firewall rules. I'm using port 8080 and 8443.

How come…?!?!

Thanks a lot,
Canefield

Still the same issue from TMG post? Did you removed the Nat?

Donny

Another bugs found on system log when I use revers proxy.

Apr 18 08:41:51
php: : The command '/usr/local/sbin/squid -k reconfigure' returned exit code '1', the output was '2012/04/18 08:41:51| redreshAddToList: Unknown option 'my.windowsupdate.website.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip)': reload-into-imsrange_offset_limit 2012/04/18 08:41:51| redreshAddToList: Unknown option 'my.windowsupdate.website.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip)': -1 2012/04/18 08:41:51| redreshAddToList: Unknown option 'symantecliveupdate.com/..(cab|exe|dll|msi)': reload-into-imsrange_offset_limit 2012/04/18 08:41:51| redreshAddToList: Unknown option 'symantecliveupdate.com/..(cab|exe|dll|msi)': -1 2012/04/18 08:41:51| redreshAddToList: Unknown option 'avast.com/..(vpu|cab|stamp|exe)': reload-into-imscache_mem 2012/04/18 08:41:51| redreshAddToList: Unknown option 'avast.com/..(vpu|cab|stamp|exe)': 1024 2012/04/18 08:41:51| redreshAddToList: Unknown option 'avast.com/.*.(vpu|cab|stamp|exe)': MB 2012/04/18 08:41:51| Warning: empty ACL: acl throttle_exts url

Solved! I found this problem because at dansquardian has banned "extension files". After I disable banned at extension tab, the error has gone.

marcelloc

Donny,
Check if does not happen if you uncheck dynamic content options on squid cache tab.

Donny

@marcelloc:

Donny,
Check if does not happen if you uncheck dynamic content options on squid cache tab.

Now, If I check or uncheck dynamic content options. The error is disappear.