Cant get squid to work properly.
-
Hello, i'm new to Pfsense and im loving it. But I came across a problem.
The thing is I want to put a pfsense box in a virtualbox on one of our company server for caching (Squid) and Routing.
Everything is working great except for squid3. I've setted it to cache youtube etc. But the thing is I cant get it to use more than 500 Mb of disk.
Filesystem Size Used Avail Capacity Mounted on
/dev/ad0s1a 67G 434M 62G 1% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.6M 48K 3.3M 1% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/devEventho I change how much to cache, it wont work, squid started properly, tried rebuilding cache folders, debugging… It just doesnt work.
Any ideas?
Virtual and Phisical Specs:
3 GB Ram, 3 Dedicated Nic cards (Bridged) WAN, LAN1, LAN2. 70 GB Disk Space on SSD, and 2 Cores @ 2.3 Ghz. -
Sure,
On the logs, appears squid started properly, loaded 72 objects in the folders, yet when I check the folders there only is 72 cached items.The actual config is:
Proxy interface: LAN
Proxy port: 3128
ICP port: none
Allow users on interface: checked
Transparent proxy: checked
Enable logging: check
Log rotate: 10 days
Visible hostname: localhost
Supress Squid Version: checked.
–-------------------------------------
Local cache:
Cache replacement policy: Heap LFUDA
Low watermark: 90
High watermark: 95
External cache managers: 127.0.0.1;192.168.1.80;
Hard disk cache size: 55000
Hard disk cache system: diskd
Level1 subdirectories: 128
Hard disk location: /var/squid/cache (tried to put it in home)
Minimum object size: 0
Maximum object size: 400000
Memory cache size: 450
Maximum object size in ram: 128
Memory replacement policy: Heap GDSF
Cache dynamic content: checked
Refresh patterns: Youtube, Windows Update, Symantec Antivirus.Assume that I did not post default values.
Thank you in advance.
-
Im wondering we're not the only ones.
I did a simple cache script when rebuilding a static page from php. Using a variable serial printed on pages. And it constantly changes plus I can see the WAN card blowing up when doing connection tests.
The weird thing is those 72 objects in the cache and not being able to cache much.
Im wondering if other people are having the same problems. I picked pfsense because of the great distro itself and the caching + queues.
Bummer I cant get squid to actually work as it should. Now im trying a clean setup on a virtualbox machine on a brand new Ubuntu Server install.
I'll keep this updated, it didnt work with squid also.
Squid or Squid3 no difference in the bottom line.
-
Yes, I second that.
Hehe yep, something similar, to that website, including images and such. Actually I'm not being able to cache anything. When I put a video from youtube for example, that I recently watched, I still see the WAN going nuts.
Those 72 cache items are :-\ who knows.
I tried changing sizes, and stuff, more ram less ram, more disk, less disk, changing cache location and such no effect. Running pfsense natively no change, going without transparent proxy… No effect...
I dont know whats going on actually, if I get to solve it I'll post it here. But still no effect. hehe this is driving me crazy. At least u get the same serial. I don't.
-
Following by, with the config I posted the logical stuff to happen is cache going nuts as I try to reload and cache everything. Yet disk space hasnt changed in HOURS.
[2.0.1-RELEASE][admin@takelan.router.takelan.com]/root(1): df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/ad0s1a 61G 330M 56G 1% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/md0 3.6M 52K 3.3M 2% /var/run
devfs 1.0K 1.0K 0B 100% /var/dhcpd/dev -
Check created squid conf with a made by hand config and see If you find a misconfigured option.
YouTube and many sites are dynamic, the same watch video URL cand send you to a different server but I agree with you that no cache hit is not normal
Similar problem solved on this post
http://forum.pfsense.org/index.php/topic,54182.msg289826.html#msg289826 -
Check created squid conf with a made by hand config and see If you find a misconfigured option.
YouTube and many sites are dynamic, the same watch video URL cand send you to a different server but I agree with you that no cache hit is not normal
Similar problem solved on this post
http://forum.pfsense.org/index.php/topic,54182.msg289826.html#msg289826Marcelloc just got it working, I had ticked "cache dynamic content" I unchecked it and now its working. ::) Seems like the directive cache dynamic content has something to do. I'll check that Im running the stable release 2.0.1
-
Thanks for the feedback, I 'll check it soon.
-
I'll post as soon as I have news.
-
The lines added seem correct according to Squid Doc:
This file is automatically generated by pfSense
Do not edit manually !
http_port 192.168.1.80:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language es
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 10
shutdown_lifetime 3 secondsAllow local network(s) on interface(s)
acl localnet src 192.168.1.0/24
httpd_suppress_version_string on
uri_whitespace stripacl dynamic urlpath_regex cgi-bin ?
cache deny dynamic
cache_mem 250 MB
maximum_object_size_in_memory 960 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 42680 64 256
minimum_object_size 0 KB
maximum_object_size 393216 KB
offline_mode oncache_swap_low 90
cache_swap_high 95No redirector configured
#Remote proxies
Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 808 3128 1025-65535
acl sslports port 443 563 808
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECThttp_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslportsAlways allow localhost connections
http_access allow localhost
quick_abort_min 0 KB
quick_abort_max 0 KB
request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrcReverse Proxy settings
Custom options
Setup allowed acls
Allow local network(s) on interface(s)
http_access allow localnet
Default block all to be sure
http_access deny allsrc
This file is automatically generated by pfSense
Do not edit manually !
http_port 192.168.1.80:3128
http_port 127.0.0.1:3128 intercept
icp_port 7
dns_v4_first on
pid_filename /var/run/squid.pid
cache_effective_user proxy
cache_effective_group proxy
error_default_language es
icon_directory /usr/local/etc/squid/icons
visible_hostname localhost
cache_mgr admin@localhost
access_log /var/squid/logs/access.log
cache_log /var/squid/logs/cache.log
cache_store_log none
sslcrtd_children 0
logfile_rotate 10
shutdown_lifetime 3 secondsAllow local network(s) on interface(s)
acl localnet src 192.168.1.0/24
httpd_suppress_version_string on
uri_whitespace stripBreak HTTP standard for flash videos. Keep them in cache even if asked not to.
refresh_pattern -i .flv$ 10080 90% 999999 ignore-no-cache override-expire ignore-private
Let the clients favorite video site through with full caching
acl youtube dstdomain .youtube.com
cache allow youtubeWindows Update refresh_pattern
range_offset_limit -1
refresh_pattern -i microsoft.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i my.windowsupdate.website.com/.*.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-imsSymantec refresh_pattern
range_offset_limit -1
refresh_pattern liveupdate.symantecliveupdate.com/..(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims
refresh_pattern symantecliveupdate.com/..(cab|exe|dll|msi) 10080 100% 43200 reload-into-imsAvast refresh_pattern
range_offset_limit -1
refresh_pattern avast.com/.*.(vpu|cab|stamp|exe) 10080 100% 43200 reload-into-imsAvira refresh_pattern
range_offset_limit -1
refresh_pattern personal.avira-update.com/.*.(cab|exe|dll|msi|gz) 10080 100% 43200 reload-into-ims
cache_mem 250 MB
maximum_object_size_in_memory 960 KB
memory_replacement_policy heap GDSF
cache_replacement_policy heap LFUDA
cache_dir aufs /var/squid/cache 42680 64 256
minimum_object_size 0 KB
maximum_object_size 393216 KB
offline_mode oncache_swap_low 90
cache_swap_high 95Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320No redirector configured
#Remote proxies
Setup some default acls
acl allsrc src all
acl localhost src 127.0.0.1/32
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 808 3128 1025-65535
acl sslports port 443 563 808
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECThttp_access allow manager localhost
http_access deny manager
http_access allow purge localhost
http_access deny purge
http_access deny !safeports
http_access deny CONNECT !sslportsAlways allow localhost connections
http_access allow localhost
quick_abort_min 0 KB
quick_abort_max 0 KB
request_body_max_size 0 KB
delay_pools 1
delay_class 1 2
delay_parameters 1 -1/-1 -1/-1
delay_initial_bucket_level 100
delay_access 1 allow allsrcReverse Proxy settings
Custom options
Setup allowed acls
Allow local network(s) on interface(s)
http_access allow localnet
Default block all to be sure
http_access deny allsrc
Difference between the two configs:
Add any of your own refresh_pattern entries above these.
refresh_pattern ^ftp: 1440 20% 10080
refresh_pattern ^gopher: 1440 0% 1440
refresh_pattern -i (/cgi-bin/|?) 0 0% 0
refresh_pattern . 0 20% 4320Break HTTP standard for flash videos. Keep them in cache even if asked not to.
refresh_pattern -i .flv$ 10080 90% 999999 ignore-no-cache override-expire ignore-private
Let the clients favorite video site through with full caching
acl youtube dstdomain .youtube.com
cache allow youtubeWindows Update refresh_pattern
range_offset_limit -1
refresh_pattern -i microsoft.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i windowsupdate.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
refresh_pattern -i my.windowsupdate.website.com/.*.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-imsSymantec refresh_pattern
range_offset_limit -1
refresh_pattern liveupdate.symantecliveupdate.com/..(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims
refresh_pattern symantecliveupdate.com/..(cab|exe|dll|msi) 10080 100% 43200 reload-into-imsAvast refresh_pattern
range_offset_limit -1
refresh_pattern avast.com/.*.(vpu|cab|stamp|exe) 10080 100% 43200 reload-into-imsAvira refresh_pattern
range_offset_limit -1
refresh_pattern personal.avira-update.com/.*.(cab|exe|dll|msi|gz) 10080 100% 43200 reload-into-imsIdeas?
-
Nope I unchecked it and now its simply caching static stuff. There is no config I can make it cache dynamic as youtube, windows update or other.
Simply when I tick cache dynamic content, all I get are TCP MISS 200, if I uncheck it, I get hits for static content.
