Cant get squid to work properly.



  • Hello, i'm new to Pfsense and im loving it. But I came across a problem.

    The thing is I want to put a pfsense box in a virtualbox on one of our company server for caching (Squid) and Routing.

    Everything is working great except for squid3. I've setted it to cache youtube etc. But the thing is I cant get it to use more than 500 Mb of disk.

    Filesystem    Size    Used  Avail Capacity  Mounted on
    /dev/ad0s1a    67G    434M    62G    1%    /
    devfs          1.0K    1.0K      0B  100%    /dev
    /dev/md0      3.6M    48K    3.3M    1%    /var/run
    devfs          1.0K    1.0K      0B  100%    /var/dhcpd/dev

    Eventho I change how much to cache, it wont work, squid started properly, tried rebuilding cache folders, debugging… It just doesnt work.

    Any ideas?

    Virtual and Phisical Specs:
    3 GB Ram, 3 Dedicated Nic cards (Bridged) WAN, LAN1, LAN2. 70 GB Disk Space on SSD, and 2 Cores @ 2.3 Ghz.



  • Sure,
    On the logs, appears squid started properly, loaded 72 objects in the folders, yet when I check the folders there only is 72 cached items.

    The actual config is:
    Proxy interface: LAN
    Proxy port: 3128
    ICP port: none
    Allow users on interface: checked
    Transparent proxy: checked
    Enable logging: check
    Log rotate: 10 days
    Visible hostname: localhost
    Supress Squid Version: checked.
    –-------------------------------------
    Local cache:
    Cache replacement policy: Heap LFUDA
    Low watermark: 90
    High watermark: 95
    External cache managers: 127.0.0.1;192.168.1.80;
    Hard disk cache size: 55000
    Hard disk cache system: diskd
    Level1 subdirectories: 128
    Hard disk location: /var/squid/cache (tried to put it in home)
    Minimum object size: 0
    Maximum object size: 400000
    Memory cache size: 450
    Maximum object size in ram: 128
    Memory replacement policy: Heap GDSF
    Cache dynamic content: checked
    Refresh patterns: Youtube, Windows Update, Symantec Antivirus.

    Assume that I did not post default values.

    Thank you in advance.



  • Im wondering we're not the only ones.

    I did a simple cache script when rebuilding a static page from php. Using a variable serial printed on pages. And it constantly changes plus I can see the WAN card blowing up when doing connection tests.

    The weird thing is those 72 objects in the cache and not being able to cache much.

    Im wondering if other people are having the same problems. I picked pfsense because of the great distro itself and the caching + queues.

    Bummer I cant get squid to actually work as it should. Now im trying a clean setup on a virtualbox machine on a brand new Ubuntu Server install.

    I'll keep this updated, it didnt work with squid also.

    Squid or Squid3 no difference in the bottom line.



  • Yes, I second that.

    Hehe yep, something similar, to that website, including images and such. Actually I'm not being able to cache anything. When I put a video from youtube for example, that I recently watched, I still see the WAN going nuts.

    Those 72 cache items are  :-\ who knows.

    I tried changing sizes, and stuff, more ram less ram, more disk, less disk, changing cache location and such no effect. Running pfsense natively no change, going without transparent proxy… No effect...

    I dont know whats going on actually, if I get to solve it I'll post it here. But still no effect. hehe this is driving me crazy. At least u get the same serial. I don't.



  • Following by, with the config I posted the logical stuff to happen is cache going nuts as I try to reload and cache everything. Yet disk space hasnt changed in HOURS.

    [2.0.1-RELEASE][admin@takelan.router.takelan.com]/root(1): df -h
    Filesystem    Size    Used  Avail Capacity  Mounted on
    /dev/ad0s1a    61G    330M    56G    1%    /
    devfs          1.0K    1.0K      0B  100%    /dev
    /dev/md0      3.6M    52K    3.3M    2%    /var/run
    devfs          1.0K    1.0K      0B  100%    /var/dhcpd/dev



  • Check created squid conf with a made by hand config and see If you find a misconfigured option.

    YouTube and many sites are dynamic, the same watch video URL cand send you to a different server but I agree with you that no cache hit is not normal

    Similar problem solved on this post
    http://forum.pfsense.org/index.php/topic,54182.msg289826.html#msg289826



  • @marcelloc:

    Check created squid conf with a made by hand config and see If you find a misconfigured option.

    YouTube and many sites are dynamic, the same watch video URL cand send you to a different server but I agree with you that no cache hit is not normal

    Similar problem solved on this post
    http://forum.pfsense.org/index.php/topic,54182.msg289826.html#msg289826

    Marcelloc just got it working, I had ticked "cache dynamic content" I unchecked it and now its working.  ::) Seems like the directive cache dynamic content has something to do. I'll check that Im running the stable release 2.0.1



  • Thanks for the feedback,  I 'll check it soon.



  • I'll post as soon as I have news.



  • The lines added seem correct according to Squid Doc:

    This file is automatically generated by pfSense

    Do not edit manually !

    http_port 192.168.1.80:3128
    http_port 127.0.0.1:3128 intercept
    icp_port 7
    dns_v4_first on
    pid_filename /var/run/squid.pid
    cache_effective_user proxy
    cache_effective_group proxy
    error_default_language es
    icon_directory /usr/local/etc/squid/icons
    visible_hostname localhost
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    sslcrtd_children 0
    logfile_rotate 10
    shutdown_lifetime 3 seconds

    Allow local network(s) on interface(s)

    acl localnet src  192.168.1.0/24
    httpd_suppress_version_string on
    uri_whitespace strip

    acl dynamic urlpath_regex cgi-bin ?
    cache deny dynamic
    cache_mem 250 MB
    maximum_object_size_in_memory 960 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    cache_dir aufs /var/squid/cache 42680 64 256
    minimum_object_size 0 KB
    maximum_object_size 393216 KB
    offline_mode oncache_swap_low 90
    cache_swap_high 95

    No redirector configured

    #Remote proxies

    Setup some default acls

    acl allsrc src all
    acl localhost src 127.0.0.1/32
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 808 3128 1025-65535
    acl sslports port 443 563 808
    acl manager proto cache_object
    acl purge method PURGE
    acl connect method CONNECT

    http_access allow manager localhost
     
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports

    Always allow localhost connections

    http_access allow localhost

    quick_abort_min 0 KB
    quick_abort_max 0 KB
    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc

    Reverse Proxy settings

    Custom options

    Setup allowed acls

    Allow local network(s) on interface(s)

    http_access allow localnet

    Default block all to be sure

    http_access deny allsrc

    This file is automatically generated by pfSense

    Do not edit manually !

    http_port 192.168.1.80:3128
    http_port 127.0.0.1:3128 intercept
    icp_port 7
    dns_v4_first on
    pid_filename /var/run/squid.pid
    cache_effective_user proxy
    cache_effective_group proxy
    error_default_language es
    icon_directory /usr/local/etc/squid/icons
    visible_hostname localhost
    cache_mgr admin@localhost
    access_log /var/squid/logs/access.log
    cache_log /var/squid/logs/cache.log
    cache_store_log none
    sslcrtd_children 0
    logfile_rotate 10
    shutdown_lifetime 3 seconds

    Allow local network(s) on interface(s)

    acl localnet src  192.168.1.0/24
    httpd_suppress_version_string on
    uri_whitespace strip

    Break HTTP standard for flash videos. Keep them in cache even if asked not to.

    refresh_pattern -i .flv$ 10080 90% 999999 ignore-no-cache override-expire ignore-private

    Let the clients favorite video site through with full caching

    acl youtube dstdomain .youtube.com
    cache allow youtube

    Windows Update refresh_pattern

    range_offset_limit -1
    refresh_pattern -i microsoft.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.
    .(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i my.windowsupdate.website.com/.*.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims

    Symantec refresh_pattern

    range_offset_limit -1
    refresh_pattern liveupdate.symantecliveupdate.com/..(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims
    refresh_pattern symantecliveupdate.com/.
    .(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims

    Avast refresh_pattern

    range_offset_limit -1
    refresh_pattern avast.com/.*.(vpu|cab|stamp|exe) 10080 100% 43200 reload-into-ims

    Avira refresh_pattern

    range_offset_limit -1
    refresh_pattern personal.avira-update.com/.*.(cab|exe|dll|msi|gz) 10080 100% 43200 reload-into-ims
    cache_mem 250 MB
    maximum_object_size_in_memory 960 KB
    memory_replacement_policy heap GDSF
    cache_replacement_policy heap LFUDA
    cache_dir aufs /var/squid/cache 42680 64 256
    minimum_object_size 0 KB
    maximum_object_size 393216 KB
    offline_mode oncache_swap_low 90
    cache_swap_high 95

    Add any of your own refresh_pattern entries above these.

    refresh_pattern ^ftp:    1440  20%  10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|?) 0  0%  0
    refresh_pattern .    0  20%  4320

    No redirector configured

    #Remote proxies

    Setup some default acls

    acl allsrc src all
    acl localhost src 127.0.0.1/32
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 808 3128 1025-65535
    acl sslports port 443 563 808
    acl manager proto cache_object
    acl purge method PURGE
    acl connect method CONNECT

    http_access allow manager localhost
     
    http_access deny manager
    http_access allow purge localhost
    http_access deny purge
    http_access deny !safeports
    http_access deny CONNECT !sslports

    Always allow localhost connections

    http_access allow localhost

    quick_abort_min 0 KB
    quick_abort_max 0 KB
    request_body_max_size 0 KB
    delay_pools 1
    delay_class 1 2
    delay_parameters 1 -1/-1 -1/-1
    delay_initial_bucket_level 100
    delay_access 1 allow allsrc

    Reverse Proxy settings

    Custom options

    Setup allowed acls

    Allow local network(s) on interface(s)

    http_access allow localnet

    Default block all to be sure

    http_access deny allsrc

    Difference between the two configs:

    Add any of your own refresh_pattern entries above these.

    refresh_pattern ^ftp:    1440  20%  10080
    refresh_pattern ^gopher:  1440  0%  1440
    refresh_pattern -i (/cgi-bin/|?) 0  0%  0
    refresh_pattern .    0  20%  4320

    Break HTTP standard for flash videos. Keep them in cache even if asked not to.

    refresh_pattern -i .flv$ 10080 90% 999999 ignore-no-cache override-expire ignore-private

    Let the clients favorite video site through with full caching

    acl youtube dstdomain .youtube.com
    cache allow youtube

    Windows Update refresh_pattern

    range_offset_limit -1
    refresh_pattern -i microsoft.com/..(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i windowsupdate.com/.
    .(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims
    refresh_pattern -i my.windowsupdate.website.com/.*.(cab|exe|ms[i|u|f]|asf|wm[v|a]|dat|zip) 4320 80% 43200 reload-into-ims

    Symantec refresh_pattern

    range_offset_limit -1
    refresh_pattern liveupdate.symantecliveupdate.com/..(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims
    refresh_pattern symantecliveupdate.com/.
    .(cab|exe|dll|msi) 10080 100% 43200 reload-into-ims

    Avast refresh_pattern

    range_offset_limit -1
    refresh_pattern avast.com/.*.(vpu|cab|stamp|exe) 10080 100% 43200 reload-into-ims

    Avira refresh_pattern

    range_offset_limit -1
    refresh_pattern personal.avira-update.com/.*.(cab|exe|dll|msi|gz) 10080 100% 43200 reload-into-ims

    Ideas?



  • Nope I unchecked it and now its simply caching static stuff. There is no config I can make it cache dynamic as youtube, windows update or other.

    Simply when I tick cache dynamic content, all I get are TCP MISS 200, if I uncheck it, I get hits for static content.


Log in to reply