• Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login
Netgate Discussion Forum
  • Categories
  • Recent
  • Tags
  • Popular
  • Users
  • Search
  • Register
  • Login

VideoCache problem… editing /usr/local/pkg/squid.inc

Scheduled Pinned Locked Moved pfSense Packages
13 Posts 5 Posters 8.2k Views
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • B
    belikeyeshua
    last edited by Jan 7, 2010, 1:10 PM

    I'm having a hard time knowing where to put this:

    # --BEGIN-- videocache config for squid
    url_rewrite_program /usr/bin/python /usr/share/videocache/videocache.py
    url_rewrite_children 10
    acl videocache_allow_url url_regex -i \.youtube\.com\/get_video
    acl videocache_allow_url url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/videoplayback
    acl videocache_allow_url url_regex -i \.cache[a-z0-9]?[a-z0-9]?[a-z0-9]?\.googlevideo\.com\/get_video
    acl videocache_allow_url url_regex -i proxy\-[0-9][0-9]\.dailymotion\.com\/
    acl videocache_allow_url url_regex -i [a-z0-9][0-9a-z][0-9a-z]?[0-9a-z]?[0-9a-z]?\.xtube\.com\/(.*)flv
    acl videocache_allow_url url_regex -i bitcast\.vimeo\.com\/vimeo\/videos\/
    acl videocache_allow_url url_regex -i va\.wrzuta\.pl\/wa[0-9][0-9][0-9][0-9]?
    acl videocache_allow_url url_regex -i \.files\.youporn\.com\/(.*)\/flv\/
    acl videocache_allow_url url_regex -i \.msn\.com\.edgesuite\.net\/(.*)\.flv
    acl videocache_allow_dom dstdomain v.mccont.com vp.video.google.com dl.redtube.com
    acl videocache_deny_url url_regex -i http:\/\/[a-z][a-z]\.youtube\.com http:\/\/www\.youtube\.com
    url_rewrite_access deny videocache_deny_url
    url_rewrite_access allow videocache_allow_url
    url_rewrite_access allow videocache_allow_dom
    redirector_bypass on
    # --END-- videocache config for squid
    

    This guide I'm reading says to put it after acls. Well, there's tons of places where acls is said.

    I imagine that they mean this:

    # Setup some default acls
    acl all src 0.0.0.0/0.0.0.0
    acl localhost src 127.0.0.1/255.255.255.255
    acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 $webgui_port $port 1025-65535
    acl sslports port 443 563 $webgui_port
    acl manager proto cache_object
    acl purge method PURGE
    acl connect method CONNECT
    acl dynamic urlpath_regex cgi-bin \?
    
    EOD;
    

    So, I pasted it in right after that. I restarted squid. It does not seem to be caching my youtube videos. In the gui, I went to the squid general setup page, and it said:

    Parse error: syntax error, unexpected T_STRING in /usr/local/pkg/squid.inc on line 778
    
    1 Reply Last reply Reply Quote 0
    • N
      n1ko
      last edited by Jan 8, 2010, 6:37 AM

      Put it before EOD and it will work. Also edit the python path to:
      url_rewrite_program /usr/local/bin/python /usr/share/videocache/videocache.py

      1 Reply Last reply Reply Quote 0
      • B
        belikeyeshua
        last edited by Jan 10, 2010, 3:15 AM

        @n1ko:

        Put it before EOD and it will work. Also edit the python path to:
        url_rewrite_program /usr/local/bin/python /usr/share/videocache/videocache.py

        Thank you for the reply.

        I've read up on how to set it up. I'm going by this guide here: http://wiki.ecnet.org/wiki/PfSenseVideoCache as its much more up to date.

        Right now, I've got it set up and it is working. However, there's a couple of problems:

        1. It does not work in transparent mode. In transparent mode it will copy the video to the cache, but not serve it from the cache. I imagine this is because I must have set the proxy settings in /etc/videocache.conf incorrectly. I set it for 192.168.1.1:3128 as 3128 is the port I've selected for the transparent proxy. But this must be wrong. Can someone please tell me what I need to set it to?

        2. It uses up 100% of my cpu and makes the internet to be slow at times. I've emailed the videocache people about this and hopefully they can help. They said that the newer version (1.9.1) is not supposed to do that, so I have no idea why it is.

        If anyone can help me, I would really appreciate it.

        Thanks,
        ~Shawn

        1 Reply Last reply Reply Quote 0
        • N
          n1ko
          last edited by Jan 10, 2010, 8:16 AM

          Check logs, if its using 100% cpu its probably in some kind of an busy loop. Also make sure that your lighthttpd is listening on :80. Basically this means that your pfsense administration (webgui) should be on http, not https.

          1 Reply Last reply Reply Quote 0
          • B
            belikeyeshua
            last edited by Jan 10, 2010, 1:04 PM

            @n1ko:

            Check logs, if its using 100% cpu its probably in some kind of an busy loop. Also make sure that your lighthttpd is listening on :80. Basically this means that your pfsense administration (webgui) should be on http, not https.

            I found out why its using 100% cpu… its some leak or something in python.

            Check this out:

            # ps aux
            USER     PID %CPU %MEM   VSZ   RSS  TT  STAT STARTED      TIME COMMAND
            proxy   3303 15.9  0.4 11084  8072  ??  R     2:54AM  83:53.06 (python) /usr/share/videocache/videocache.py (python
            proxy   3302 15.6  0.4 11084  8048  ??  R     2:54AM  84:02.09 (python) /usr/share/videocache/videocache.py (python
            proxy   3304 14.7  0.4 11084  8064  ??  R     2:54AM  83:57.69 (python) /usr/share/videocache/videocache.py (python
            proxy   3299 13.8  0.4 11084  8036  ??  R     2:54AM  83:56.99 (python) /usr/share/videocache/videocache.py (python
            proxy   3300 12.9  0.4 11084  8048  ??  R     2:54AM  83:55.44 (python) /usr/share/videocache/videocache.py (python
            proxy   3301 12.6  0.4 11084  8048  ??  R     2:54AM  83:45.79 (python) /usr/share/videocache/videocache.py (python
            proxy   3296  9.9  0.4 11084  8148  ??  R     2:54AM  67:53.90 (python) /usr/share/videocache/videocache.py (python
            root   51301  0.5  0.6 40712 13200  ??  D    12:42PM   0:10.14 /usr/local/bin/php
            root   53450  0.5  0.6 40712 13180  ??  S    12:48PM   0:07.47 /usr/local/bin/php
            root   57965  0.5  0.1  3492  1404  ??  S     1:00PM   0:00.00 sh -c /sbin/dmesg -a | /usr/bin/grep net4801
            root   40859  0.2  0.6 40712 13140  ??  S    12:13PM   0:07.08 /usr/local/bin/php
            root   57966  0.2  0.0  3188   920  ??  R     1:00PM   0:00.01 /sbin/dmesg -a
            root       0  0.0  0.0     0     0  ??  DLs   2:50AM   0:00.19 [swapper]
            
            

            I have no idea how to fix it, but I know its got to be python.

            I do not think there is anything odd in my logs… except for some url_rewriter warning, and also a warning telling me I should remove 192.168.0.0 (how do I do that?).

            2010/01/10 02:54:48| Beginning Validation Procedure
            2010/01/10 02:54:48|   Completed Validation Procedure
            2010/01/10 02:54:48|   Validated 6081 Entries
            2010/01/10 02:54:48|   store_swap_size = 238522k
            2010/01/10 02:54:49| storeLateRelease: released 0 objects
            2010/01/10 02:54:51| Reconfiguring Squid Cache (version 2.7.STABLE7)...
            2010/01/10 02:54:51| FD 27 Closing HTTP connection
            2010/01/10 02:54:51| FD 28 Closing HTTP connection
            2010/01/10 02:54:51| logfileClose: closing log /var/squid/log/access.log
            2010/01/10 02:54:51| Including Configuration File: /usr/local/etc/squid/squid.conf (depth 0)
            2010/01/10 02:54:51| WARNING: '192.168.0.0/255.255.255.0' is a subnetwork of '0.0.0.0/0.0.0.0'
            2010/01/10 02:54:51| WARNING: because of this '0.0.0.0/0.0.0.0' is ignored to keep splay tree searching predictable
            2010/01/10 02:54:51| WARNING: You should probably remove '192.168.0.0/255.255.255.0' from the ACL named 'localnet'
            2010/01/10 02:54:51| WARNING: '192.168.0.0/255.255.255.0' is a subnetwork of '0.0.0.0/0.0.0.0'
            2010/01/10 02:54:51| WARNING: because of this '0.0.0.0/0.0.0.0' is ignored to keep splay tree searching predictable
            2010/01/10 02:54:51| WARNING: You should probably remove '192.168.0.0/255.255.255.0' from the ACL named 'localnet'
            2010/01/10 02:54:51| Cache dir '/cache/squid' size remains unchanged at 46080000 KB
            2010/01/10 02:54:51| parseConfigFile: squid.conf:87 unrecognized: 'delay_pools'
            2010/01/10 02:54:51| parseConfigFile: squid.conf:88 unrecognized: 'delay_class'
            2010/01/10 02:54:51| parseConfigFile: squid.conf:89 unrecognized: 'delay_parameters'
            2010/01/10 02:54:51| parseConfigFile: squid.conf:90 unrecognized: 'delay_initial_bucket_level'
            2010/01/10 02:54:51| parseConfigFile: squid.conf:91 unrecognized: 'delay_access'
            2010/01/10 02:54:51| Initialising SSL.
            2010/01/10 02:54:51| logfileOpen: opening log /var/squid/log/access.log
            2010/01/10 02:54:51| Store logging disabled
            2010/01/10 02:54:51| DNS Socket created at 0.0.0.0, port 47692, FD 13
            2010/01/10 02:54:51| Adding domain local from /etc/resolv.conf
            2010/01/10 02:54:51| Adding nameserver 67.142.166.10 from /etc/resolv.conf
            2010/01/10 02:54:51| Adding nameserver 67.142.166.11 from /etc/resolv.conf
            2010/01/10 02:54:51| helperOpenServers: Starting 7 'python' processes
            2010/01/10 02:54:51| Accepting proxy HTTP connections at 192.168.1.1, port 3128, FD 25.
            2010/01/10 02:54:51| Accepting proxy HTTP connections at 192.168.0.2, port 3128, FD 26.
            2010/01/10 02:54:51| WCCP Disabled.
            2010/01/10 02:54:51| Loaded Icons.
            2010/01/10 02:54:51| Ready to serve requests.
            2010/01/10 02:56:08| WARNING: url_rewriter #7 (FD 20) exited
            2010/01/10 02:56:08| WARNING: url_rewriter #6 (FD 19) exited
            2010/01/10 02:56:09| WARNING: url_rewriter #2 (FD 15) exited
            2010/01/10 02:56:10| WARNING: url_rewriter #3 (FD 16) exited
            2010/01/10 02:56:10| Too few url_rewriter processes are running
            2010/01/10 02:56:10| Starting new helpers
            2010/01/10 02:56:10| helperOpenServers: Starting 7 'python' processes
            2010/01/10 02:56:11| WARNING: url_rewriter #1 (FD 14) exited
            2010/01/10 02:56:13| WARNING: url_rewriter #5 (FD 18) exited
            2010/01/10 02:56:14| WARNING: url_rewriter #4 (FD 17) exited
            
            

            Thanks,
            ~Shawn

            1 Reply Last reply Reply Quote 0
            • N
              n1ko
              last edited by Jan 10, 2010, 9:34 PM

              I meant videocache logs.If I my memory serves right they are under /var/log/videocache. I would first check that directories have correct rights and ownerships

              1 Reply Last reply Reply Quote 0
              • B
                belikeyeshua
                last edited by Jan 11, 2010, 12:45 AM

                There is nothing in the videocache log file. I don't know why but I go to /var/log/videocache/videocache.log and then it says that videocache.log is a directory.

                But I know its a file, that is empty.

                1 Reply Last reply Reply Quote 0
                • A
                  Alan87i
                  last edited by Mar 8, 2010, 4:43 PM

                  ps aux

                  USER    PID %CPU %MEM  VSZ  RSS  TT  STAT STARTED      TIME COMMAND
                  proxy  33018 100.0  0.8 12108  8340  ??  R    10:51AM  44:34.22 (python) /usr/share/videocache/videocache.py (python)
                  proxy  33019 98.4  0.8 12108  8240  ??  R    10:51AM  44:41.27 (python) /usr/share/videocache/videocache.py (python)

                  Does this mean I have 2 sessions of videocache running.
                  How can I remove 1 or both and start over?
                  Thanks

                  1 Reply Last reply Reply Quote 0
                  • Y
                    yellowhat89
                    last edited by Mar 30, 2010, 3:19 PM

                    why make ir hard, just use this for pfsense.

                    Stay hungry, Stay Foolish

                    1 Reply Last reply Reply Quote 0
                    • A
                      Alan87i
                      last edited by Mar 30, 2010, 4:16 PM

                      @yellowhat89:

                      why make ir hard, just use this for pfsense.

                      Use what ?

                      1 Reply Last reply Reply Quote 0
                      • A
                        andylai
                        last edited by Mar 30, 2010, 6:38 PM

                        I had almost the same problem, the VideoCache 1.9.3 cause the CPU to overload to 100%. Everything seem to be working other then the /var/log/videocache/videocache.log keep pile up with the same message.

                        2010-03-29 02:38:17,777 4955 - - RELOAD - videocache plugin was reloaded.
                        2010-03-29 02:38:17,779 4960 - - RELOAD - videocache plugin was reloaded.
                        2010-03-29 02:38:17,780 4961 - - RELOAD - videocache plugin was reloaded.
                        2010-03-29 02:38:17,781 4962 - - RELOAD - videocache plugin was reloaded.
                        2010-03-29 02:38:17,782 4963 - - RELOAD - videocache plugin was reloaded.
                        2010-03-29 02:38:17,783 4964 - - RELOAD - videocache plugin was reloaded.

                        And it may keep growing begger,

                        total 337860
                        -rwxr-xr-x  1 proxy  proxy  165M Mar 29 02:39 videocache.log.1
                        -rw-r–---  1 proxy  proxy  444B Mar 29 02:38 videocache.log

                        I have contacted the VideoCache admin on the situation. I was been advice to change SELinux from "Enforcing mode" to "Permissive mode" or totally disable it. I try to get some info about this particular on the web and found this website http://www.crypt.gen.nz/selinux/disable_selinux.html#DIS2. But it didn't mention how to do it in FreeBSD. I guest the only way is to add "enforcing=0" on /boot/grub/grub.conf,

                        title SE-Linux Test System
                        root (hd0,0)
                        kernel /boot/vmlinuz-2.4.20-selinux-2003040709 ro root=/dev/hda1 nousb enforcing=0
                        #initrd /boot/initrd-2.4.20-selinux-2003040709.img

                        But I was not absolutely sure about it. Any pfSense (FreeBSD) expert have any idea on how to do it please HELP!!! THANKS in advance.

                        1 Reply Last reply Reply Quote 0
                        • Y
                          yellowhat89
                          last edited by Apr 10, 2010, 4:45 PM

                          just follow this instruction to do videocache (CDN) http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca

                          Stay hungry, Stay Foolish

                          1 Reply Last reply Reply Quote 0
                          • A
                            andylai
                            last edited by Apr 11, 2010, 6:45 AM

                            @yellowhat89:

                            just follow this instruction to do videocache (CDN) http://code.google.com/p/pfsense-cacheboy/wiki/Pfsense_Lusca

                            Absolute right. I use pfSense Lusca for a while already and it work just fantastic. Recently updated it to R14560.

                            1 Reply Last reply Reply Quote 0
                            • First post
                              Last post
                            Copyright 2025 Rubicon Communications LLC (Netgate). All rights reserved.
                              This community forum collects and processes your personal information.
                              consent.not_received