Squid3 Caching for xbox marketplace
-
Next week Call of Duty Ghosts is coming out and we have 4 xboxes in the house, we opted to purchase the digital version on release date. I estimate somewhere around a 20GB download, onto 4-5 xboxes. We have 200Mb downlink here, but i'd really like to optimize that to get everyone up and running as quick as possible. I'm new to squid, i'd like a bit of advice getting this set up to cache the download locally to minimize the overall wait. Has anyone done this before? I did a quick trial run by downloading a demo game on the xbox on one system, then signing off and onto another and attempting to download again with no luck.
-
Test this way:
-
Change cache system to other then null
-
Disable dynamic cache(it was based on squid wiki but users report that doesn't work)
-
Increase object size and cache size to a value higher then the largest file on your 20G download.
-
-
Thanks, updated object size to 104857600 (100GB)
I tested all the other options, but misread and instead entered Mb instead of the requested Kb. I will test again and let you know the outcome
-
Still no luck. Tested with a 1Gb download.
EDIT: Sorry, after further research this is working. However it is very slow to pull from the cache server. May be the the xbox only has a 10/100 port? seems kinda silly with how new they are.
-
Okay, sorry for all of the posts. but I verified that the hardware is in fact 10/100, however I am experiencing speeds much slower. Speeds from the cache server are only 10-12 Mb, when I would expect 10-12MB?
Just ran a test on a gig connection, same results. it is caching and pulling from cache but at very slow speeds. Where should I start troubleshooting?
-
check for pfsense disk performance optimizing. During install, you can enable soft updates. After install there are some sysctrl options to speed up disk.
-
check for pfsense disk performance optimizing. During install, you can enable soft updates. After install there are some sysctrl options to speed up disk.
I have 5x 75gb scsi drives in raid5, this "shouldn't" be a disk issue, however I have not looked into that possibility yet.
I monitored the initial download again after clearing the cache server. it was at ~15Mb from both WAN and Cache server. I think this may be an xbox limitation. I am checking on the xbox forum, though I doubt the professionals with the knowledge I am seeking are on that forum.
-
check for pfsense disk performance optimizing. During install, you can enable soft updates. After install there are some sysctrl options to speed up disk.
I have 5x 75gb scsi drives in raid5, this "shouldn't" be a disk issue, however I have not looked into that possibility yet.
I monitored the initial download again after clearing the cache server. it was at ~15Mb from both WAN and Cache server. I think this may be an xbox limitation. I am checking on the xbox forum, though I doubt the professionals with the knowledge I am seeking are on that forum.
RAID5 Is the worst of em all as far as performance.
-
check for pfsense disk performance optimizing. During install, you can enable soft updates. After install there are some sysctrl options to speed up disk.
I have 5x 75gb scsi drives in raid5, this "shouldn't" be a disk issue, however I have not looked into that possibility yet.
I monitored the initial download again after clearing the cache server. it was at ~15Mb from both WAN and Cache server. I think this may be an xbox limitation. I am checking on the xbox forum, though I doubt the professionals with the knowledge I am seeking are on that forum.
RAID5 Is the worst of em all as far as performance.
is it ~15Mb/s bad? i'd suspect better performance than that at least
-
The problem is not the disk itself. I had serious performance problems with pfsense and postfix package on ibm 3650. After enabling soft updates I coud get all disk performance.
Try scp a file from and to pfsense to check how fast you can go.
-
RAID5 Is the worst of em all as far as performance.
I agree, if u read a little about squid tunning, raid-5 is not on the list.
1; What are the technical details of your machine?
2; How much ram u have in your system?
3; use aufs, null is went u want to disable cache.
4; raid-1
5; U know the file size of the files u will be download? if can fit into your ram size for squid u can change maximum object size to that value if it fit into your ram size.
6; add noatime in the fstab file.
7; big files big nic 1GB+.
8; setting up 1TB for squid won't give u better results, start with 16GB, this will help the system to get the files more faster, the head of the disk will move less in 16GB than 100GB
I will setup the disk layout like this:/ 8GB
/var 64GBI will start the cache disk size wih 16GB and start caching and see access.log, run calamaris to see cache performance and start tuning the system base on 1 day caching.
add this to squid custom settings:
buffered_logs off
client_db off
log_fqdn off
log_icp_queries off
memory_pools off
check_hostnames offmy 2 cents.
-
1. It's a dedicated Dell PowerEdge 2850, as you know running raid 5
2. has 4GB Ram
3. I am currently using aufs
4. I really don't want to destroy and rebuild the raid right now, but I will as a last resort.
5. The files will most likely exceed the total ram on the system.
6. I have doubts this will cause any significant increase in speed. I will try it and let you know the results.
7. Server has Gb nic, client has 100Mb, still we are only seeing about 1/10th of the proper speed. I also tested this on my PC with a 100MB file. I pulled it faster from the net than I did the cache server.
8. the total download is likely ~40GB, I will likely need at least 50GB cache to pull this off.
Version 2.1-RELEASE (amd64)
built on Wed Sep 11 18:17:37 EDT 2013
FreeBSD 8.3-RELEASE-p11You are on the latest version.
Platform pfSense
CPU Type Intel(R) Xeon(TM) CPU 3.20GHz
4 CPUs: 2 package(s) x 1 core(s) x 2 HTT threadsState table size
0% (145/407000)Show states
MBUF Usage
9% (2196/25600)Load average
0.00, 0.00, 0.00CPU usage
0%Memory usage
17% of 4074 MBSWAP usage
0% of 8192 MBDisk usage
1% of 190GI know, overkill for what I am using it for, but I don't see the system being the issue. I am barely touching its capabilities with pfsense.
-
The problem is not the disk itself. I had serious performance problems with pfsense and postfix package on ibm 3650. After enabling soft updates I coud get all disk performance.
Try scp a file from and to pfsense to check how fast you can go.
SFTP will copy to and from at 35MB/s
-
Just tested a download on my PC. 400-500KB from WAN, only 5-10KB from cache.
Also tested a local cp, 1GB file was completed in under 30 sec.
I think something else is wonky here.
-
If u see the web page slow, try enable "resolv dns v4" check box, some times this cause issues.
Remember that the proxy will use first the RAM and will use the disk once he decide is time to send the object to hd.
-
Everything pulling from cache is at 5KB/s now. I have to disable squid for now. Hopefully someone has more thoughts as to why this is happening.
-
Everything pulling from cache is at 5KB/s now. I have to disable squid for now. Hopefully someone has more thoughts as to why this is happening.
Have you try the stable build?
You are using the devel build. -
WOW! Problem solved!
Went to wipe the raid and install as raid 0
Found a failed drive and removed it. set up as raid 0 and did a fresh install, set everything back up manually. rebooted and it works like a champ.
Issue was I was running in degraded mode. I still find it interesting everything else was working like a champ.
-
Great, if u are going to have this for a long time, move to raid1 for your own safety.
my 2 cents!!! -
Great, if u are going to have this for a long time, move to raid1 for your own safety.
my 2 cents!!!+1
