[SHARE] Update Lusca Release - r14371 (November 18, 2009)

grage95

http://code.google.com/p/lusca-cache/wiki/LuscaChangeLog

Changes between Squid-2.HEAD and Lusca-1.0

* Squid-2 defaults to use async disk operations for disk read() / open() ; Lusca uses the async disk operations for -all- operations. This improves performance on FreeBSD/Solaris, where write() / close() may block more often than not.
* Wide-scale source reorganisation into separate libraries - facilitating code modularity, code reuse and much easier unit testing.
* An IPv4/IPv6 aware socket address type has been added - see libsqinet/ in the top-level source directory.
* Various core modules have been prepared for IPv6 support - including core networking support and internal DNS routines.
* An example single-threaded TCP proxy - see app/tcptest/ in the top-level source directory - which demonstrates re-using the core libraries in other applications.

Changes between Lusca-1.0 and Lusca-HEAD

* COSS now does not write out swap log files which it doesn't use - improving performance during normal operation and logfile rotation.
* The server-side code (src/http.c) has been restructured to remove extra data copying and in preparation for further improvements performance and memory utilisation.
* The memory caching layer ("MemPools") has been turned into purely statistics - this is in preparation for fully threading the core libraries.
* The majority of code which uses String has been adapted (and rewritten in some places) in preparation for reference counted string management.
* The threaded IO code (via aufs) has been turned into a generic library for doing asynchronous IO. A new option exists for tweaking the number of IO threads to use - "n_aiops_threads" The default is to use 4 threads per AUFS storedir and 3 threads per COSS storedir.
* The diskd and ufs storage types have been removed, leaving only aufs, coss and null. These types will return later on when the disk io layers have been tidied up and re-unified.
* The ZPH code has been slightly reworked (in r14164 / issue 40 ) to make the evaluation order both clearer (it now is documented in the configuration file!) and hopefully more useful in production.

cache_mem untuk menyimpan hot transit object
cache_mem default 8 mb
disalah satu warnet sy gunakan 6 mb
di parent proxy sy gunakan 256 mb

terserah, cache mau banyak di taroh di ram/hadisk, jika cache_mem besar, agak lambat write ke hardiks, keuntungan cache lebih cepat diakses client

depen on your situation

zass

Tapi kalu tinggikan max filesize kan terjadi begini

" If you wish to increase speed more than you want to save bandwidth, this should be set to a low value. "

maknanya kita kena gunakan hardware yng lebih tinggi spt harddisk yng lebih cepat - thanks

grage95

yes

syarat server squd ideal jika request perdetik > 100/s
rpm hardisk besar,jumlah hardisk fisik minimal 2 buah (recomended scasi 10000 rpm) & memory ram minimal 2 Gb

atau kita bisa membuat
cache_dir null 0 /dev/null = (0 Mb)
cache_mem 1024M

hanya menggunakan RAM sahaja, tidak menggunakan hardisk

zass

Salam , satu lagi pertanyaan selepas update ke lusca ini bolehkah kita buat tuning seperti

di http://forum.pfsense.org/index.php/topic,20001.0.html

"fetch http://shakau.googlepages.com/tunning.conf
fetch http://shakau.googlepages.com/storeurl.pl
chmod +x storeurl.pl
chown proxy:proxy storeurl.pl
chown proxy:proxy tunning.conf "

harap bantuan :)

grage95

boleh,
di sesuikan lagi option refresh_pattern
silahkan lihat option-nya di /usr/local/etc/squid.conf.default

serangku

streaming indowebster masih belum bersahabat om …
ada yg ngerasain juga gak ...

utak-atik belum ketemu juga

ipoelnet

@serangku:

streaming indowebster masih belum bersahabat om …
ada yg ngerasain juga gak ...

utak-atik belum ketemu juga

Btw Mau ngeblock apa mau nglimit?

serangku

gak utk ngeblock
atau juga ngelimit

hanya kok gak smooth streamingnya, alias loading mulu, lama ….
gak langsung streaming gitu ... seperti youtube
apa yah yg salah yah ...

thanks atas pencerahan lbh lanjut

4r31

@grage95:

delete dulu squid/lusca yang lama

pkg_delete squid*
pkg_delete lusca*
pkg_add -rv http://shakau.googlepages.com/vanila-lusca-1.4_2.tbz
rehash
squid -v

restart squid dengan mengclick stop/start di webgui services squid

NB: LUSCA hanya support file system COSS dan AUFS tdk support UFS dan DISKD

Mas grage95 …Mantab..2 jempol buat grage95 ;D

lansung uninstall squid

pkg_delete squid*

trus..

pkg_add -rv http://shakau.googlepages.com/vanila-lusca-1.4_2.tbz

then...
squid -v

last tekan stop/start atau restart squid...

jalan tuh Luscanya.. ::)

squid -v

Squid Cache: Version LUSCA_HEAD
configure options: '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--disable-auth' '--disable-wccpv2' '--with-maxfd=16384' '--with-pthreads' '--enable-storeio=null aufs coss' '--enable-delay-pools' '--disable-carp' '--disable-wccp' '--disable-ident-lookups' '--enable-pf-transparent' '--with-large-files' '--enable-large-cache-files' '--enable-err-languages=English' '--enable-default-err-language=English' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2' 'CC=cc' 'CFLAGS=-O2 -pipe -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS='

dah di monitoring selama 4 hari pakai lightsquid :

Date Group Users Oversize Bytes Average Hit %
25 Nov 2009 grp 28 20 951.4 M 34.0 M 20.46%
24 Nov 2009 grp 29 23 2.3 G 79.8 M 8.38%
23 Nov 2009 grp 29 23 2.5 G 86.8 M 10.15%
19 Nov 2009 grp 21 13 468.6 M 22.3 M 17.31%

Lumayan ...ada perubahan walau pun belum significant.. ;D

Mas grage95 and rekans, Gimana biar bisa hit 50% ya..? :P

grage95

harus di tunning :
step2nya

tambahkan ini di squid.inc
include /usr/local/etc/squid/tunning.conf
setelah baris
acl dynamic urlpath_regex cgi-bin ?

jadinya nanti
_**acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 $webgui_port $port 1025-65535
acl sslports port 443 563 $webgui_port
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
acl dynamic urlpath_regex cgi-bin ?

include /usr/local/etc/squid/tunning.conf

EOD;**_

dan masuk ke directory
cd /usr/local/etc/squid/
dan download file2 ini :
fetch http://freebsd-squid-system.googlecode.com/files/tunning.conf
fetch http://freebsd-squid-system.googlecode.com/files/storeurl.pl
chmod +x storeurl.pl
chown proxy:proxy storeurl.pl
chown proxy:proxy tunning.conf
cd /etc
fetch http://freebsd-squid-system.googlecode.com/files/sysctl.conf
cd /boot
fetch http://freebsd-squid-system.googlecode.com/files/loader.conf

trus reboot server

ipoelnet

Tuning tambahan Experimen:

System -> General Setup :

DNS servers :
127.0.0.1
203.130.196.155 ---> DNS prymary ISP

buat file /usr/local/etc/dnsmasq.conf

port=53
cache-size=10000
expand-hosts

/etc/sysctl.conf

net.inet.ip.fastforwarding=1
net.inet.ip.portrange.last=65535
net.inet.ip.portrange.first=1024
net.inet.icmp.icmplim=0
net.inet.icmp.icmplim_output=0
net.inet.tcp.msl=3000
net.inet.tcp.hostcache.expire=3900
net.inet.tcp.inflight.enable=0
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
net.inet.tcp.delayed_ack=0
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=1048576
net.local.stream.sendspace=1048576
net.inet.ip.intr_queue_maxlen=5000
net.inet.tcp.sendbuf_max=65536
net.inet.tcp.recvbuf_max=65536
net.inet.tcp.slowstart_flightsize=54
net.inet.tcp.local_slowstart_flightsize=10
net.inet.tcp.nolocaltimewait=1
kern.ipc.maxsockbuf=16777216
kern.ipc.maxsockets=65536
kern.ipc.somaxconn=32768
kern.ipc.nmbclusters=131072
kern.polling.burst_max=1000
kern.polling.each_burst=50
kern.maxfiles=262144
kern.maxfilesperproc=65536
kern.ipc.shmall=32768
kern.ipc.shmmax=134217728
kern.ipc.semmap=256
kern.dirdelay=6
kern.metadelay=5
kern.filedelay=7

/boot/loader.conf

autoboot_delay="1"
kern.ipc.maxsockbuf="16777216"
kern.ipc.nmbclusters="131072"
kern.ipc.msgmnb="16384"
kern.ipc.msgssz="64"
kern.ipc.msgtql="4096"
kern.ipc.shmseg="16"
kern.ipc.somaxconn="32768"
kern.ipc.nmbclusters="131072"
kern.ipc.maxsockets="65536"
kern.maxfiles="262144"
kern.maxfilesperproc="65536"
net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.hostcache.hashsize="1024"

untuk tuning sysctl.conf silahkan sesuaikan memory dan HW and lihat manual FreeBSD;
lihat configurasi :
sysctl [option]

# sysctl -a

agar tidak melebihi layar

# sysctl -a | more

atau jika melihat valunya saja :
sysctl [key]
# sysctl net.inet.tcp.hostcache.hashsize
net.inet.tcp.hostcache.hashsize:1024

untuk ngeset value :
sysctl [key]=[value]
# sysctl net.inet.tcp.hostcache.hashsize=2048
net.inet.tcp.hostcache.hashsize:1024 -> 2048
jika menunjukkan read only, biasanya setting di lettakkan di /boot/loader.conf lalu reboot pf.

Atau pingin melihat key pergroup :
misal key dengan awalan net.inet

# sysctl net.inet
net.inet.tcp.hostcache.hashsize:2048
---------
---------
---------
---------dst

# sysctl kern.ipc
kern.ipc.msgssz:64
kern.ipc.msgtql:4096
kern.ipc.shmseg:16
kern.ipc.somaxconn:32768
kern.ipc.nmbclusters:131072
kern.ipc.maxsockets:65536
---------
---------
---------
---------dst

Selamat ber-eksperimen.

serangku

kawan2 … mohon pencerahan ...

last pid: 29871; load averages: 0.13, 0.07, 0.03 up 8+01:57:47 11:32:06
237 processes: 6 running, 211 sleeping, 2 zombie, 18 waiting
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
Mem: 378M Active, 1322M Inact, 215M Wired, 34M Cache, 112M Buf, 41M Free
Swap: 2048M Total, 100K Used, 2048M Free

itu hasil top dari mesin LUSCA
ram pisik 2 Gig, cahe dir 100 GB, baru 34% terisi ...
sudah "mau" swapkah ... atau sudah terengah2 LUSCA nya ?

makasih yooo ...

grage95

RAM : 1322M Inact = memory yg pernah terpakai cuman tidak aktif
RAM FREE (belum pernah terpakai sama sekali) = 41M Free
SWAP: 2048M Total, 100K Used, 2048M Free

masih belum kena swap, 2048M-2048M= 0

untuk menggunakan top sebaiknya jangan yg relatime, gunakan option -n {nilai baris yang mau di tampilkan}

contoh top -n 60

aplikasi top realtime makan memory banyak

serangku

thanks om @grage95 atas pencerahannya …

dapat disimpulkan masih aman2 saja :D
keragu2 an hilang sudah, waktunya swapping ke LUSCA

om ...
sekiranya dibuat thread baru ttg performance LUSCA, gimana om ...
sekedar "mengintimidasi" kawan2 utk tidak ragu pake LUSCA sebagai aternatif dari yg sudah ada
juga ... ubek2 di sub international sptnya belum ada
regional Indo bisa jadi pelopornya ... ;D :o

grage95

mungkin yang perlu di bahas bareng2:

1. Sarat2 & formula ideal (hardisk & ram)
2. optimasi cache dengan fitur store_rewrite_url (hanya ada di lusca & squid-2.7.x)
3. optimasi refresh_pattern
4. optimasi kernel
5. optimasi dnsmasq utk menghandle client besar
6. ….

silahkan barangkali ada yang perlu di tambahkan, nanti kita garap bareng2, masukan/saran di tunggu, kita bongkar rahasia proxy highperformance

berita menarik, setelah penasaran dengan performance freebsd, opensolaris & linux, ternyata dengan install minimalis OS dan optimasi kernel, spek mesin dan config yang hampir sama (beda di pengaturan di directory log saja ) lebih unggul di freebsd, saking penasaranya dengan freebsd ini, kernel freebsd ini di ganti dengan mengcopy kernel dari pfsense saja tepar hihihi. monitoring dengan menggunakan mrtg eksternal (mrtg di box lain hanya menyedot snmp proxy "enable-snmp"), urutannya keunggulan ini di tinjau dari efisiensi memory, kecepatan untuk menyimpan cache dan ke stabilan dari req/hits yang tinggi.

1. freebsd-7.2-release (ufs dgn option noatime mantap tenan)
2. opensolaris (zfs nya edun euy, quick tapi boros memory hiks hiks)
3. linux (reiserfs (utk file2 kecil) dan ext4 (utk file2 besar) podo wae walopun udah di tuning di fstab notail, noatime, nyoba reiser4 waktu load tinggi kernel panic, udah utak atik sysctl.conf tetep saja hiks)

servis yang jalan hanya dns cache dan lusca

utk cache hits di total hampir sama, cuma beda timing saja
salute 4 jempol untuk developer pfsense ini (quick, easy, fast & stable)

zass

Salam , mahu bertanya, bagaimana mahu aktifkan SNMP , sudah cuba baca di cacti forum , tapi tak berhasil

T/K
zass

grage95

![](http://lethe.uwa.edu.au/munin/Servers/styx.uwa.edu.au-squid_hits-month.png[/img<br /><br /><br />[img]http://lethe.uwa.edu.au/munin/Servers/styx.uwa.edu.au-squid_requests-week.png)

download package pfSense Lusca Release - r14371 (November 18, 2009) with patch + snmp + arp

pkg_delete lusca*
or
pkg_delete squid*

and then install
pkg_add -rv http://squid-proxy-pkg.googlecode.com/files/lusca-pfsense-arp-snmp-r14371_2.tbz

enable snmp on squid.inc

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all

info :
build with

pf-bsd72# squid -v
Squid Cache: Version LUSCA_HEAD
configure options:  '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--with-pthreads' '--enable-storeio=aufs coss' '--enable-delay-pools' '--enable-snmp' '--disable-carp' '--disable-wccp' '--disable-ident-lookups' '--enable-arp-acl' '--enable-pf-transparent' '--with-large-files' '--enable-large-cache-files' '--enable-err-languages=English' '--enable-default-err-language=English' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2' 'CC=cc' 'CFLAGS=-O2 -pipe -funroll-loops -ffast-math  -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS='

patch with : aggressive.patch, 2451x.patch,lusca-vary.patch,loop2.patch (fix bug looping for use store_url_rewrite)

tips install squidstats

pkg_add -rv squidstats

after finisih, lets go config

_mkdir -p /var/db/squidstats/graphs
mkdir -p /var/db/squidstats/rrd
chown www:wheel /var/db/squidstats/graphs
chown proxy:wheel /var/db/squidstats/rrd

ln -s /var/db/squidstats/graphs /usr/local/www/data/
ln -s /usr/local/www/cgi-bin/graph-summary.cgi /usr/local/www/graph-summary.cgi
su -m proxy -c "/usr/local/bin/squidstats.pl createdb"
su -m proxy -c "/usr/local/bin/squidstats.pl gather 2"_
edit /etc/crontab
add this line

*/5 * * * * /usr/local/bin/squidstats.pl gather 2 > /dev/null

restart crontab /etc/rc.d/cron restart

test open with your browser

http://your-ip/graph-summary.cgi

zass

Ertinya kalau saya buat sampai di sini saja :

"pkg_delete lusca*
or
pkg_delete squid*

and then install
pkg_add -rv http://squid-proxy-pkg.googlecode.com/files/lusca-pfsense-arp-snmp-r14371_2.tbz

enable snmp on squid.inc

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all "

squid sudah support snmp > dan boleh di ambil datanya dari server lain ,

misalnya gunakan cacti ?

tq

grage95

jika menggunkan mrtg di cacti other box

acl snmp_host src your-ip-cacti
acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all "

dan di firewall open port 3401 protocol udp

di box cacti install net-snmp dan cacti-template-squidstats

edit /etc/snmpd.conf
tambahkan line ini :

#sec.name source community
com2sec local localhost public
com2sec mynetwork 1.2.3.0/24 public

check with snmpwalk

snmpwalk -v2c -c public your-ip-pfsense-squid-box:3401 .1.3.6.1.4.1.3495.1.1

zass

tq,tq.tq ;D saya coba dulu !