[SHARE] Update Lusca Release - r14371 (November 18, 2009)

grage95

harus di tunning :
step2nya

tambahkan ini di squid.inc
include /usr/local/etc/squid/tunning.conf
setelah baris
acl dynamic urlpath_regex cgi-bin ?

jadinya nanti
_**acl all src 0.0.0.0/0.0.0.0
acl localhost src 127.0.0.1/255.255.255.255
acl safeports port 21 70 80 210 280 443 488 563 591 631 777 901 $webgui_port $port 1025-65535
acl sslports port 443 563 $webgui_port
acl manager proto cache_object
acl purge method PURGE
acl connect method CONNECT
acl dynamic urlpath_regex cgi-bin ?

include /usr/local/etc/squid/tunning.conf

EOD;**_

dan masuk ke directory
cd /usr/local/etc/squid/
dan download file2 ini :
fetch http://freebsd-squid-system.googlecode.com/files/tunning.conf
fetch http://freebsd-squid-system.googlecode.com/files/storeurl.pl
chmod +x storeurl.pl
chown proxy:proxy storeurl.pl
chown proxy:proxy tunning.conf
cd /etc
fetch http://freebsd-squid-system.googlecode.com/files/sysctl.conf
cd /boot
fetch http://freebsd-squid-system.googlecode.com/files/loader.conf

trus reboot server

ipoelnet

Tuning tambahan Experimen:

System -> General Setup :

DNS servers :
127.0.0.1
203.130.196.155 ---> DNS prymary ISP

buat file /usr/local/etc/dnsmasq.conf

port=53
cache-size=10000
expand-hosts

/etc/sysctl.conf

net.inet.ip.fastforwarding=1
net.inet.ip.portrange.last=65535
net.inet.ip.portrange.first=1024
net.inet.icmp.icmplim=0
net.inet.icmp.icmplim_output=0
net.inet.tcp.msl=3000
net.inet.tcp.hostcache.expire=3900
net.inet.tcp.inflight.enable=0
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=65536
net.inet.tcp.delayed_ack=0
net.inet.udp.recvspace=65535
net.inet.udp.maxdgram=57344
net.local.stream.recvspace=1048576
net.local.stream.sendspace=1048576
net.inet.ip.intr_queue_maxlen=5000
net.inet.tcp.sendbuf_max=65536
net.inet.tcp.recvbuf_max=65536
net.inet.tcp.slowstart_flightsize=54
net.inet.tcp.local_slowstart_flightsize=10
net.inet.tcp.nolocaltimewait=1
kern.ipc.maxsockbuf=16777216
kern.ipc.maxsockets=65536
kern.ipc.somaxconn=32768
kern.ipc.nmbclusters=131072
kern.polling.burst_max=1000
kern.polling.each_burst=50
kern.maxfiles=262144
kern.maxfilesperproc=65536
kern.ipc.shmall=32768
kern.ipc.shmmax=134217728
kern.ipc.semmap=256
kern.dirdelay=6
kern.metadelay=5
kern.filedelay=7

/boot/loader.conf

autoboot_delay="1"
kern.ipc.maxsockbuf="16777216"
kern.ipc.nmbclusters="131072"
kern.ipc.msgmnb="16384"
kern.ipc.msgssz="64"
kern.ipc.msgtql="4096"
kern.ipc.shmseg="16"
kern.ipc.somaxconn="32768"
kern.ipc.nmbclusters="131072"
kern.ipc.maxsockets="65536"
kern.maxfiles="262144"
kern.maxfilesperproc="65536"
net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.tcbhashsize="4096"
net.inet.tcp.hostcache.hashsize="1024"

untuk tuning sysctl.conf silahkan sesuaikan memory dan HW and lihat manual FreeBSD;
lihat configurasi :
sysctl [option]

# sysctl -a

agar tidak melebihi layar

# sysctl -a | more

atau jika melihat valunya saja :
sysctl [key]
# sysctl net.inet.tcp.hostcache.hashsize
net.inet.tcp.hostcache.hashsize:1024

untuk ngeset value :
sysctl [key]=[value]
# sysctl net.inet.tcp.hostcache.hashsize=2048
net.inet.tcp.hostcache.hashsize:1024 -> 2048
jika menunjukkan read only, biasanya setting di lettakkan di /boot/loader.conf lalu reboot pf.

Atau pingin melihat key pergroup :
misal key dengan awalan net.inet

# sysctl net.inet
net.inet.tcp.hostcache.hashsize:2048
---------
---------
---------
---------dst

# sysctl kern.ipc
kern.ipc.msgssz:64
kern.ipc.msgtql:4096
kern.ipc.shmseg:16
kern.ipc.somaxconn:32768
kern.ipc.nmbclusters:131072
kern.ipc.maxsockets:65536
---------
---------
---------
---------dst

Selamat ber-eksperimen.

serangku

kawan2 … mohon pencerahan ...

last pid: 29871; load averages: 0.13, 0.07, 0.03 up 8+01:57:47 11:32:06
237 processes: 6 running, 211 sleeping, 2 zombie, 18 waiting
CPU: 0.0% user, 0.0% nice, 0.0% system, 0.0% interrupt, 0.0% idle
Mem: 378M Active, 1322M Inact, 215M Wired, 34M Cache, 112M Buf, 41M Free
Swap: 2048M Total, 100K Used, 2048M Free

itu hasil top dari mesin LUSCA
ram pisik 2 Gig, cahe dir 100 GB, baru 34% terisi ...
sudah "mau" swapkah ... atau sudah terengah2 LUSCA nya ?

makasih yooo ...

grage95

RAM : 1322M Inact = memory yg pernah terpakai cuman tidak aktif
RAM FREE (belum pernah terpakai sama sekali) = 41M Free
SWAP: 2048M Total, 100K Used, 2048M Free

masih belum kena swap, 2048M-2048M= 0

untuk menggunakan top sebaiknya jangan yg relatime, gunakan option -n {nilai baris yang mau di tampilkan}

contoh top -n 60

aplikasi top realtime makan memory banyak

serangku

thanks om @grage95 atas pencerahannya …

dapat disimpulkan masih aman2 saja :D
keragu2 an hilang sudah, waktunya swapping ke LUSCA

om ...
sekiranya dibuat thread baru ttg performance LUSCA, gimana om ...
sekedar "mengintimidasi" kawan2 utk tidak ragu pake LUSCA sebagai aternatif dari yg sudah ada
juga ... ubek2 di sub international sptnya belum ada
regional Indo bisa jadi pelopornya ... ;D :o

grage95

mungkin yang perlu di bahas bareng2:

1. Sarat2 & formula ideal (hardisk & ram)
2. optimasi cache dengan fitur store_rewrite_url (hanya ada di lusca & squid-2.7.x)
3. optimasi refresh_pattern
4. optimasi kernel
5. optimasi dnsmasq utk menghandle client besar
6. ….

silahkan barangkali ada yang perlu di tambahkan, nanti kita garap bareng2, masukan/saran di tunggu, kita bongkar rahasia proxy highperformance

berita menarik, setelah penasaran dengan performance freebsd, opensolaris & linux, ternyata dengan install minimalis OS dan optimasi kernel, spek mesin dan config yang hampir sama (beda di pengaturan di directory log saja ) lebih unggul di freebsd, saking penasaranya dengan freebsd ini, kernel freebsd ini di ganti dengan mengcopy kernel dari pfsense saja tepar hihihi. monitoring dengan menggunakan mrtg eksternal (mrtg di box lain hanya menyedot snmp proxy "enable-snmp"), urutannya keunggulan ini di tinjau dari efisiensi memory, kecepatan untuk menyimpan cache dan ke stabilan dari req/hits yang tinggi.

1. freebsd-7.2-release (ufs dgn option noatime mantap tenan)
2. opensolaris (zfs nya edun euy, quick tapi boros memory hiks hiks)
3. linux (reiserfs (utk file2 kecil) dan ext4 (utk file2 besar) podo wae walopun udah di tuning di fstab notail, noatime, nyoba reiser4 waktu load tinggi kernel panic, udah utak atik sysctl.conf tetep saja hiks)

servis yang jalan hanya dns cache dan lusca

utk cache hits di total hampir sama, cuma beda timing saja
salute 4 jempol untuk developer pfsense ini (quick, easy, fast & stable)

zass

Salam , mahu bertanya, bagaimana mahu aktifkan SNMP , sudah cuba baca di cacti forum , tapi tak berhasil

T/K
zass

grage95

![](http://lethe.uwa.edu.au/munin/Servers/styx.uwa.edu.au-squid_hits-month.png[/img<br /><br /><br />[img]http://lethe.uwa.edu.au/munin/Servers/styx.uwa.edu.au-squid_requests-week.png)

download package pfSense Lusca Release - r14371 (November 18, 2009) with patch + snmp + arp

pkg_delete lusca*
or
pkg_delete squid*

and then install
pkg_add -rv http://squid-proxy-pkg.googlecode.com/files/lusca-pfsense-arp-snmp-r14371_2.tbz

enable snmp on squid.inc

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all

info :
build with

pf-bsd72# squid -v
Squid Cache: Version LUSCA_HEAD
configure options:  '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--with-pthreads' '--enable-storeio=aufs coss' '--enable-delay-pools' '--enable-snmp' '--disable-carp' '--disable-wccp' '--disable-ident-lookups' '--enable-arp-acl' '--enable-pf-transparent' '--with-large-files' '--enable-large-cache-files' '--enable-err-languages=English' '--enable-default-err-language=English' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2' 'CC=cc' 'CFLAGS=-O2 -pipe -funroll-loops -ffast-math  -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS='

patch with : aggressive.patch, 2451x.patch,lusca-vary.patch,loop2.patch (fix bug looping for use store_url_rewrite)

tips install squidstats

pkg_add -rv squidstats

after finisih, lets go config

_mkdir -p /var/db/squidstats/graphs
mkdir -p /var/db/squidstats/rrd
chown www:wheel /var/db/squidstats/graphs
chown proxy:wheel /var/db/squidstats/rrd

ln -s /var/db/squidstats/graphs /usr/local/www/data/
ln -s /usr/local/www/cgi-bin/graph-summary.cgi /usr/local/www/graph-summary.cgi
su -m proxy -c "/usr/local/bin/squidstats.pl createdb"
su -m proxy -c "/usr/local/bin/squidstats.pl gather 2"_
edit /etc/crontab
add this line

*/5 * * * * /usr/local/bin/squidstats.pl gather 2 > /dev/null

restart crontab /etc/rc.d/cron restart

test open with your browser

http://your-ip/graph-summary.cgi

zass

Ertinya kalau saya buat sampai di sini saja :

"pkg_delete lusca*
or
pkg_delete squid*

and then install
pkg_add -rv http://squid-proxy-pkg.googlecode.com/files/lusca-pfsense-arp-snmp-r14371_2.tbz

enable snmp on squid.inc

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all "

squid sudah support snmp > dan boleh di ambil datanya dari server lain ,

misalnya gunakan cacti ?

tq

grage95

jika menggunkan mrtg di cacti other box

acl snmp_host src your-ip-cacti
acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all "

dan di firewall open port 3401 protocol udp

di box cacti install net-snmp dan cacti-template-squidstats

edit /etc/snmpd.conf
tambahkan line ini :

#sec.name source community
com2sec local localhost public
com2sec mynetwork 1.2.3.0/24 public

check with snmpwalk

snmpwalk -v2c -c public your-ip-pfsense-squid-box:3401 .1.3.6.1.4.1.3495.1.1

zass

tq,tq.tq ;D saya coba dulu !

zass

Grage5, mengapa saya punya tiada option snmp ?

$ squid -v
Squid Cache: Version LUSCA_HEAD
configure options: '–bindir=/usr/local/sbin'
'--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid'
'--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid'
'--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=lru heap'
'--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll'
'--with-pthreads' '--enable-storeio=aufs' '--enable-delay-pools'
'--disable-carp' '--disable-wccp' '--disable-ident-lookups'
'--enable-pf-transparent' '--with-large-files' '--enable-large-cache-files'
'--enable-err-languages=English' '--enable-default-err-language=English'
'--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/'
'--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2'
'CC=cc' 'CFLAGS=-O2 -pipe -funroll-loops -ffast-math -fno-strict-aliasing'
'LDFLAGS=' 'CPPFLAGS='

tq

grage95

sudah mengupdate package dnegan http://squid-proxy-pkg.googlecode.com/files/lusca-pfsense-arp-snmp-r14371_2.tbz
dan sudah di rehash ?

dedieko

Bos Grage, ada sedikit masalah:

su -m proxy -c "/usr/local/bin/squidstats.pl createdb"

Can't locate RRDs.pm in @INC (@INC contains: /usr/local/lib/perl5/5.8.8/BSDPAN /usr/local/lib/perl5/site_perl/5.8.8/mach /usr/local/lib/perl5/site_perl/5.8.8 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.8/mach /usr/local/lib/perl5/5.8.8 .) at /usr/local/bin/squidstats.pl line 4.
BEGIN failed–compilation aborted at /usr/local/bin/squidstats.pl line 4.

grage95

kok perl nya versinya beda ya,
perl5/5.8.8

kalau di saya

perl5/5.8.9
pfsense vesi 1.3.xx

perlu di upgrade perlnya mungkin,

jika menggunakan pfsense 1.3

setenv PACKAGESITE "ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.2-release/Latest/"

pkg-delete -f perl-5*
pkg_add -rv perl

dedieko

@grage95:

kok perl nya versinya beda ya,
perl5/5.8.8

kalau di saya

perl5/5.8.9
pfsense vesi 1.3.xx

perlu di upgrade perlnya mungkin,

jika menggunakan pfsense 1.3

setenv PACKAGESITE "ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.2-release/Latest/"

pkg-delete -f perl-5*
pkg_add -rv perl

Saya pakai Pfsense 1.2.3 bos, upgraded from 1.2

zass

Akhir , berhasil tkasih grage5 ;D

ssnmp.JPG_thumb

ssnmp2.JPG_thumb

111ichael

@grage95:

kok perl nya versinya beda ya,
perl5/5.8.8

kalau di saya

perl5/5.8.9
pfsense vesi 1.3.xx

perlu di upgrade perlnya mungkin,

jika menggunakan pfsense 1.3

setenv PACKAGESITE "ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-7.2-release/Latest/"

pkg-delete -f perl-5*
pkg_add -rv perl

tolong dong di bantuin saya pkg_add -rv perl hasilx nongol seperti ini….

# pkg_add -rv perl
pkg_add: can't stat package file 'perl'
pkg_add: 1 package addition(s) failed

perl saya sudah tdak ada… gimana???? :-[

dedieko

Hi everybody,

LUSCA sudah running selama 2/3 hari di pfsense 1.2.3

Pagi ini saya memperhatikan ada yang line seperti ini:

2009/12/13 07:25:22| Ready to serve requests.
2009/12/13 07:25:23| Store rebuilding is 100.0% complete
2009/12/13 07:25:23| /var/squid/cache: completed rebuild
2009/12/13 07:25:23| Done scanning /var/squid/cache (27283 entries)
2009/12/13 07:25:23| Finished rebuilding storage from disk.
2009/12/13 07:25:23| 27283 Entries scanned
2009/12/13 07:25:23| 0 Invalid entries.
2009/12/13 07:25:23| 0 With invalid flags.
2009/12/13 07:25:23| 24072 Objects loaded.
2009/12/13 07:25:23| 0 Objects expired.
2009/12/13 07:25:23| 1338 Objects cancelled.
2009/12/13 07:25:23| 221 Duplicate URLs purged.
2009/12/13 07:25:23| 1426 Swapfile clashes avoided.
2009/12/13 07:25:23| Took 1.0 seconds (24494.6 objects/sec).
2009/12/13 07:25:23| Beginning Validation Procedure
2009/12/13 07:25:23| Completed Validation Procedure
2009/12/13 07:25:23| Validated 22734 Entries
2009/12/13 07:25:23| store_swap_size = 363114k
2009/12/13 07:25:24| storeLateRelease: released 0 objects
2009/12/13 07:41:38| squidaio_queue_request: WARNING - Queue congestion
2009/12/13 07:53:28| squidaio_queue_request: WARNING - Queue congestion

Hasil penelusuran di google menunjukkan kalo "Queue Congestion" terjadi jika CPU overload atau disk saya I/O-nya sudah mentok

Berikut Data System

# dmesg | grep CPU

CPU: Intel(R) Celeron(R) CPU 2.00GHz (1999.95-MHz 686-class CPU)
cpu0: <acpi cpu="">on acpi0
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0

atacontrol list

ATA channel 0:
Master: ad0 <st340014a 8.01="">ATA/ATAPI revision 6
Slave: no device present
ATA channel 1:
Master: no device present
Slave: no device present

dmesg | grep memory

real memory = 1065287680 (1015 MB)
avail memory = 1028685824 (981 MB)
agp0: detected 8060k stolen memory

swapinfo -k

Device 1K-blocks Used Avail Capacity
/dev/ad0s1b 2097152 0 2097152 0%

#top
last pid: 6357; load averages: 0.12, 0.11, 0.09 up 0+00:42:45 08:06:17
111 processes: 2 running, 92 sleeping, 17 waiting
CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle
Mem: 70M Active, 32M Inact, 54M Wired, 632K Cache, 52M Buf, 829M Free
Swap: 2048M Total, 2048M Free
…</st340014a></cpu></acpi>

Mohon pencerahannya
Terima kasih, Matur nuwun, thank you

grage95

kalau menggunakan lusca
perbesar n_aiops_threads jadi 32