[SHARE] Update Lusca Release - r14371 (November 18, 2009)

dedieko

Hi everybody,

LUSCA sudah running selama 2/3 hari di pfsense 1.2.3

Pagi ini saya memperhatikan ada yang line seperti ini:

2009/12/13 07:25:22| Ready to serve requests.
2009/12/13 07:25:23| Store rebuilding is 100.0% complete
2009/12/13 07:25:23| /var/squid/cache: completed rebuild
2009/12/13 07:25:23| Done scanning /var/squid/cache (27283 entries)
2009/12/13 07:25:23| Finished rebuilding storage from disk.
2009/12/13 07:25:23| 27283 Entries scanned
2009/12/13 07:25:23| 0 Invalid entries.
2009/12/13 07:25:23| 0 With invalid flags.
2009/12/13 07:25:23| 24072 Objects loaded.
2009/12/13 07:25:23| 0 Objects expired.
2009/12/13 07:25:23| 1338 Objects cancelled.
2009/12/13 07:25:23| 221 Duplicate URLs purged.
2009/12/13 07:25:23| 1426 Swapfile clashes avoided.
2009/12/13 07:25:23| Took 1.0 seconds (24494.6 objects/sec).
2009/12/13 07:25:23| Beginning Validation Procedure
2009/12/13 07:25:23| Completed Validation Procedure
2009/12/13 07:25:23| Validated 22734 Entries
2009/12/13 07:25:23| store_swap_size = 363114k
2009/12/13 07:25:24| storeLateRelease: released 0 objects
2009/12/13 07:41:38| squidaio_queue_request: WARNING - Queue congestion
2009/12/13 07:53:28| squidaio_queue_request: WARNING - Queue congestion

Hasil penelusuran di google menunjukkan kalo "Queue Congestion" terjadi jika CPU overload atau disk saya I/O-nya sudah mentok

Berikut Data System

# dmesg | grep CPU

CPU: Intel(R) Celeron(R) CPU 2.00GHz (1999.95-MHz 686-class CPU)
cpu0: <acpi cpu="">on acpi0
p4tcc0: <cpu frequency="" thermal="" control="">on cpu0

atacontrol list

ATA channel 0:
Master: ad0 <st340014a 8.01="">ATA/ATAPI revision 6
Slave: no device present
ATA channel 1:
Master: no device present
Slave: no device present

dmesg | grep memory

real memory = 1065287680 (1015 MB)
avail memory = 1028685824 (981 MB)
agp0: detected 8060k stolen memory

swapinfo -k

Device 1K-blocks Used Avail Capacity
/dev/ad0s1b 2097152 0 2097152 0%

#top
last pid: 6357; load averages: 0.12, 0.11, 0.09 up 0+00:42:45 08:06:17
111 processes: 2 running, 92 sleeping, 17 waiting
CPU: 0.0% user, 0.0% nice, 100% system, 0.0% interrupt, 0.0% idle
Mem: 70M Active, 32M Inact, 54M Wired, 632K Cache, 52M Buf, 829M Free
Swap: 2048M Total, 2048M Free
…</st340014a></cpu></acpi>

Mohon pencerahannya
Terima kasih, Matur nuwun, thank you

grage95

kalau menggunakan lusca
perbesar n_aiops_threads jadi 32

grage95

@111ichael:

# pkg_add -rv perl
pkg_add: can't stat package file 'perl'
pkg_add: 1 package addition(s) failed

perl saya sudah tdak ada… gimana???? :-[

[/quote]

cek dengan pkg_info |grep perl
jika sudah tidak ada install ulang, tapi harus sesuaikan dengan versi kernel pfsensenya
jika masih ada silangkan di uninstall
pkg_delete -f perl*

coba di uname -a

kalau menggunakan
FreeBSD 7.2-Release : pkg_add -rv http://125.160.17.22/freebsd/ports/i386/packages-7.2-release/lang/perl-5.8.9_2.tbz
FreeBSD 7-STABLE : pkg_add -rv http://125.160.17.22/freebsd/ports/i386/packages-7-stable/lang/perl-5.8.9_3.tbz

dan jangan lupa di rehash

111ichael

@grage95:

@111ichael:

cek dengan pkg_info |grep perl
jika sudah tidak ada install ulang, tapi harus sesuaikan dengan versi kernel pfsensenya
jika masih ada silangkan di uninstall
pkg_delete -f perl*

coba di uname -a

kalau menggunakan
FreeBSD 7.2-Release : pkg_add -rv http://125.160.17.22/freebsd/ports/i386/packages-7.2-release/lang/perl-5.8.9_2.tbz
FreeBSD 7-STABLE : pkg_add -rv http://125.160.17.22/freebsd/ports/i386/packages-7-stable/lang/perl-5.8.9_3.tbz

dan jangan lupa di rehash

Makasih atas pencerahannya….. om Grage ;D

ipoelnet

Kalau keluar gini pada cache.log

2009/12/15 17:57:10| Beginning Validation Procedure
2009/12/15 17:57:10| 262144 Entries Validated so far.
2009/12/15 17:57:10| 524288 Entries Validated so far.
2009/12/15 17:57:10| Completed Validation Procedure
2009/12/15 17:57:10| Validated 559523 Entries
2009/12/15 17:57:10| store_swap_size = 6777879k
2009/12/15 17:57:10| storeLateRelease: released 0 objects

apa penyebabnya, dan apa berpengaruh Om2?

tomplee

om grage saya pake pfsense 1.2.2 lusca pake yang versi ini http://shakau.googlepages.com/vanila-arp-lusca-1.4_2.tbz
yang lusca vanila versi ini kok gak mau di gabung ma adzap versi ini adzap-20090301.tbz dia pake perl 5.8.9_3 kalau perl-nya yang tak pake yang ini sesuai permintaan adzap luscanya gak mau jalan. solusinya kira bgm ya..

grage95

coba cek dulu perlnya ang sudah terinstall

pkg_info perl*

jika masih menggunakan perl 5.8.8, update perlnya ke versi 5.8.9

coba lusca paket baru lebih responsip hasil patch utak atik gathuk :) , lumayan responsif dan stabil,
cuman cache swap nya harus di delete dulu baru di squid -z,

http://squid-proxy-pkg.googlecode.com/files/pfsense-lusca-head-r14371_3.tbz (menggunakan perl-5.8.9_3.tbz) dan bisa jalan dengan perl-5.8.9_2.tbz

atau update versi pfsensenya,
versi 1.2.3 release lebih maknyuss paralel threadnya utk proxy daripada versi 1.2.2 release dan banyak lagi perbaikannya

tomplee

@grage95:

coba cek dulu perlnya ang sudah terinstall

pkg_info perl*

jika masih menggunakan perl 5.8.8, update perlnya ke versi 5.8.9

coba lusca paket baru lebih responsip hasil patch utak atik gathuk :) , lumayan responsif dan stabil,
cuman cache swap nya harus di delete dulu baru di squid -z,

http://squid-proxy-pkg.googlecode.com/files/pfsense-lusca-head-r14371_3.tbz (menggunakan perl-5.8.9_3.tbz) dan bisa jalan dengan perl-5.8.9_2.tbz

atau update versi pfsensenya,
versi 1.2.3 release lebih maknyuss paralel threadnya utk proxy daripada versi 1.2.2 release dan banyak lagi perbaikannya

Oke tak cobanya update lusca yang baru..

zass

grage95

tukar kepada perl 5.8.9 , Lightsquid graph 500 Internal Server - Error , ???

grage95

upgrade lightsquidnya :

http://125.160.17.22/freebsd/ports/i386/packages-7-stable/www/lightsquidtbz
http://125.160.17.22/freebsd/ports/i386/packages-7.2-release/www/lightsquidtbz

yantisna

@grage95:

![](http://lethe.uwa.edu.au/munin/Servers/styx.uwa.edu.au-squid_hits-month.png[/img<br /><br /><br />[img]http://lethe.uwa.edu.au/munin/Servers/styx.uwa.edu.au-squid_requests-week.png)

download package pfSense Lusca Release - r14371 (November 18, 2009) with patch + snmp + arp

pkg_delete lusca*
or
pkg_delete squid*

and then install
pkg_add -rv http://squid-proxy-pkg.googlecode.com/files/lusca-pfsense-arp-snmp-r14371_2.tbz

enable snmp on squid.inc

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all

info :
build with
pf-bsd72# squid -v
Squid Cache: Version LUSCA_HEAD
configure options:  '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--with-pthreads' '--enable-storeio=aufs coss' '--enable-delay-pools' '--enable-snmp' '--disable-carp' '--disable-wccp' '--disable-ident-lookups' '--enable-arp-acl' '--enable-pf-transparent' '--with-large-files' '--enable-large-cache-files' '--enable-err-languages=English' '--enable-default-err-language=English' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2' 'CC=cc' 'CFLAGS=-O2 -pipe -funroll-loops -ffast-math  -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS='
patch with : aggressive.patch, 2451x.patch,lusca-vary.patch,loop2.patch (fix bug looping for use store_url_rewrite)

tips install squidstats

pkg_add -rv squidstats

after finisih, lets go config

_mkdir -p /var/db/squidstats/graphs
mkdir -p /var/db/squidstats/rrd
chown www:wheel /var/db/squidstats/graphs
chown proxy:wheel /var/db/squidstats/rrd

ln -s /var/db/squidstats/graphs /usr/local/www/data/
ln -s /usr/local/www/cgi-bin/graph-summary.cgi /usr/local/www/graph-summary.cgi
su -m proxy -c "/usr/local/bin/squidstats.pl createdb"
su -m proxy -c "/usr/local/bin/squidstats.pl gather 2"_
edit /etc/crontab
add this line

*/5 * * * * /usr/local/bin/squidstats.pl gather 2 > /dev/null

restart crontab /etc/rc.d/cron restart

test open with your browser

http://your-ip/graph-summary.cgi

su -m proxy -c "/usr/local/bin/squidstats.pl createdb"

/etc/rc.initial: cannot create /tmp/last_term_seen: Permission denied

maksudnya apa ya om ? padahal saya kan udah login pake root?

grage95

@yantisna:

su -m proxy -c "/usr/local/bin/squidstats.pl createdb"

/etc/rc.initial: cannot create /tmp/last_term_seen: Permission denied

maksudnya apa ya om ? padahal saya kan udah login pake root?

apakah sudah di
chown www:wheel /var/db/squidstats/graphs
chown proxy:wheel /var/db/squidstats/rrd
kalau Permission denied dengan user proxy tidak bisa berarti data sudah tercreate sebagai root, gunakan user proxy utk keamanan, jangan root
coba delet dulu,
cek ls -ll /var/db/squidstats/
apakah masih menggunakan user root ???

disini ada yang berhasil mencontoh instalasi squidstats,
nama usernya kweteng,
step2nya sama

kweteng

Wah jadi sungkan
jadi step2nya

1. delete squid atau lusca
pkg_delete lusca*
atau
pkg_delete squid*

2. update package
http://squid-proxy-pkg.googlecode.com/files/pfsense-lusca-head-r14371_3.tbz

3. lalu masuk ke squid.inc atau tunning.conf, untuk mengaktifkan snmp tambahkan

acl snmp_host src 127.0.0.1
snmp_port 3401
acl snmppublic snmp_community public
snmp_access allow snmppublic snmp_host
snmp_access deny all

4. pastikan dengan squid -v

squid -v

Squid Cache: Version LUSCA_HEAD
configure options: '–bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-wccpv2' '--disable-epoll' '--with-pthreads' '--enable-storeio=aufs' '--enable-delay-pools' '–enable-snmp' '–disable-carp' '--disable-wccp' '--disable-ident-lookups' '--enable-pf-transparent' '--enable-follow-x-forwarded-for' '--with-large-files' '--enable-large-cache-files' '--enable-err-languages=English' '--enable-default-err-language=English' '--prefix=/usr/local' '--mandir=/usr/local/man' '--infodir=/usr/local/info/' '--build=i386-portbld-freebsd7.2' 'build_alias=i386-portbld-freebsd7.2' 'CC=/usr/local/libexec/ccache/world-cc' 'CFLAGS=-O2 -pipe -funroll-loops -ffast-math -fno-strict-aliasing' 'LDFLAGS=' 'CPPFLAGS='

5. lalu delete cache yang ada, dengan tujuan agar lebih stabil
stop service lusca, lewat webgui aja, tunggu bentar, br delete
rm -rf /directory-cache/*
squid -z
lalu di start lagi squidnya

6. sistem sudah siap, sekarang waktunya install squidstats
pkg_add -rv squidstats
tunggu, gak lama dikit ko

7. tunningnya

mkdir -p /var/db/squidstats/graphs
mkdir -p /var/db/squidstats/rrd
chown www:wheel /var/db/squidstats/graphs
chown proxy:wheel /var/db/squidstats/rrd

ln -s /var/db/squidstats/graphs /usr/local/www/
ln -s /usr/local/www/cgi-bin/graph-summary.cgi /usr/local/www/graph-summary.cgi
su -m proxy -c "/usr/local/bin/squidstats.pl createdb"
su -m proxy -c "/usr/local/bin/squidstats.pl gather 2"

8. edit crontab
edit /etc/crontab
tambahkan line berikut

*/5 * * * * /usr/local/bin/squidstats.pl gather 2 > /dev/null

lalu restat
restart crontab /etc/rc.d/cron restart

9. test dengan browser
http://your-ip/graph-summary.cgi

kalo ada yang kurang mohon maaf
semua berhasil atas bantuan dari bos grage, trimakasih yang sebesar2nya kepada bos grage

biatche

# /usr/local/bin/squidstats.pl createdb
Can't locate RRDs.pm in @INC (@INC contains: /usr/local/lib/perl5/5.8.8/BSDPAN /usr/local/lib/perl5/site_perl/5.8.8/mach /usr/local/lib/perl5/site_perl/5.8.8 /usr/local/lib/perl5/site_perl /usr/local/lib/perl5/5.8.8/mach /usr/local/lib/perl5/5.8.8 .) at /usr/local/bin/squidstats.pl line 4.
BEGIN failed--compilation aborted at /usr/local/bin/squidstats.pl line 4.

grage95

update perlnya
default install squidsstats jalan di pfsense veris 1.2.3.xx, jika hasil upgrade dri 1.2 upgrede juga perlnya

biatche

aku pkg_add -rfv perl 5.8.9.. sekarang ok sudah tapi … graph tak ada apa data
apa problem kalau macam ni

# su -m proxy -c "/usr/local/bin/squidstats.pl gather 2"
Got: cacheSysVMsize
Got: cacheSysStorage
Got: cacheNumObjCount
Got: cacheMemUsage
Got: cacheCpuTime
Got: cacheCurrentFileDescrCnt
Got: cacheCurrentFileDescrMax
Got: cacheProtoClientHttpRequests
Got: cacheHttpHits
Got: cacheHttpErrors
Got: cacheHttpInKb
Got: cacheHttpOutKb
Got: cacheClients
Got: cacheIcpPktsSent
Got: cacheIcpPktsRecv
Got: cacheIcpKbSent
Got: cacheIcpKbRecv
Got: cacheServerRequests
Got: cacheServerErrors
Got: cacheServerInKb
Got: cacheServerOutKb
Got: cacheHttpAllSvcTime.5
Got: cacheHttpMissSvcTime.5
Got: cacheHttpNmSvcTime.5
Got: cacheHttpNhSvcTime.5
Got: cacheHttpHitSvcTime.5
Got: cacheIcpQuerySvcTime.5
Got: cacheIcpReplySvcTime.5
Got: cacheDnsSvcTime.5
Got: cacheIpEntries
Got: cacheIpRequests
Got: cacheIpHits
Got: cacheIpMisses
Got: cacheFqdnEntries
Got: cacheFqdnRequests
Got: cacheFqdnHits
Got: cacheFqdnMisses
Got: cacheDnsRequests
Got: cacheDnsReplies
READ: 85816
READ: 311865
READ: 11059
READ: 98443
READ: 168
READ: 38
READ: 63
READ: 28847
READ: 6058
READ: 0
READ: 28496
READ: 529684
READ: 1
READ: 0
READ: 0
READ: 0
READ: 0
READ: 25908
READ: 0
READ: 387379
READ: 27479
READ: 685
READ: 851
READ: 0
READ: 649
READ: 1
READ: 0
READ: 0
READ: 0
READ: 771
READ: 13895
READ: 12469
READ: 1325
READ: 2
READ: 4894
READ: 9
READ: 4885
READ: 6
READ: 6
names: SysVMsize:SysStorage:NumObjCount:MemUsage:CpuTime:CurrentFileDescrCnt:CurrentFileDescrMax:ClientHttpRequests:HttpHits:HttpErrors:HttpInKb:HttpOutKb:Clients:IcpPktsSent:IcpPktsRecv:IcpKbSent:IcpKbRecv:ServerRequests:ServerErrors:ServerInKb:ServerOutKb:HttpAllSvcTime:HttpMissSvcTime:HttpNmSvcTime:HttpNhSvcTime:HttpHitSvcTime:IcpQuerySvcTime:IcpReplySvcTime:DnsSvcTime:IpEntries:IpRequests:IpHits:IpMisses:FqdnEntries:FqdnRequests:FqdnHits:FqdnMisses:DnsRequests:DnsReplies
values: 85816:311865:11059:98443:168:38:63:28847:6058:0:28496:529684:1:0:0:0:0:25908:0:387379:27479:685:851:0:649:1:0:0:0:771:13895:12469:1325:2:4894:9:4885:6:6
OK

UPDATE: tak apa, fix sudah

incoming.1d.png
incoming.1d.png_thumb

grage95

tunggu 15 menitan ?

apakah crond sudah aktif ??

edit crontab
edit /etc/crontab
tambahkan line berikut

*/5 * * * * /usr/local/bin/squidstats.pl gather 2 > /dev/null
dan restart crontab, tunggu 1 jam apakah sudah menampillan graph :)

agar supaya crontab tetap aktif saat di reboot,
edit /cf/conf/config.xml dan tambahkan list rules squidstats pada bagian crontab setelah kata2 <command></command>/usr/local/sbin/squid -k rotate

contoh :

```
<minute>0</minute>
<hour>0</hour>
<mday></mday>
<month></month>
<wday>*</wday>
<who>root</who>
<command></command>/usr/local/sbin/squid -k rotate

<minute>/5</minute>
<hour></hour>
<mday></mday>
<month></month>
<wday>*</wday>
<who>proxy</who>
<command></command>/usr/local/bin/squidstats.pl gather 2 > /dev/null

serangku

om grage95 …
mohon pencerahan ...

spek mesin sbb :
2 gb ram, 1 disk utk pfsense (ata), 1 disk utk coss (sata/1gb), 2 disk utk aufs (sata/2x50gb)
hasil top

top -n 3

last pid: 35439; load averages: 0.00, 0.02, 0.00 up 9+00:17:31 16:15:10
121 processes: 1 running, 117 sleeping, 3 zombie

Mem: 248M Active, 1475M Inact, 193M Wired, 59M Cache, 112M Buf, 22M Free
Swap: 4096M Total, 80K Used, 4096M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
28855 proxy 65 44 0 216M 165M ucond 1 0:00 0.49% squid
958 root 1 45 0 1412K 712K select 1 3:48 0.00% supervise
943 root 1 -8 0 1404K 464K piperd 1 1:14 0.00% readproctitle

inactive memori begitu besar, free memori hanya 22M ...
normalkah ?
dan pernah swap ke disk ...
juga pernah squid page fault erorr 1121 .. ;D, setelah di restart, error kembali ke 0, menambah ram sedang direncanakan ...
hanya nilai "1475M Inact" mengganggu ... dipake kemana saja
apa karena coss kah ...

terima kasih yah om ...

tomplee

@tomplee:

@grage95:

coba cek dulu perlnya ang sudah terinstall

pkg_info perl*

jika masih menggunakan perl 5.8.8, update perlnya ke versi 5.8.9

coba lusca paket baru lebih responsip hasil patch utak atik gathuk :) , lumayan responsif dan stabil,
cuman cache swap nya harus di delete dulu baru di squid -z,

http://squid-proxy-pkg.googlecode.com/files/pfsense-lusca-head-r14371_3.tbz (menggunakan perl-5.8.9_3.tbz) dan bisa jalan dengan perl-5.8.9_2.tbz

atau update versi pfsensenya,
versi 1.2.3 release lebih maknyuss paralel threadnya utk proxy daripada versi 1.2.2 release dan banyak lagi perbaikannya

Oke tak cobanya update lusca yang baru..

om grage tanya lagi..
referensi adzap saya dari sini http://forum.pfsense.org/index.php/topic,12737.0.html
kalau nurut turorial itu redirect_program /usr/local/libexec/adzap; di taroh si custom Custom Options squid, tapi kalo tak taruh di situ squidnya gak mau jalan .. kalau di taruh di tunning.conf bisa gak? redirect_program tu seharusnya naruh urutannya gimana?

grage95

@serangku:

om grage95 …
mohon pencerahan ...

spek mesin sbb :
2 gb ram, 1 disk utk pfsense (ata), 1 disk utk coss (sata/1gb), 2 disk utk aufs (sata/2x50gb)
hasil top

top -n 3

last pid: 35439; load averages: 0.00, 0.02, 0.00 up 9+00:17:31 16:15:10
121 processes: 1 running, 117 sleeping, 3 zombie

Mem: 248M Active, 1475M Inact, 193M Wired, 59M Cache, 112M Buf, 22M Free
Swap: 4096M Total, 80K Used, 4096M Free

PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND
28855 proxy 65 44 0 216M 165M ucond 1 0:00 0.49% squid
958 root 1 45 0 1412K 712K select 1 3:48 0.00% supervise
943 root 1 -8 0 1404K 464K piperd 1 1:14 0.00% readproctitle

inactive memori begitu besar, free memori hanya 22M ...
normalkah ?
dan pernah swap ke disk ...
juga pernah squid page fault erorr 1121 .. ;D, setelah di restart, error kembali ke 0, menambah ram sedang direncanakan ...
hanya nilai "1475M Inact" mengganggu ... dipake kemana saja
apa karena coss kah ...

terima kasih yah om ...

kalo menggunakan coss jangan sering di restart squidnya atau servernya down,
coss smenggunakan memory banyak pada asaat rebuilding swap,

[SHARE] Update Lusca Release - r14371 (November 18, 2009)

# dmesg | grep CPU

atacontrol list

dmesg | grep memory

swapinfo -k

su -m proxy -c "/usr/local/bin/squidstats.pl createdb"

su -m proxy -c "/usr/local/bin/squidstats.pl createdb"

squid -v

top -n 3

top -n 3

Used only with memory_pools on:

memory_pools_limit 50 MB

If set to a non-zero value, Squid will keep at most the specified

limit of allocated (but unused) memory in memory pools. All free()

requests that exceed this limit will be handled by your malloc

library. Squid does not pre-allocate any memory, just safe-keeps

objects that otherwise would be free()d. Thus, it is safe to set

memory_pools_limit to a reasonably high value even if your

configuration will use less memory.

If not set (default) or set to zero, Squid will keep all memory it

can. That is, there will be no limit on the total amount of memory

used for safe-keeping.

To disable memory allocation optimization, do not set

memory_pools_limit to 0. Set memory_pools to "off" instead.

An overhead for maintaining memory pools is not taken into account

when the limit is checked. This overhead is close to four bytes per

object kept. However, pools may actually save memory because of

reduced memory thrashing in your malloc library.