Еще раз о ipcad+squid+lightsquid

rubic

Подскажите пожалуйста что я не правильно сделал в конфигах так как ipcad не пишет ничего в лог.
Процесс в памяти висит.

Сделайте как написано здесь: http://ru.doc.pfsense.org/index.php/%D0%9F%D0%BE%D0%B4%D1%81%D1%87%D0%B5%D1%82_%D1%82%D1%80%D0%B0%D1%84%D0%B8%D0%BA%D0%B0_%D1%81_%D0%BF%D0%BE%D0%BC%D0%BE%D1%89%D1%8C%D1%8E_Squid_%D0%B8_ipcad_%D0%B2_pfSense_1.2.3
а то вы как-то слишком уж творчески переработали оба руководства, приведя их к чему-то среднему))
С одной стороны:
"aggregate 80 into 0;" и "if ($5 != 0)"
а с другой:
" and not src port 80"
это вещи взаимоисключающие

Ilyuha

dr.gopher
rubic
Спасибо. Про remote shell забыл. Теперь работает.

Ilyuha

Вот еще что обнаружил: качал 100 мб с mirror.yandex.ru [213.180.204.183] по 21 порту а в логе вижу следующее

57319 это наверное мой сырц порт, зачем он посчитался как дест порт сервера?

rubic

21-й порт - это управление, а не закачка. То, что вы видите - особенность современной реализации ftp - passive mode. Сервер кагбэ говорит вам с какого порта качать, и вы качаете))
http://ru.wikipedia.org/wiki/FTP

Eugene

@rubic:

…То, что вы видите - особенность современной реализации ftp passive mode.

Современной? разве когда-то пассив по-другому работал? -)

rubic

казнить нельзя помиловать ))
исправит ли ситуацию установка тире между ftp и passive mode?
сомневается

Eugene

Не понял, ну да ладно - проехали -)

Antip

не понимаю в чём проблема, ipcad считает всё кроме нужного :)
/usr/local/etc/ipcad.conf

interface bge0 filter "ip and dst net 192.168.0.0/24 and not src net 192.168.0.0/24";
aggregate 192.168.0.0/24 strip 32;      /* Aggregate external networks */
rsh root@127.0.0.1 admin;
rsh root@127.0.0.1 backup;
rsh root@127.0.0.1;
#rsh root@127.0.0.1 deny;
rsh 127.0.0.1 view-only;

rsh ttl = 3;
rsh timeout = 30;
dumpfile = ipcad.dump;  # The file is inside chroot(), see below...

chroot = /usr/logs/ipcad;
pidfile = ipcad.pid;
memory_limit = 5m;

остальное как в статье

в ipcad.dump пишется типа:

77.73.25.114     192.168.x.y         1           72      0     0     1 bge0
 77.73.25.114     192.168.x.y        88        80843      0 65535     6 bge0
 10.0.0.1        192.168.x.y         9         2076  65535 65535    17 bge0
 77.73.25.114     192.168.x.z        97        80898      0 65535     6 bge0

10.0.0.1 ДНС сервер

в access.log например

1298034900.000 1 192.168.x.y TCP_MISS/200 366 CONNECT 74.125.232.81:0 - DIRECT/74.125.232.81 -
1298034900.000 1 192.168.x.z TCP_MISS/200 729 CONNECT 209.85.148.138:0 - DIRECT/209.85.148.138 -

в самсе трафик удваивается и отображаются и урл и по ip:0, но по другим портам-то нет!!! хотя постоянно работает аська, почта, фтп, ssh :( куда копать?!
У пользователей настроен шлюзом, т.е. прописан в сетевых подключениях.

dr.gopher

Дык укажите то, что вам нужно считать, раскоментировав -

#aggregate 1-19 into 65535;
#aggregate 20-21 into 21;
#aggregate 22-23 into 22;
#aggregate 25 into 25;
#aggregate 24 into 65535;
#aggregate 26-79 into 65535;
aggregate 80-81 into 0;
#aggregate 82-109 into 65535;
#aggregate 110 into 110;
#aggregate 111-442 into 65535;
#aggregate 443 into 443;
#aggregate 444-3127 into 65535;
aggregate 3128 into 0;
#aggregate 3129-65535 into 65535;

Antip

да, сори, по сути не указал самого важного)
в общем, даже если они закоментированы, то считать будет по всем портам, но не в этом суть, у меня вот так


aggregate 1-19 into 65535;
aggregate 20-21 into 21;
aggregate 22-23 into 22;
aggregate 25 into 25;
aggregate 24 into 65535;
aggregate 26-79 into 65535;
aggregate 80-81 into 0;
aggregate 82-109 into 65535;
aggregate 110 into 110;
aggregate 111-442 into 65535;
aggregate 443 into 443;
aggregate 444-3127 into 65535;
aggregate 3128 into 0;
aggregate 3129-65535 into 65535;

Закоментил для проверки, в ipcad.dump стало:


  Source           Destination    Packets        Bytes  SrcPt DstPt Proto   IF
 69.64.6.21       192.168.x.y         2           88      0  1310     6 bge0
 94.179.162.78    192.168.x.z         4          593      0 25669     6 bge0
 94.179.162.78    192.168.x.z         5          832      0 25666     6 bge0
 94.179.162.78    192.168.x.z         5          832      0 25665     6 bge0
 94.179.162.78    192.168.x.z         4          592      0 25667     6 bge0
 94.179.162.78    192.168.x.z         6          980      0 25668     6 bge0
 94.179.162.78    192.168.x.y         2           88      0  1308     6 bge0
 74.125.232.28    192.168.x.y         3          366      0  1304     6 bge0
 74.125.232.28    192.168.x.y         6          724      0  1300     6 bge0
 129.42.56.216    192.168.x.z         2          244      0 25663     6 bge0
 94.179.162.78    192.168.x.z         6          516      0 25662     6 bge0

в общем по порту отображается только для днс с самого сервера в виде
10.0.0.1 192.168.x.y 9 2076 65535 65535 17 bge0

значит по сути работает, но почему остальное не считает?!

Ilyuha

Граждане скажите пжл, что и где написать в ipcad.conf чтобы исключить из логирования один серый IP ?

Ilyuha

up

rubic

interface em0 filter "ip and dst net 192.168.0.0/24 and not dst net IP/32 and not src net 192.168.0.0/24 and not src port 80";
попробуйте так, но входящий трафик с 80 порта для этого IP все равно будет помещен в лог squid'ом, если не прописать IP в Bypass proxy for these source IPs в настройках squid

Broodval

А есть ли у кого небитая ссылка на RSH?
Тут http://forum.pfsense.org/index.php/topic,18366.msg94510.html#msg94510 все битые(.
Заранее спасибо!

DasTieRR

Обновил http://narod.ru/disk/7290606001/rsh.html

Broodval

У меня вот такая проблема:

ipcad

Opening bge0… [LCap] [ERSH] [4096] [bge0/interactive] Initialized as 1
Aggregate network 192.168.1.0/255.255.255.128 -> 255.255.255.255
Aggregate network 0.0.0.0/0.0.0.0 -> 255.255.255.255
Can't start: another instance running, pid=525
Can't initialize pid file /var/log/ipcad/ipcad.pid: Operation not permitted
Make sure you have . under /var/log/ipcad used as new root. man 2 chroot.

Подскажите куда копать?

zar0ku1

@Broodval:

У меня вот такая проблема:
Подскажите куда копать?

Can't initialize pid file /var/log/ipcad/ipcad.pid: Operation not permitted

проверь есть ли папка /var/log/ipcad
если есть проверь, есть ли права на запись и все

Broodval

Can't initialize pid file /var/log/ipcad/ipcad.pid: Operation not permitted

если есть проверь, есть ли права на запись и все

Спасибо что помогаете!
Папка есть, права на папку 0777, группа: whell, владелец: root.
На ipcad.pid и ipcad.dump выставил такие же права (0777), не помогло.
Пробовал различный варианты, которые были описаны по этой проблеме выше.

Мои текущие параметры:
1. Версия pfSense
1.2.3-release

2. Лог загрузки системы в части касающейся ipcad и rsh
less /var/log/dmesg.boot | grep ipcad
less /var/log/dmesg.boot | grep rsh
Ничего не выводят.

3. Права на ipcad.dump, ipcad.pid, rsh, tolog.sh
/var/log/ipcad/ipcad.dump: -rw–-----
/var/log/ipcad/ipcad.pid: -rwxrwxrwx
/root/tolog.sh: -rwxr-xr-x
/usr/bin/rsh: -r-xr-xr-x

4. ipcad.conf

#
# Configuration file for ipcad - Cisco IP accounting simulator daemon.
# Copyright (c) 2001, 2002, 2003, 2004, 2005
# 	Lev Walkin <vlm@lionet.info>.
#
# Please see ipcad.conf(5) for additional explanations.
# Please contact me if you have troubles configuring ipcad. My goal is to make
# initial configuration easier for new users, so your input is valuable.
#

##################
# GLOBAL OPTIONS #
##################

#
# Enable or disable capturing UDP and TCP port numbers, IP protocol and
# ICMP types for RSH output.
#
#     capture-ports {enable|disable} ;
#
# Enabling this will BREAK Cisco RSH output format compatibility,
# increase memory requirements and may slow down traffic processing.
# This option takes effect IMMEDIATELY, that is, it can be specified
# multiple times, even between interfaces configuration.
# This option has NO effect on NetFlow operation (NetFlow always captures
# port information).
#

capture-ports enable;

#
# Buffers to be used for transferring the data from the kernel,
# if applicable (BPF, ULOG).
# Using larger buffers may increase the performance but will
# affect responsiveness.
#
# buffers = <number>[{k|m}] ;
#
# Reasonable defaults are used if this parameter is not set.
#

## buffers = 64k;

#####################
# INTERFACE OPTIONS #
#####################

#
# interface <iface>[ promisc ] [ input-only ]
#			[ netflow-disable ] [ filter "<pcap_filter>" ] ;
# OR
# interface ulog group <group>[, group <group>...]
#			[ netflow-disabled ];
# OR
# interface ipq [ netflow-disabled ];	# man libipq(3)
# OR
# interface {divert|tee} port <divert-port># man divert(4)
#			[ input-only ] [ netflow-disabled ];
# OR
# interface file <tcpdump-output.pcap>[ netflow-disabled ];
#
# Options meaning:
#
# promisc:
# 	Put interface into promiscuous mode.
# 	This enables listening for the packets which are not destined for
# 	this host and thus ipcad will count and display all the traffic within
# 	the local network. Note that the interface might be in promiscuous mode
# 	for some other reason.
#
# input-only: 
# 	Use kernel feature of counting only incoming packets.
#
# netflow-sampled: 	(DO NOT ENABLE THIS OPTION, unless you have to!)
# 	If the NetFlow export mechanism is used, this option instructs
#	the interface to supply only one out of N packets to the NetFlow
#	accounting code, thus lowering the CPU requirements. The value of N
#	is configured explicitly in a NetFlow configuration section.
#	NOTE: This option is NOT used to enable NetFlow on the interface,
#	it just modifies the NetFlow behavior on this interface.
#	DO NOT ENABLE THIS OPTION, UNLESS YOU HAVE TO!
#
# netflow-disable: 
#	By default, all interfaces are included into NetFlow accounting.
#	This option is used to disable NetFlow on a particular interface.
#
# filter: 
# 	Install a custom filter on packets instead of basic
#	IP protocol filter. Requires libpcap (even if BPF is being used).
#	May be employed to eliminate CPU overhead on passing unnecessary
#	data between the kernel and user space (by installing the filter
#	directly into the kernel).
#
# NOTES:
#  * "input-only" directive must be supported by kernel.
#    Probably, you were noticed about it during the compilation process
#    if it was not supported.
#    FreeBSD 3.x and elder kernels do not support this feature.
#  * ULOG packet source (interface ulog) is supported under
#    Linux >= 2.4.18-pre8.
#    You should configure iptables to dump the packet stream
#    into the appropriate group, i.e.:
#        iptables -A OUTPUT -j ULOG --ulog-nlgroup <group>#    Given ULOG groups will be OR'ed together.
#  * A wildcard (*) may be specified as part of an interface name.
#

interface bge0 filter "ip and dst net 10.10.200.0/24 and not src net 10.10.200.0/24";

#
# aggregate <ip>/ <masklen>strip <maskbits>;
#
# Aggregate addresses from the specified network (<ip>/<masklen>),
# by AND'ing with specified mask (<maskbits>).
#
#

aggregate 10.10.200.0/24 strip 32; /* Don't aggregate internal range */
aggregate 0.0.0.0/0 strip 32;	/* Aggregate external networks */

#
# aggregate <port_range_start>[-<port_range_end>] into <port>;
#
# Aggregate port numbers. Meaningful only if capture-ports is enabled.
#

#aggregate 1024-65535	into 65535;	/* Aggregate wildly */
#aggregate 3128-3128	into 3128;	/* Protect these ports */
#aggregate 150-1023	into 1023;	/* General low range */
aggregate 3128-3128 into 0;
aggregate 80-80 into 0;
##########################
# NetFlow EXPORT OPTIONS #
##########################

#
# Enable Cisco NetFlow export method.
# NetFlow uses UDP to feed flow information to the receiver.
# If the destination is not specified, NetFlow is disabled.
#

# netflow export destination 127.0.0.1 9996;
netflow export version 5;	# NetFlow export format version {1|5}
netflow timeout active 30;	# Timeout when flow is active, in minutes
netflow timeout inactive 15;	# Flow inactivity timeout, in seconds
netflow engine-type 73;		# v5 engine_type; 73='I' for "IPCAD"
netflow engine-id 1;		# Useful to differentiate multiple ipcads.

# The following option is enabled by the "netflow-sampled" interface flag.
#netflow sampling-mode packet-interval 10;   # 1 out of 10 packets accounted
# DO NOT ENABLE THIS UNLESS YOU KNOW WHAT ARE YOU DOING.

#
# NetFlow protocol exports an SNMP id instead of the interface name
# (i.e., "eth0", "ppp32"). The following statements options define
# mapping between the interface names and a set of "SNMP identifiers".
#
netflow ifclass eth mapto 0-99;		# i.e., "eth1"->1, "eth3"->3
netflow ifclass fxp mapto 0-99;		# i.e., "fxp4"->4, "fxp0"->0
netflow ifclass ppp mapto 100-199;	# i.e., "ppp32"->532, "ppp7"->507
netflow ifclass gre mapto 200-299;
netflow ifclass tun mapto 300-399;	# i.e., "tun0"->300

######################
# RSH SERVER OPTIONS #
######################

#
# Enable RSH Server:
#
#   rsh {enable|yes|on|disable|no|off} [at <listen_ip>];
#
# If "at <listen_ip>" omitted, rsh server listens on IP address 0.0.0.0,
# which may be undesirable.
#

rsh enable at 127.0.0.1;

#
# RSH access rules:
#
# rsh [<user>@] <host_addr>{admin|backup|[default]|view-only|deny} ;
#

rsh root@127.0.0.1 admin;	/* Can shutdown ipcad */
rsh root@127.0.0.1 backup;	/* Can dump/restore/import accounting table */
rsh root@127.0.0.1;		/* Can view and modify accounting tables */
/* Note the order! */
#rsh luser@127.0.0.1 deny;	/* Deny this user from even viewing tables */
rsh 127.0.0.1 view-only;	/* Other users can view current tables */

# Keep IP packet time to live reasonably low to avoid remote attacks.
# (The rsh client must reside no more than three hops away from the
# router running ipcad.)
rsh ttl = 3;

# Set rsh timeout for the same purpose.
rsh timeout = 30;

#
# Dump active IP accounting table to this file on exit and read on startup.
# (read about -s and -r options in ipcad(8) manual page)
# NOTE: This setting has no effect on NetFlow operation. The flow cache
#       contents are flushed to the collector upon ipcad termination.
#

dumpfile = /var/log/ipcad/ipcad.dump;	# The file is inside chroot(), see below...

#################
# OTHER OPTIONS #
#################

#
# Chroot to this directory before processing.
#
# Of course, you could disable chroot()'ing by commenting it out,
# but it is not recommended, so I left this confusing default
# to encourage you to change it.
#

chroot = /var/log/ipcad;

#
# File to keep getpid() in it. ipcad will also hold a lock.
#
# WARNING: Pidfile is created AFTER chroot()'ing, so if you're using
# chroot statement above, make sure the path to the pidfile exists
# inside chrooted environment.
#

pidfile = ipcad.pid;

#
# UID/GID privileges dropping
# Please note: RSH service will be UNAVAILABLE when uid is not zero.
# Use it only when you know what are you doing (i.e., NetFlow without RSH).
#
# uid = 65534;
# gid = 65534;

#
# Few useful settings.
#
#
# Memory usage limit for storing per-stream entries.
# 
# memory_limit = <number>[{k|m|e}] ;
# Where k, m and g are for kilobytes, megabytes or table "entries".
#

memory_limit = 10m;</number></host_addr></user></listen_ip></listen_ip></port></port_range_end></port_range_start></maskbits></masklen></ip></maskbits></masklen></ip></group></tcpdump-output.pcap></divert-port></group></group></pcap_filter></iface></number></vlm@lionet.info>

5. tolog.sh

#!/bin/sh
net1="10.10.200"
ttime=`/usr/bin/rsh localhost sh ip acco | grep 'Accounting data saved' | awk '{print ($4)}'`
rsh localhost clear ip accounting
rsh localhost show ip accounting checkpoint | grep $net1 | awk -v vtime=$ttime '{print (vtime".000",1,$2,"TCP_MISS/200",$4,"CONNECT",$1":"$5,"-","DIRECT/"$1,"-")}' >> /var/squid/log/access.log

6. Вырезка из config.xml с настройками cron:

 <cron><minute>0</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 newsyslog 
		 <minute>1,31</minute>
			<hour>0-5</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 adjkerntz -a 
		 <minute>1</minute>
			<hour>3</hour>
			<mday>1</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /etc/rc.update_bogons.sh 
		 <minute>*/60</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 sshlockout 
		 <minute>1</minute>
			<hour>1</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /etc/rc.dyndns.update 
		 <minute>*/60</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/nice -n20 /usr/local/sbin/expiretable -v -t 3600 virusprot 
		 <minute>*/5</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/local/bin/checkreload.sh 
		 <minute>*/5</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/etc/ping_hosts.sh 
		 <minute>*/140</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/local/sbin/reset_slbd.sh 
		 <minute>*/1</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/root/tolog.sh 
		 <task_name>lightsquid_parser_today</task_name>
			<minute>*/30</minute>
			<hour>*</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/perl /usr/local/www/lightsquid/lightparser.pl today 
		 <task_name>lightsquid_parser_yesterday</task_name>
			<minute>15</minute>
			<hour>0</hour>
			<mday>*</mday>
			<month>*</month>
			<wday>*</wday>
			<who>root</who>
			<command></command>/usr/bin/perl /usr/local/www/lightsquid/lightparser.pl yesterday</cron>

7. squid mode и к каким интерфесам привязан:
Transparent, привязан к LAN (bge0 - 10.10.200.10)

8. system лог на тему squid parent process
less /var/log/system.log | grep squid: Mar 13 12:06:16 pfSense squid[906]: Squid Parent: child process 909 started

DasTieRR

А Вы ничего и не считаете:
#aggregate 1024-65535 into 65535; /* Aggregate wildly /
#aggregate 3128-3128 into 3128; / Protect these ports /
#aggregate 150-1023 into 1023; / General low range */
aggregate 3128-3128 into 0;
aggregate 80-80 into 0;

Добавьте строчку и попробуйте:
aggregate 443 into 443

После загрузки системы в консоли наберите top - на моей рабочей конфигурации светится ipcad.

P.S. На всякий случай перечитайте форум, т.к. когда я лично повторно настраивал счётчик я забыл как это делать и пришлось поднимать инфу на форуме. (первые 3 страницы как минимум)

Broodval

Строчку добавил. Но мня больше волнует вот это:

# ipcad
Opening bge0... [LCap] [ERSH] [4096] [bge0/interactive] Initialized as 1
Aggregate network 10.10.200.0/255.255.255.0 -> 255.255.255.255
Aggregate network 0.0.0.0/0.0.0.0 -> 255.255.255.255
Aggregate ports 3128..3128 into 0
Aggregate ports 80..80 into 0
Aggregate ports 443..443 into 443
Configured RSH Server listening at 127.0.0.1
Can't start: another instance running, pid=521
Can't initialize pid file /var/log/ipcad/ipcad.pid: Operation not permitted
Make sure you have . under /var/log/ipcad used as new root. man 2 chroot.

и то, что Squid ничего не ложит в лог.