Python scrip for OWL-Intuition

stephenw10 · Dec 17, 2012, 1:15 PM

Nice one.
So you didn't have to install any further libraries or anything python related?

I don't like the look of those network mods. It's almost always a bad idea to add a gateway to the LAN interface. It makes pfSense treat it as a WAN.

Steve

vbhoj74 · Dec 17, 2012, 1:46 PM

Steve,

Forgot to mention that you need to install the python package but no other libraries or anything else python related.

I installed python using the below commands at the shell prompt on an alix embedded system:

/etc/rc.conf_mount_rw
setenv PKG_TMPDIR /root/
pkg_add -r http://files.pfsense.org/packages/8/All/python27-2.7.2_3.tbz
/etc/rc.conf_mount_ro

I had to add the gateway to route 224.192.32.19 on the local interface, otherwise it just does not seems to capturing the packets. Maybe there is something wrong with the routing & needs further probing. Before this workaround I tried all kind of things, like opening the LAN firewall filter to all packets, disabling packet filter all together, checking up NAT if something is twisted up there, disabling VPNs, nothing seemed to work. Tried finding on this forum too if anything exists on multicast packet related on pfsense ports, did not find much except that its enabled by default on 2.0.1 version.

stephenw10 · Dec 17, 2012, 3:29 PM

Ah. Thanks for the update. Why have you set PKG_TMPDIR to root?

If it works for you then I wouldn't worry about it. You might have problems with routing if you use multiwan though.
I feel sure there must be a better way of handling the multicast traffic though I can't think of anything right now. Multicast proxy perhaps?
Looking at your additions:

Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1

This looks like you are adding a gateway with the name 'LocalNetwork' into the interface 'Lanbridge' (I don't know what that interface consists of) where the gateway address is 192.168.1.1. That would normally be the address of the LAN interface is that now the address of Lanbridge? If so you have set the gateway as the interface address itself. :-\

Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge

So this looks like you have added a static route to send all traffic for 224.192.32.19/32 to the Localnetwork gateway on the Lanbridge interface but that is the Lanbridge interface and that is presumably the same interface this traffic came in on anyway? :-\

Confusing! ::)

Perhaps you could simply add a virtual IP to Lanbridge with the address 224.192.32.19. I'm pretty much a total noob when it comes to multicast though I confess.

Steve

vbhoj74 · Dec 17, 2012, 4:52 PM

PKG_TMPDIR to root is probably a bad idea, it was a cut & paste from somewhere this forum, never thought what I was pasting. Changed it to /home/tmp should be good.

I got two LAN ports + wifi card all bridged together into bridge0 interface which I call LANBRIDGE. And your assumption is correct that 192.168.1.1 is the local IP of this interface.

Again correct that I'm trying to route back the traffic on the same interface it came up on. I'm a noob with multicasts too, what ever little I know, is that the ports should process the multicast packets until its filtered somewhere. Since pfsense is a router, it will not forward multicast onto another network interface which is not the case here. Ideally pfsense should receive the multicast traffic from the LAN port, which I think it is, else how will it route it back. Or maybe its not capturing the packets because i've bridged the ports ? a possible bug ?

Adding a multicast range IP to an interface does not sound a good idea, don't know if this really can be done.

stephenw10 · Dec 17, 2012, 9:56 PM

@vbhoj74:

Adding a multicast range IP to an interface does not sound a good idea, don't know if this really can be done.

Mmm, yes. Adding a virtual IP that can receive/respond to multicast traffic might do it though. Or perhaps the proxy like I suggested. This must have been solved before. :-\

Steve

vbhoj74 · Dec 18, 2012, 9:10 AM

Trying with IGMPProxy, some error it says and not adding to routing:

Dec 18 12:55:08	igmpproxy: Warn: MRT_DEL_MFC; Errno(49): Can't assign requested address
Dec 18 12:55:08	igmpproxy: Note: Removing MFC: 192.168.1.14 -> 224.192.32.19, InpVIf: 3
Dec 18 12:55:08	igmpproxy: Warn: age_table_entry: SIOCGETSGCNT failing for (192.168.1.14 224.192.32.19); Errno(49): Can't assign requested address
Dec 18 12:55:06	igmpproxy: Note: New origin for route 224.192.32.19 is 192.168.1.14, flood -1

also tried adding an Alias to the Lanbridge interface, which broke my system and I had to flash restore it. I have a modem attached on the WAN interface which I've configured using the virtual interface method and then natting it. Maybe if proxy does not work I'll try it with virtual interface method.

EDIT: Forgot that adding virtual interface works with only WLAN and with PPPoE WANs. So left with proxy solution which does not seem to work.

stephenw10 · Dec 18, 2012, 1:11 PM

Try using a virtual IP rather than a virtual interface:
http://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses%3F
I would try IP Alias.

Steve

vbhoj74 · Dec 18, 2012, 2:59 PM

That was my first try, as soon as I gave it a virtual ip alias it locked me out, saying i might me in a man in the middle attack, and then could not access the box at all. I tried it with a console cable but it looked in a crashed state, so I booted it, and it hung at every boot attempt I made. I've a vanilla pfsense flashed on another cflash, I'll try this again with this build to see how it goes.

Once I add an alias, what else do I need to do to make that subnet work. It's already passed on the firewall rules. I think i wold not need to NAT it, since this is IGMP subnet that we need to enable and not another routable subnet. Clueless what crashed my install.

stephenw10 · Dec 18, 2012, 3:21 PM

I'm pretty much guessing at this point! I've never tried adding an IP Alias to a bridge interface, could be some incompatibility you've discovered. It's an unusual config to say the least.

Steve

vbhoj74 · Dec 18, 2012, 4:34 PM

This document below does not seem to suggest usage of ip alias with ver 2 installs. This if for modem access configuration, but I guess provides a clue that ip alias may not work with ver 2?

http://doc.pfsense.org/index.php/Accessing_modem_from_inside_firewall

stephenw10 · Dec 18, 2012, 5:39 PM

Nope that's a different reason. The way PPP connections are handled changed which meant no longer had to use a virtual IP, you can just use the real interface. In fact IP Alias capability became stronger with 2.0.

Steve

vbhoj74 · Dec 19, 2012, 3:20 AM

I poped in a vanilla install which too had bridged interface (two physical interfaces excluding the wifi this time), and the system does not crash. However, the DHCP server does not start and seems to be expecting me to give it IP ranges in the alias subnet:

Dec 19 08:44:54	php: /status_services.php: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf bridge0' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.3 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ bad range, address 192.168.1.200 not in subnet 224.192.0.0 netmask 255.255.0.0 If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. If you intend to request help from the dhcp-server@isc.org mailing list, please read the section on the README about submitting bug reports and requests for help. Please do not under any circumstances send requests for help directly to the authors of this software - please send them to the appropriate mailing list as desc
Dec 19 08:44:54	dhcpd: exiting.

This was another msg at pfsense login screen I dont think I should much worry about but is a sore in the eye :)

You are accessing this router by an IP address not configured locally, which may be forwarded by NAT or other means. 

If you did not setup this forwarding, you may be the target of a man-in-the-middle attack.

vbhoj74 · Dec 20, 2012, 6:43 AM

I tried adding a simple alias to another embedded install, this time with address 192.168.x.0/24 range, allowed firewall rules from the alias subnet to any, also allowed any to the alias IP of the interface just in case I get lockout of admin access, saved it, was still working, rebooted the system and it was broken, never came up working. One thing to note is that, even this install had its LAN interface bridged. Really wanted the IP Alias thing to work and seems the right solution going forward, maybe I need to take time out and start with a factory image and see how it goes.

–-----------------
Program UPDATE

I did some bug fixes and upgrades to the python scrip which is now attached as ver 1.0.1 with changes as below:

1. Various bug fixes.
2. Writes two CSV files now, one as Event logger, another Day logger. Day logger just logs once at end of day.
3. You can control the frequency at which it logs to the event logger, presently it's set to log every 65th packet received. Approx writes once in 45-60 mins.
4. Added Currency Symbol to the cost.
5. Rounded figures to 2 decimal points.
6. Remember to change the NTP server to pool.ntp.org on pfsense, the default server runs couple of minutes late which kills the day logger & mail.
7. I changed the local time format to dd/mm/yy, you can probably change it back as required.

For the program to work, install python package if not already installed:

to INSTALL python ----

/etc/rc.conf_mount_rw
mkdir /home/tmp
setenv PKG_TMPDIR /home/tmp/
pkg_add -r http://files.pfsense.org/packages/8/All/python27-2.7.2_3.tbz
/etc/rc.conf_mount_ro

Installation Steps:

1. You may place both of them in /home and rename to *.py
2. Edit both files, check the comment areas to modify.
3. #chmod +x /home/owl.py
4. #chmod +x /home/send_gmail.py
5. Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1
6. Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge
7. Pfsense>Diagnostic>Backup>Download Backup config.xml
find /system, and add just below:
<shellcmd>python /home/owl.py</shellcmd>
save the file structure and restore.
8. Pfsense>System>general Setup>NTP time server> change to "pool.ntp.org"

Notes:

Step 5 & 6 :
edit the gateway & route as per your local LAN IP and interface names. Trying to find a better way of adding IP alias to the interface to get this working, until then the above works.

owl.txt
send_gmail.txt

vbhoj74 · Dec 21, 2012, 10:48 AM

Steve, I removed the port from LAN bridge in which I had plugged in the owl gateway and put it in a separate network. Assigned virtual ip alias to this interface, did respective firewall rules and NAT, and voila, it works. I guess virtual IP does not work with bridged interfaces properly. Thanks!

–----

So we can just add a virtual IP address in the subnet 224.192.32.0/24 to the LAN interface on which the owl gateway is plugged to and omit steps 5 & 6 in the above post.

stephenw10 · Dec 21, 2012, 1:13 PM

Nice. :)

I guess adding an Alias IP to a bridge interface is a pretty rare usage scenario.

Steve

vbhoj74 · Jan 3, 2013, 8:38 AM

Program UPDATE
–--------------

Bug fixes.
Check & create sub-directories by itself, no need to create directories manually
Support for db file log of daily kwH using sqlite
Now requires sqlite port
It has now two parts that remains resident, owl.py which write the log files and:
responder.py which responds to email query.
You can send an email (ID as defined in responder.py) with subject "OWL" and from and to dates in 1st & 2nd line of the mail body in the format yyyy-mm-dd as a query. The code will reply with an attached txt file containing statement of usage within dates, total kWH and avg kWH.

Version 1.0.2

Installation Steps:
1. Download and UNzip owl.rar https://dl.dropbox.com/u/2185098/generic/owl.rar
2. You may place all files in /home and rename to .py
2. Edit all .py files, check the comment areas to modify.
3. #chmod +x /home/.py
4. Add Firewall>Virtual IP>IP Alias 224.192.32.20/24 to your local interface
5. Add Firewall Rules>local interface:
Allow UDP * * 224.192.32.19 * * note
Allow IGMP * * * * * none
Allow * 224.192.32.19/24 * * * default none
6. Pfsense>Diagnostic>Backup>Download Backup config.xml
find /system, and add just below:
<shellcmd>python /home/owl.py &</shellcmd>
save the file structure and restore.
7. Pfsense>System>general Setup>NTP time server> change to "pool.ntp.org"

Notes:

to INSTALL python with sqlite port ----
/etc/rc.conf_mount_rw
mkdir /home/tmp
setenv PKG_TMPDIR /home/tmp/
pkg_add -r http://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/8.1-RELEASE/packages/All/py26-sqlite3-2.6.5_1.tbz
/etc/rc.conf_mount_ro

I would be glad to know if you have used the code or taken any help from it.

EDIT: It seems to be working now, I made responder.py a subprocess of the main scrip instead of trying to start both the scrips using shellcmd.