Python scrip for OWL-Intuition

wallabybob

I've a simplified version of the above scrip for seeking resolution here:

I notice you don't check the following calls for errors:


s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)

s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
s.bind((HOST, PORT))

Not that I know Python or its library, but the C calls of similar names all return error codes.

@vbhoj74:

mreq = struct.pack("4sl", socket.inet_aton(HELLO_GROUP), socket.INADDR_ANY)

The above might need some tweaking for FreeBSD: FreeBSD is not Linux.

@vbhoj74:

Set the interface to listen for multicast sent from the Owl Intuition box

s.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

Error return should be checked.

@vbhoj74:


data = s.recv(MSGBUFSIZE)

Assuming all the preceding calls were successful the error code returned MIGHT indicate there is no data currently available - a highly likely event if you have successfully put the socket into "non blocking" mode!

I would also use packet capture to verify your box is actually receiving data with destination IP address of the multicast address in the script and that the destination port in the data matches the destination port specified in the script.

vbhoj74

@stephenw10:

Well I'm completely in agreement with your goals. Perhaps with the exception of email. ;)

I would not like to have it send me a daily mail, but if it gets working maybe a weekly or monthly goal would serve the purpose.

@stephenw10:

I would have thought that long term logging is the most important feature of a device like this? You need to be able to compare your usage with an equivalent week/month from the previous year or two years or more. 14 months seems like a disappointingly short time.

I quote from their FAQ http://www.theowl.com/helpcentre/intuition.html "Detailed data will be available for 4 months and historical daily data will be available for a minimum of 13 months."

@stephenw10:

Can you not export the data from their logging software? Not ideal perhaps.
Since you are using the nano image where are you hoping to store the data long term?

There is a CSV download link on their website, but kind of do not like the manual process and hard to maintain. Nano seems a perfect choice to me as its only 5-6w, already on my network so I'm not adding any new device. I wish to just capture & store the last multicast of the day. OWL is already doing a fantastic job for short term data analysis in terms (<1Yr) of graphs, so would not need to capture everything that is being sent. But later on I would love to compare YoY figures which just seems very important too.

@stephenw10:

I know nothing of python (or PHP for that matter!) so cant help you with coding. I only suggested using php because I imagine almost everything required is installed by default so you won't have modify pfSense any more than is necessary. This would be useful in terms of making a package or if you have to reinstall it.

Well even i dont know anything about python or PHP, but i'm so willing to learn & make this working. :)

@stephenw10:

When you spoke to Owl did they not send you details of whatever protocol they are using?

I have a currentcost meter that is similar but connects to the box via serial. Potentially this is easier to talk to but still beyond my skills. :(

They were not helpful at all, they just answered that its beyond their support area.

@wallabybob:

Assuming all the preceding calls were successful the error code returned MIGHT indicate there is no data currently available - a highly likely event if you have successfully put the socket into "non blocking" mode!

I being a non-programmer now, this scrip motivated me to start with python. I did check each & every line of the scrip by running it in the python shell interpreter. Initially the non-blocking was not there, the interpreter then used to endlessly wait after the s.recv function, so I put it to non-blocking.

Surprisingly the s.recv function did capture the below sample 4 times before stopping.

 "<electricity id="443719000861"><signal rssi="-52" lqi="13"><battery level="100%"><chan id="0"><curr units="w">209.00</curr><day units="wh">2412.21</day></chan><chan id="1"><curr units="w">32.00</curr><day units="wh">527.07</day></chan><chan id="2"><curr units="w">32.00</curr><day units="wh">1505.04</day></chan></battery></signal></electricity>"

@wallabybob:

I would also use packet capture to verify your box is actually receiving data with destination IP address of the multicast address in the script and that the destination port in the data matches the destination port specified in the script.

Thus I was suspect of the same issue, if the OWL was not multicasting, I ran pftop on the pfsense and it was there. But it is a periodic (i think every 10 secs) multicast. Even looping the above s.recv statement does not capture the packets once it stops.

EDIT: ptop capture below:

udp   I 192.168.1.14:22600    224.192.32.19:22600   0:1     12    18     3  1062
udp   O 192.168.1.14:22600    224.192.32.19:22600   1:0     12    48     1   354

vbhoj74

friendly bump!

I tested the scrip removing the non-blocking on a raspberry-pi on my network, it works just fine. With the pfsense, it keeps on waiting for data but nothing comes up. Since running the scrip at the python shell captures the data randomly at times (only 4 times to be precise), I feel clueless.

vbhoj74

RESOLVED

There is nothing wrong with the code. There was some routing issues due to which the ethernet ports on the pfsense does not seems to capture the multicast packets. here is what I did:

Added a Gateway:
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1

Added a Route:
224.192.32.19/32 LocalNetwork - 192.168.1.1 Lanbridge

Dont know if there is a better way of achieving it.

I'll finetune the code and post it here if anyone would like to use it.

vbhoj74

I've successfully ported & modified the python code, have attached two files, owl.py and send_gmail.py.
(Original code credit goes to PiFan over http://www.raspberrypiusers.com/?p=7486 )

Here is what it does at its present form:

1. Captures all the multicast data of your OWL Intuition-LC, and saves it in a csv file. It saves phase1,2,3 & totals.
2. You can set a cost limit per day, exceeding which it sends an email. I've modified the original code to use gmail a/c for sending mails.
3. It mails a daily summary usage at EOD.

In future development I would probably want it to:

1. Maintain a weekly avg Kwh consumption.
2. email consumption details based on electricity supply billing cycles.
3. Weekly limits Vs daily limits.
4. Add a 4 line LCD display which shows live usage of all 3 phases. (Ambitious, at present this is beyond my scope of knowledge and would love some help)

Installation Steps:

1. Rename the attached files to .py extension. You may place both of them in /home
2. Edit both files, check the comment areas to modify.
3. #chmod +x /home/owl.py
4. #chmod +x /home/send_gmail.py
5. Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1
6. Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge
7. Pfsense>Diagnostic>Backup>Download Backup config.xml
find /system, and add just below:
<shellcmd>python /home/owl.py</shellcmd>
save the file structure and restore.

Notes:

Step 5 & 6 :
edit the gateway & route as per your local LAN IP and interface names.

owl.txt
send_gmail.txt

stephenw10

Nice one.
So you didn't have to install any further libraries or anything python related?

I don't like the look of those network mods. It's almost always a bad idea to add a gateway to the LAN interface. It makes pfSense treat it as a WAN.

Steve

vbhoj74

Steve,

Forgot to mention that you need to install the python package but no other libraries or anything else python related.

I installed python using the below commands at the shell prompt on an alix embedded system:

/etc/rc.conf_mount_rw
setenv PKG_TMPDIR /root/
pkg_add -r http://files.pfsense.org/packages/8/All/python27-2.7.2_3.tbz
/etc/rc.conf_mount_ro

I had to add the gateway to route 224.192.32.19 on the local interface, otherwise it just does not seems to capturing the packets. Maybe there is something wrong with the routing & needs further probing. Before this workaround I tried all kind of things, like opening the LAN firewall filter to all packets, disabling packet filter all together, checking up NAT if something is twisted up there, disabling VPNs, nothing seemed to work. Tried finding on this forum too if anything exists on multicast packet related on pfsense ports, did not find much except that its enabled by default on 2.0.1 version.

stephenw10

Ah. Thanks for the update. Why have you set PKG_TMPDIR to root?

If it works for you then I wouldn't worry about it. You might have problems with routing if you use multiwan though.
I feel sure there must be a better way of handling the multicast traffic though I can't think of anything right now. Multicast proxy perhaps?
Looking at your additions:

Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1

This looks like you are adding a gateway with the name 'LocalNetwork' into the interface 'Lanbridge' (I don't know what that interface consists of) where the gateway address is 192.168.1.1. That would normally be the address of the LAN interface is that now the address of Lanbridge? If so you have set the gateway as the interface address itself. :-\

Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge

So this looks like you have added a static route to send all traffic for 224.192.32.19/32 to the Localnetwork gateway on the Lanbridge interface but that is the Lanbridge interface and that is presumably the same interface this traffic came in on anyway? :-\

Confusing! ::)

Perhaps you could simply add a virtual IP to Lanbridge with the address 224.192.32.19. I'm pretty much a total noob when it comes to multicast though I confess.

Steve

vbhoj74

PKG_TMPDIR to root is probably a bad idea, it was a cut & paste from somewhere this forum, never thought what I was pasting. Changed it to /home/tmp should be good.

I got two LAN ports + wifi card all bridged together into bridge0 interface which I call LANBRIDGE. And your assumption is correct that 192.168.1.1 is the local IP of this interface.

Again correct that I'm trying to route back the traffic on the same interface it came up on. I'm a noob with multicasts too, what ever little I know, is that the ports should process the multicast packets until its filtered somewhere. Since pfsense is a router, it will not forward multicast onto another network interface which is not the case here. Ideally pfsense should receive the multicast traffic from the LAN port, which I think it is, else how will it route it back. Or maybe its not capturing the packets because i've bridged the ports ? a possible bug ?

Adding a multicast range IP to an interface does not sound a good idea, don't know if this really can be done.

stephenw10

@vbhoj74:

Adding a multicast range IP to an interface does not sound a good idea, don't know if this really can be done.

Mmm, yes. Adding a virtual IP that can receive/respond to multicast traffic might do it though. Or perhaps the proxy like I suggested. This must have been solved before. :-\

Steve

vbhoj74

Trying with IGMPProxy, some error it says and not adding to routing:

Dec 18 12:55:08	igmpproxy: Warn: MRT_DEL_MFC; Errno(49): Can't assign requested address
Dec 18 12:55:08	igmpproxy: Note: Removing MFC: 192.168.1.14 -> 224.192.32.19, InpVIf: 3
Dec 18 12:55:08	igmpproxy: Warn: age_table_entry: SIOCGETSGCNT failing for (192.168.1.14 224.192.32.19); Errno(49): Can't assign requested address
Dec 18 12:55:06	igmpproxy: Note: New origin for route 224.192.32.19 is 192.168.1.14, flood -1

also tried adding an Alias to the Lanbridge interface, which broke my system and I had to flash restore it. I have a modem attached on the WAN interface which I've configured using the virtual interface method and then natting it. Maybe if proxy does not work I'll try it with virtual interface method.

EDIT: Forgot that adding virtual interface works with only WLAN and with PPPoE WANs. So left with proxy solution which does not seem to work.

stephenw10

Try using a virtual IP rather than a virtual interface:
http://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses%3F
I would try IP Alias.

Steve

vbhoj74

That was my first try, as soon as I gave it a virtual ip alias it locked me out, saying i might me in a man in the middle attack, and then could not access the box at all. I tried it with a console cable but it looked in a crashed state, so I booted it, and it hung at every boot attempt I made. I've a vanilla pfsense flashed on another cflash, I'll try this again with this build to see how it goes.

Once I add an alias, what else do I need to do to make that subnet work. It's already passed on the firewall rules. I think i wold not need to NAT it, since this is IGMP subnet that we need to enable and not another routable subnet. Clueless what crashed my install.

stephenw10

I'm pretty much guessing at this point! I've never tried adding an IP Alias to a bridge interface, could be some incompatibility you've discovered. It's an unusual config to say the least.

Steve

vbhoj74

This document below does not seem to suggest usage of ip alias with ver 2 installs. This if for modem access configuration, but I guess provides a clue that ip alias may not work with ver 2?

http://doc.pfsense.org/index.php/Accessing_modem_from_inside_firewall

stephenw10

Nope that's a different reason. The way PPP connections are handled changed which meant no longer had to use a virtual IP, you can just use the real interface. In fact IP Alias capability became stronger with 2.0.

Steve

vbhoj74

I poped in a vanilla install which too had bridged interface (two physical interfaces excluding the wifi this time), and the system does not crash. However, the DHCP server does not start and seems to be expecting me to give it IP ranges in the alias subnet:

Dec 19 08:44:54	php: /status_services.php: The command '/usr/local/sbin/dhcpd -user dhcpd -group _dhcp -chroot /var/dhcpd -cf /etc/dhcpd.conf bridge0' returned exit code '1', the output was 'Internet Systems Consortium DHCP Server 4.2.3 Copyright 2004-2011 Internet Systems Consortium. All rights reserved. For info, please visit https://www.isc.org/software/dhcp/ bad range, address 192.168.1.200 not in subnet 224.192.0.0 netmask 255.255.0.0 If you did not get this software from ftp.isc.org, please get the latest from ftp.isc.org and install that before requesting help. If you did get this software from ftp.isc.org and have not yet read the README, please read it before requesting help. If you intend to request help from the dhcp-server@isc.org mailing list, please read the section on the README about submitting bug reports and requests for help. Please do not under any circumstances send requests for help directly to the authors of this software - please send them to the appropriate mailing list as desc
Dec 19 08:44:54	dhcpd: exiting.

This was another msg at pfsense login screen I dont think I should much worry about but is a sore in the eye :)

You are accessing this router by an IP address not configured locally, which may be forwarded by NAT or other means. 

If you did not setup this forwarding, you may be the target of a man-in-the-middle attack.

vbhoj74

I tried adding a simple alias to another embedded install, this time with address 192.168.x.0/24 range, allowed firewall rules from the alias subnet to any, also allowed any to the alias IP of the interface just in case I get lockout of admin access, saved it, was still working, rebooted the system and it was broken, never came up working. One thing to note is that, even this install had its LAN interface bridged. Really wanted the IP Alias thing to work and seems the right solution going forward, maybe I need to take time out and start with a factory image and see how it goes.

–-----------------
Program UPDATE

I did some bug fixes and upgrades to the python scrip which is now attached as ver 1.0.1 with changes as below:

1. Various bug fixes.
2. Writes two CSV files now, one as Event logger, another Day logger. Day logger just logs once at end of day.
3. You can control the frequency at which it logs to the event logger, presently it's set to log every 65th packet received. Approx writes once in 45-60 mins.
4. Added Currency Symbol to the cost.
5. Rounded figures to 2 decimal points.
6. Remember to change the NTP server to pool.ntp.org on pfsense, the default server runs couple of minutes late which kills the day logger & mail.
7. I changed the local time format to dd/mm/yy, you can probably change it back as required.

For the program to work, install python package if not already installed:

to INSTALL python ----

/etc/rc.conf_mount_rw
mkdir /home/tmp
setenv PKG_TMPDIR /home/tmp/
pkg_add -r http://files.pfsense.org/packages/8/All/python27-2.7.2_3.tbz
/etc/rc.conf_mount_ro

Installation Steps:

1. You may place both of them in /home and rename to *.py
2. Edit both files, check the comment areas to modify.
3. #chmod +x /home/owl.py
4. #chmod +x /home/send_gmail.py
5. Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1
6. Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge
7. Pfsense>Diagnostic>Backup>Download Backup config.xml
find /system, and add just below:
<shellcmd>python /home/owl.py</shellcmd>
save the file structure and restore.
8. Pfsense>System>general Setup>NTP time server> change to "pool.ntp.org"

Notes:

Step 5 & 6 :
edit the gateway & route as per your local LAN IP and interface names. Trying to find a better way of adding IP alias to the interface to get this working, until then the above works.

owl.txt
send_gmail.txt

vbhoj74

Steve, I removed the port from LAN bridge in which I had plugged in the owl gateway and put it in a separate network. Assigned virtual ip alias to this interface, did respective firewall rules and NAT, and voila, it works. I guess virtual IP does not work with bridged interfaces properly. Thanks!

–----

So we can just add a virtual IP address in the subnet 224.192.32.0/24 to the LAN interface on which the owl gateway is plugged to and omit steps 5 & 6 in the above post.

stephenw10

Nice. :)

I guess adding an Alias IP to a bridge interface is a pretty rare usage scenario.

Steve