Python scrip for OWL-Intuition

vbhoj74 · Dec 5, 2012, 10:52 AM

I've an Own Intuition electricity monitor that sends multicast on the network which I need to capture on the pfsense 2.0.1 (nanoBSD) using python. There is already a scrip for the raspberryPi for the same which I'm trying to port on the pfsense.

http://www.raspberrypiusers.com/?p=7484

The modified scrip that I'm trying to run:

#! /usr/bin/env python2.7

# pftop logs:
# pfTop: Up State 1-63/63, View: default, Order: bytes
# PR    D SRC                   DEST                 STATE   AGE   EXP  PKTS BYTES
# tcp   I 192.168.1.206:1150    91.197.33.240:5222    4:4  70120 86390 28998 9126K
# udp   O 192.168.1.206:22600   224.192.32.19:22600   1:0     90     0     2   711
# udp   I 192.168.1.206:22600   224.192.32.19:22600   0:1     30     0     3  1065
# sample receive buffer:
# "<electricity id="443719000861"><signal rssi="-52" lqi="13"><battery level="100%"><chan id="0"><curr units="w">209.00</curr><day units="wh">2412.21</day></chan><chan id="1"><curr units="w">32.00</curr><day units="wh">527.07</day></chan><chan id="2"><curr units="w">32.00</curr><day units="wh">1505.04</day></chan></battery></signal></electricity>"

# Echo server program
import re
import socket
import struct
import time
from email.mime.text import MIMEText
from subprocess import Popen,PIPE

# define the constants
HELLO_GROUP='224.192.32.19' # this is the multicast address used by Owl
HOST = ''                 # Symbolic name meaning all available interfaces
PORT = 22600              # Arbitrary non-privileged port
MSGBUFSIZE=800
DAY_RATE=0.36    # 36p standing charge per day
UNIT_PRICE=8.43    # 10.37p per unit

# setup the network connection
# make it UDP
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)

# enable the socket to be reused, esp. if were recovering from an error
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
s.bind((HOST, PORT))
mreq = struct.pack ("4sl", socket.inet_aton(HELLO_GROUP), socket.INADDR_ANY)

# Set the interface to listen for multicast sent from the Owl Intuition box
s.setsockopt (socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
# added by me
s.setblocking(0)

outString = "Pi Electricity Monitor Initialised"
msg=MIMEText (outString);
msg["From"] = "mypi@here.com"
msg["To"] = "vbhoj74@gmail.com" # put your email in here
msg["Subject"] = "Electricity used"
# p= Popen(["/usr/sbin/sendmail", "-t"], stdin=PIPE)
# p.communicate(msg.as_string())

# receive the data
#print(time.strftime('%x %X'))
msgNotSent = 1
day = time.strftime ('%d')
prevday = day
prevCost = 0 #just in case we meet this var in our testing

while True:

      try:
         buffer = s.recv(MSGBUFSIZE)

         # the following extracts the watts being consumed and total watt/hrs being spent
         matchObj = re.match (r'.*chan id=\'0\'><curr units="\'w\'">([0-9]+\.[0-9]+)</curr><day units="\'wh\'">([0-9]+.[0-9]+)</day>.*', buffer, re.I|re.S)

         # test line
         print matchObj

         if matchObj:
            watts = matchObj.group(1)
            watt_hrs = matchObj.group(2)
            # divide by 1000 to get Kwh
            kwhs = float(watt_hrs)/1000
            cost = DAY_RATE + (kwhs*UNIT_PRICE)
            localtime = time.strftime('%x %X')
            f = open ('/root/energy/electricData.csv','a') # you can change the path, etc
            outstr = localtime
            #localtime+","+watts+","+kwhs+","+ cost+"\n")
            f.write (outstr+","+watts+","+str(kwhs)+","+str(cost)+"\n")
            f.close()

         if cost > 50 and msgNotSent:
            outString= "Electricity usage cost today "+ outstr + " is " + str(cost) + "\n"
            msg=MIMEText (outString);
            msg["From"] = "mypi@here.com"
            msg["To"] = "vbhoj74@gmail.com"
            msg["Subject"] = "Electricity used"
            # p= Popen(["/usr/sbin/sendmail", "-t"], stdin=PIPE)
            # p.communicate(msg.as_string())
            msgNotSent = 0
            day = time.strftime('%d')

         if day <> prevday:
            # send email summary
            outString= "Summary of electricity usage cost yesterday "+ outstr + " is " + str(prevCost) + "\n"
            msg=MIMEText (outString);
            msg["From"] = "mypi@here.com"
            msg["To"] = "vbhoj74@gmail.com"
            msg["Subject"] = "Electricity used yesterday"
            # p= Popen(["/usr/sbin/sendmail", "-t"], stdin=PIPE)
            # p.communicate(msg.as_string())
            msgNotSent = 1
            prevday=day
            prevCost=cost

      except socket.error as err:
        print(err)
        # [Errno 11] Resource temporarily unavailable
        # sys.exit()

        # setup the network connection
        # make it UDP        
        s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
        # enable the socket to be reused, esp. if were recovering from an error
        s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
        s.bind((HOST, PORT))
        mreq = struct.pack ("4sl", socket.inet_aton(HELLO_GROUP), socket.INADDR_ANY)        
        # Set the interface to listen for multicast sent from the Owl Intuition box
        s.setsockopt (socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)
        # added by me
        s.setblocking(0)

      # finally:
        # s.close()

      # test line
      print buffer

I've commented out the options of sending mails etc as I'm stuck on s.recv(MSGBUFSIZE), python used to endlessly wait on that line, so I added s.setblocking(0) to get [Errno 35] Resource temporarily unavailable. I tried the run the scrip in the python CLI interpreter step my step and I was able to capture packets 4 times before it again started to give the same error. I'm new to python thus not able to troubleshoot this on my own. I'm looking for help from this forum to get this working. Thanks.

vbhoj74 · Dec 9, 2012, 7:14 AM

I've a simplified version of the above scrip for seeking resolution here:

#! /usr/bin/env python2.7
import socket
import struct

# define the constants
HELLO_GROUP='224.192.32.19' # this is the multicast address used by Owl

HOST = socket.gethostbyname(socket.gethostname())

# Or maybe just define host using:
# HOST = ''                 # Symbolic name meaning all available interfaces

PORT = 22600              # Arbitrary non-privileged port
MSGBUFSIZE=800

# setup the network connection
# make it UDP
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)

# enable the socket to be reused, esp. if were recovering from an error
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
s.bind((HOST, PORT))
mreq = struct.pack("4sl", socket.inet_aton(HELLO_GROUP), socket.INADDR_ANY)

# Set the interface to listen for multicast sent from the Owl Intuition box
s.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

# added to show error msg
s.setblocking(0)

data = s.recv(MSGBUFSIZE)

print data

this code above is returning the same error :

Traceback (most recent call last):
  File "./testsoc.py", line 32, in <module>data = s.recv(MSGBUFSIZE)
socket.error: [Errno 35] Resource temporarily unavailable</module>

Any help in the right direction will be greatly appreciated.

stephenw10 · Dec 9, 2012, 3:29 PM

If I was trying this I would be tempted to try to re-write it in php since that's what almost everything else in pfSense is written in. OWL seem willing to offer quite a bit of help if you email asking them.
However what are you hoping to achieve with this in the end?

Steve

vbhoj74 · Dec 9, 2012, 4:03 PM

I was a programmer in the middle to late 80s, so do not know anything of the new programming languages, need some catching up. Since there was already a code written for OWL in python I thought it would be easier to port it on pfsense and modify it to my requirements.

I tried getting some help from OWL, they put their foot down saying its out of their support area.

Here is what I'm trying to achieve:

OWL maintains historical data for 14 months as per their website, I would like to keep my own local database.
with OWL's website there is no way to find total cost by dates, which I think is very useful to verify/anticipate energy bills. They only help you compare one's usage by days and months.
as with the initial code that's been written, I would like to be mailed when a days energy consumption crosses a limit that I set. Various kind of active monitoring can be done.

stephenw10 · Dec 9, 2012, 5:37 PM

Well I'm completely in agreement with your goals. Perhaps with the exception of email. ;)
I would have thought that long term logging is the most important feature of a device like this? You need to be able to compare your usage with an equivalent week/month from the previous year or two years or more. 14 months seems like a disappointingly short time.
Can you not export the data from their logging software? Not ideal perhaps.

Since you are using the nano image where are you hoping to store the data long term?

I know nothing of python (or PHP for that matter!) so cant help you with coding. I only suggested using php because I imagine almost everything required is installed by default so you won't have modify pfSense any more than is necessary. This would be useful in terms of making a package or if you have to reinstall it.

When you spoke to Owl did they not send you details of whatever protocol they are using?

I have a currentcost meter that is similar but connects to the box via serial. Potentially this is easier to talk to but still beyond my skills. :(

Steve

wallabybob · Dec 9, 2012, 8:28 PM

@vbhoj74:

I've a simplified version of the above scrip for seeking resolution here:

I notice you don't check the following calls for errors:

@vbhoj74:


s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)

s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR,1)
s.bind((HOST, PORT))

Not that I know Python or its library, but the C calls of similar names all return error codes.

@vbhoj74:

mreq = struct.pack("4sl", socket.inet_aton(HELLO_GROUP), socket.INADDR_ANY)

The above might need some tweaking for FreeBSD: FreeBSD is not Linux.

@vbhoj74:

Set the interface to listen for multicast sent from the Owl Intuition box

s.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq)

Error return should be checked.

@vbhoj74:


data = s.recv(MSGBUFSIZE)

Assuming all the preceding calls were successful the error code returned MIGHT indicate there is no data currently available - a highly likely event if you have successfully put the socket into "non blocking" mode!

I would also use packet capture to verify your box is actually receiving data with destination IP address of the multicast address in the script and that the destination port in the data matches the destination port specified in the script.

vbhoj74 · Dec 10, 2012, 2:40 AM

@stephenw10:

Well I'm completely in agreement with your goals. Perhaps with the exception of email. ;)

I would not like to have it send me a daily mail, but if it gets working maybe a weekly or monthly goal would serve the purpose.

@stephenw10:

I would have thought that long term logging is the most important feature of a device like this? You need to be able to compare your usage with an equivalent week/month from the previous year or two years or more. 14 months seems like a disappointingly short time.

I quote from their FAQ http://www.theowl.com/helpcentre/intuition.html "Detailed data will be available for 4 months and historical daily data will be available for a minimum of 13 months."

@stephenw10:

Can you not export the data from their logging software? Not ideal perhaps.
Since you are using the nano image where are you hoping to store the data long term?

There is a CSV download link on their website, but kind of do not like the manual process and hard to maintain. Nano seems a perfect choice to me as its only 5-6w, already on my network so I'm not adding any new device. I wish to just capture & store the last multicast of the day. OWL is already doing a fantastic job for short term data analysis in terms (<1Yr) of graphs, so would not need to capture everything that is being sent. But later on I would love to compare YoY figures which just seems very important too.

@stephenw10:

I know nothing of python (or PHP for that matter!) so cant help you with coding. I only suggested using php because I imagine almost everything required is installed by default so you won't have modify pfSense any more than is necessary. This would be useful in terms of making a package or if you have to reinstall it.

Well even i dont know anything about python or PHP, but i'm so willing to learn & make this working. :)

@stephenw10:

When you spoke to Owl did they not send you details of whatever protocol they are using?

I have a currentcost meter that is similar but connects to the box via serial. Potentially this is easier to talk to but still beyond my skills. :(

They were not helpful at all, they just answered that its beyond their support area.

@wallabybob:

Assuming all the preceding calls were successful the error code returned MIGHT indicate there is no data currently available - a highly likely event if you have successfully put the socket into "non blocking" mode!

I being a non-programmer now, this scrip motivated me to start with python. I did check each & every line of the scrip by running it in the python shell interpreter. Initially the non-blocking was not there, the interpreter then used to endlessly wait after the s.recv function, so I put it to non-blocking.

Surprisingly the s.recv function did capture the below sample 4 times before stopping.

 "<electricity id="443719000861"><signal rssi="-52" lqi="13"><battery level="100%"><chan id="0"><curr units="w">209.00</curr><day units="wh">2412.21</day></chan><chan id="1"><curr units="w">32.00</curr><day units="wh">527.07</day></chan><chan id="2"><curr units="w">32.00</curr><day units="wh">1505.04</day></chan></battery></signal></electricity>"

@wallabybob:

I would also use packet capture to verify your box is actually receiving data with destination IP address of the multicast address in the script and that the destination port in the data matches the destination port specified in the script.

Thus I was suspect of the same issue, if the OWL was not multicasting, I ran pftop on the pfsense and it was there. But it is a periodic (i think every 10 secs) multicast. Even looping the above s.recv statement does not capture the packets once it stops.

EDIT: ptop capture below:

udp   I 192.168.1.14:22600    224.192.32.19:22600   0:1     12    18     3  1062
udp   O 192.168.1.14:22600    224.192.32.19:22600   1:0     12    48     1   354

vbhoj74 · Dec 15, 2012, 10:27 AM

friendly bump!

I tested the scrip removing the non-blocking on a raspberry-pi on my network, it works just fine. With the pfsense, it keeps on waiting for data but nothing comes up. Since running the scrip at the python shell captures the data randomly at times (only 4 times to be precise), I feel clueless.

vbhoj74 · Dec 16, 2012, 11:25 AM

RESOLVED

There is nothing wrong with the code. There was some routing issues due to which the ethernet ports on the pfsense does not seems to capture the multicast packets. here is what I did:

Added a Gateway:
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1

Added a Route:
224.192.32.19/32 LocalNetwork - 192.168.1.1 Lanbridge

Dont know if there is a better way of achieving it.

I'll finetune the code and post it here if anyone would like to use it.

vbhoj74 · Dec 17, 2012, 1:48 PM

I've successfully ported & modified the python code, have attached two files, owl.py and send_gmail.py.
(Original code credit goes to PiFan over http://www.raspberrypiusers.com/?p=7486 )

Here is what it does at its present form:

1. Captures all the multicast data of your OWL Intuition-LC, and saves it in a csv file. It saves phase1,2,3 & totals.
2. You can set a cost limit per day, exceeding which it sends an email. I've modified the original code to use gmail a/c for sending mails.
3. It mails a daily summary usage at EOD.

In future development I would probably want it to:

1. Maintain a weekly avg Kwh consumption.
2. email consumption details based on electricity supply billing cycles.
3. Weekly limits Vs daily limits.
4. Add a 4 line LCD display which shows live usage of all 3 phases. (Ambitious, at present this is beyond my scope of knowledge and would love some help)

Installation Steps:

1. Rename the attached files to .py extension. You may place both of them in /home
2. Edit both files, check the comment areas to modify.
3. #chmod +x /home/owl.py
4. #chmod +x /home/send_gmail.py
5. Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1
6. Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge
7. Pfsense>Diagnostic>Backup>Download Backup config.xml
find /system, and add just below:
<shellcmd>python /home/owl.py</shellcmd>
save the file structure and restore.

Notes:

Step 5 & 6 :
edit the gateway & route as per your local LAN IP and interface names.

owl.txt
send_gmail.txt

stephenw10 · Dec 17, 2012, 1:15 PM

Nice one.
So you didn't have to install any further libraries or anything python related?

I don't like the look of those network mods. It's almost always a bad idea to add a gateway to the LAN interface. It makes pfSense treat it as a WAN.

Steve

vbhoj74 · Dec 17, 2012, 1:46 PM

Steve,

Forgot to mention that you need to install the python package but no other libraries or anything else python related.

I installed python using the below commands at the shell prompt on an alix embedded system:

/etc/rc.conf_mount_rw
setenv PKG_TMPDIR /root/
pkg_add -r http://files.pfsense.org/packages/8/All/python27-2.7.2_3.tbz
/etc/rc.conf_mount_ro

I had to add the gateway to route 224.192.32.19 on the local interface, otherwise it just does not seems to capturing the packets. Maybe there is something wrong with the routing & needs further probing. Before this workaround I tried all kind of things, like opening the LAN firewall filter to all packets, disabling packet filter all together, checking up NAT if something is twisted up there, disabling VPNs, nothing seemed to work. Tried finding on this forum too if anything exists on multicast packet related on pfsense ports, did not find much except that its enabled by default on 2.0.1 version.

stephenw10 · Dec 17, 2012, 3:29 PM

Ah. Thanks for the update. Why have you set PKG_TMPDIR to root?

If it works for you then I wouldn't worry about it. You might have problems with routing if you use multiwan though.
I feel sure there must be a better way of handling the multicast traffic though I can't think of anything right now. Multicast proxy perhaps?
Looking at your additions:

Add under Pfsense>System>Routing>gateway
LocalNetwork Lanbridge 192.168.1.1 192.168.1.1

This looks like you are adding a gateway with the name 'LocalNetwork' into the interface 'Lanbridge' (I don't know what that interface consists of) where the gateway address is 192.168.1.1. That would normally be the address of the LAN interface is that now the address of Lanbridge? If so you have set the gateway as the interface address itself. :-\

Add under Pfsense>System>Routing>routing
224.192.32. 19/32 LocalNetwork - 192.168.1.1 Lanbridge

So this looks like you have added a static route to send all traffic for 224.192.32.19/32 to the Localnetwork gateway on the Lanbridge interface but that is the Lanbridge interface and that is presumably the same interface this traffic came in on anyway? :-\

Confusing! ::)

Perhaps you could simply add a virtual IP to Lanbridge with the address 224.192.32.19. I'm pretty much a total noob when it comes to multicast though I confess.

Steve

vbhoj74 · Dec 17, 2012, 4:52 PM

PKG_TMPDIR to root is probably a bad idea, it was a cut & paste from somewhere this forum, never thought what I was pasting. Changed it to /home/tmp should be good.

I got two LAN ports + wifi card all bridged together into bridge0 interface which I call LANBRIDGE. And your assumption is correct that 192.168.1.1 is the local IP of this interface.

Again correct that I'm trying to route back the traffic on the same interface it came up on. I'm a noob with multicasts too, what ever little I know, is that the ports should process the multicast packets until its filtered somewhere. Since pfsense is a router, it will not forward multicast onto another network interface which is not the case here. Ideally pfsense should receive the multicast traffic from the LAN port, which I think it is, else how will it route it back. Or maybe its not capturing the packets because i've bridged the ports ? a possible bug ?

Adding a multicast range IP to an interface does not sound a good idea, don't know if this really can be done.

stephenw10 · Dec 17, 2012, 9:56 PM

@vbhoj74:

Adding a multicast range IP to an interface does not sound a good idea, don't know if this really can be done.

Mmm, yes. Adding a virtual IP that can receive/respond to multicast traffic might do it though. Or perhaps the proxy like I suggested. This must have been solved before. :-\

Steve

vbhoj74 · Dec 18, 2012, 9:10 AM

Trying with IGMPProxy, some error it says and not adding to routing:

Dec 18 12:55:08	igmpproxy: Warn: MRT_DEL_MFC; Errno(49): Can't assign requested address
Dec 18 12:55:08	igmpproxy: Note: Removing MFC: 192.168.1.14 -> 224.192.32.19, InpVIf: 3
Dec 18 12:55:08	igmpproxy: Warn: age_table_entry: SIOCGETSGCNT failing for (192.168.1.14 224.192.32.19); Errno(49): Can't assign requested address
Dec 18 12:55:06	igmpproxy: Note: New origin for route 224.192.32.19 is 192.168.1.14, flood -1

also tried adding an Alias to the Lanbridge interface, which broke my system and I had to flash restore it. I have a modem attached on the WAN interface which I've configured using the virtual interface method and then natting it. Maybe if proxy does not work I'll try it with virtual interface method.

EDIT: Forgot that adding virtual interface works with only WLAN and with PPPoE WANs. So left with proxy solution which does not seem to work.

stephenw10 · Dec 18, 2012, 1:11 PM

Try using a virtual IP rather than a virtual interface:
http://doc.pfsense.org/index.php/What_are_Virtual_IP_Addresses%3F
I would try IP Alias.

Steve

vbhoj74 · Dec 18, 2012, 2:59 PM

That was my first try, as soon as I gave it a virtual ip alias it locked me out, saying i might me in a man in the middle attack, and then could not access the box at all. I tried it with a console cable but it looked in a crashed state, so I booted it, and it hung at every boot attempt I made. I've a vanilla pfsense flashed on another cflash, I'll try this again with this build to see how it goes.

Once I add an alias, what else do I need to do to make that subnet work. It's already passed on the firewall rules. I think i wold not need to NAT it, since this is IGMP subnet that we need to enable and not another routable subnet. Clueless what crashed my install.

stephenw10 · Dec 18, 2012, 3:21 PM

I'm pretty much guessing at this point! I've never tried adding an IP Alias to a bridge interface, could be some incompatibility you've discovered. It's an unusual config to say the least.

Steve

vbhoj74 · Dec 18, 2012, 4:34 PM

This document below does not seem to suggest usage of ip alias with ver 2 installs. This if for modem access configuration, but I guess provides a clue that ip alias may not work with ver 2?

http://doc.pfsense.org/index.php/Accessing_modem_from_inside_firewall