currentcost « knolleary

25 May 2011

I’ve been running my CurrentCost MQTT Bridge for a couple of months now and it has been working well, but for one thing; on occasion it would lose its connection and not reconnect.

It has taken some time to iron out the various issues that were causing this, but I think I’ve finally got there so I’ve updated the gist and herewith a run down of what it took to get things working properly.

Whenever the code realised it had lost its connection, it would simply try to re-establish the connection. Now, as soon as the connection is lost, it performs a reset of the ethernet hardware, re-obtains a DHCP address and runs the DNS query. If either of these latter stages fails to complete in a reasonable time, it loops back to the hardware reset.

In an earlier version of the reset code I wasn’t timing out the DHCP look-up. On more than one occasion I spotted, thanks to Wireshark, that the DHCP request being sent by the Bridge had a source MAC value of 00:00:00:00:00 – which was correctly being ignored. Without a time-out, the bridge was stuck sending out invalid requests and waiting for a reply that wasn’t coming. I haven’t been able to identify exactly why the bridge appeared to have forgotten its MAC address, but to rule out the obvious, I added code to restore the MAC from EEPROM as a part of the reset code.

This combination of changes made things much more reliable, but there remained one final issue – it relied on the code noticing it had lost its connection. In most cases, this worked fine, but there was still a case where the bridge would be sat publishing its data with no-one listening.

MQTT has a facility that is supposed to deal with this; the keep-alive timer. But my MQTT library only implemented half of this feature – it ensured a ping was sent within the keep-alive time if nothing else was sent, but it never checked that it received the expected response. As the bridge code publishes data regularly but doesn’t subscribe to any topics, then proper handling of the keep-alive pings is all the more vital to detecting loss of connection. I’ve made the necessary updates to the MQTT library on github, but I haven’t made a formal release of it yet.

The lesson to learn from this is that it is easy to write code that works most of the time, but when you’re writing code to run on an unattended, embedded, device you need as many belts and braces as possible to ensure it keeps going.

a comment

CurrentCost MQTT Bridge

4 Apr 2011

tech

6 comments

Getting the data from a CurrentCost meter on-line has required either an always-on PC or something like the Arduino/Ethernet Shield set-up I’ve been using. That was the case until CurrentCost released their Bridge device.

CurrentCost Bridge

This little box plugs into the data port of the meter on one side and into your network on the other. It then sends the energy readings over HTTP, so you can then view it on your own dashboard.

Under the covers, the Bridge is essentially an Arduino and Ethernet shield relaid onto a single PCB by the good folk at the sadly-no-more TinkerLondon. This makes it relatively easy to reprogram for your own nefarious needs – or in my case, getting it to publish the data over MQTT.

John Crouchley has written about some of the specifics of the board and how it can be reprogrammed with the standard Arduino development environment using a custom USB cable. Not wanting to bother with making a custom cable, I stuck to using my USBtinyISP to flash the board directly. Herewith an outline for how to do that (on Linux at least):

Plug the ISP into the 6-pin header on the bridge board. If you have the ISP set to power the board make sure you haven’t got the bridge’s own power supply plugged in.
Make sure you’ve followed the instructions at the bottom of the page here about udev to ensure you have the appropriate permissions to access the programmer.
From the Arduino IDE, burn the bootloader. To do this, first ensure Tools->Board->Arduino Pro or Pro Mini (3.3V, 8 MHz) w/ ATmega328 is selected. Then do Tools->Burn Bootloader->w/ USBtinyISP. Of course, if you’re using a different ISP, select the one for you. This takes a little while to complete, but it is worth the wait.
Once you have a sketch you want to upload, select Sketch->Verify/Compile. This generates the hex file we need in a temporary directory with a name like /tmp/build3473620529065332403.tmp. The hex file itself will have the same name as your sketch, but with a .cpp.hex file extension.

Use avrdude to upload the file and you are done.

$ cd /tmp/build3473620529065332403.tmp/
$ ls *.cpp.hex
MyBridgeCode.cpp.hex
$ avrdude -pm328p -cusbtiny -D \
      -Uflash:w:MyBridgeCode.cpp.hex
...

As for a sketch to use, the basic one I wrote over two years ago needs some considerable updating. It uses absolute positioning in the XML feed to find the data it wants. This makes it fragile to any timing issues in the serial feed; something the CurrentCost can occasionally suffer from. Nor does the sketch handle the multiple sensors and channels that the newer hardware supports.

On top of that, there are some differences specific to the Bridge that need to be accounted for. The serial connection from the CurrentCost is connected to digital pins 0 and 1 which means we can use the built-in Serial object to access it rather than use the SoftwareSerial library.

As John mentions in his post, the Ethernet chip’s reset pin is connected to a digital pin on the Arduino – so something like the following can be used to to cleanly enable it:

pinMode(7,OUTPUT);
digitalWrite(7,LOW);
delay(250);
digitalWrite(7,HIGH);
pinMode(7,INPUT);
delay(500);

With all that in mind, I’ve put a new sketch on Github that does all of this and more. It uses the UUID of the bridge (stored in EEPROM) to generate both the client ID to connect with and the topic to publish to. This lets the same sketch run on multiple bridges without having to recompile with instance-specific settings. Some of the other features of the sketch are:

generates a new 16-byte UUID if it doesn’t find one in EEPROM
uses the last 6-bytes of the UUID as the MAC address
uses DHCP to obtain a local IP address
uses DNS to look-up the server address if SERVER_HOST_NAME is defined – otherwise expects SERVER_IP_ADDRESS to define the address to use.
reconnects to the MQTT server if the connection is lost
publishes readings to the topic structure “cc/[uuid]/[sensor]/[channel]“
if PUBLISH_CONNECTION_STATE is defined, connection status is maintained on the topic “cc/[uuid]/s“. A retained message is published with the value ’1′ when the bridge connects. A Will message is set so that if the bridge loses its connection, a retained message will be published to this topic with the value ’0′.
if PUBLISH_TEMPERATURE is defined, temperature is published to the topic “cc/[uuid]/t“. It is published as a retained message and will only republish a temperature when it is 0.5 different to the last one sent

The sketch was developed using arduino-0022, the latest version of my PubSubClient library and the DHCP/DNS libraries from Adrian – links to these are in the sketch. Once the next version of Arduino is released, I will refresh the sketch to use that – as the DHCP/DNS libraries ought to be part of the standard distribution by then.

Power Graph

6 comments

Monitoring energy use with an Orb

5 Jan 2009

tech

12 comments

I spoke at Homecamp recently about how an ambient orb could be used to monitor home energy usage. I’ve finally gotten around to putting some of it into practice so thought I would share some of the details of the setup as well as some more of my thoughts on the subject.

There are three key pieces of hardware in use. The Viglen MPC-L is the heart of the system. As I’ve mentioned previously, this is a low powered linux box running Ubuntu. The CurrentCost meter is connected to the MPC over USB-serial and my trusty arduino acts as an integration point for homebrew toys – including my ambient orb.

The MPC is running a Really Small Message Broker (RSMB). This is a small-footprint pub-sub message broker that talks MQTT. Each time the CurrentCost sends out a update, a piece of perl (“cc_pub.pl“) parses the data, sticks it into an RRDTool database for graphing and also publishes it to the home/cc/power and home/cc/temp topics.

Another piece of perl (“orbcontrol.pl“) is subscribed to the house/orb topic. When messages arrive on that topic, they are passed over serial to the arduino.

The sketch on the arduino currently listens on its serial port for commands that are then passed to the BlinkM in the orb. The format of the commands is identical to those in the BlinkMTester sketch that comes with the BlinkM. In the future this will do more as more things are attached to the arduino.

So far I have described how the orb is controlled and how the power data gets into the system. The next piece is how the two are plumbed together. Unsurprisingly, yet another perl script is running the on the MPC that provides the glue for this mixed metaphor.

orblogic.pl is subscribed to the home/cc/power topic so it receives all of the updates from the CurrentCost. It then makes a decision as to what colour the orb should be and then publishes the appropriate command to the house/orb topic.

Orb setup sketch

That is pretty much it – simple eh? Well, I did skip over the most interesting part – how to decide what colour the orb should be.

There are two key philosophies that come into play here:

Alert the unusual – have the ‘default’ state be the least obtrusive, such as ‘off’
Band the data – don’t react to single-point changes in the value. For example, when monitoring temperature, nothing is worse than constantly saying “its 19°C, its 18°C, its 19°C…”

These points should be applied whenever thinking about how to turn a raw stream of data into a useful stream of information.

I currently have a very simple piece of logic controlling the orb:

If power < 400, turn the orb off
If power between 400 and 1000, turn the orb orange
If power > 1000, turn the orb red

When we’re sat in the living room watching TV, we typically use under 400 watts, so the orb will only show anything if we’re over this value. It is very unusual to go over 1 kw unless we’re cooking, which is why that gets a stronger alert.

powergraph sketch

Despite its simplicity, this has already had the effect of making us more aware of when lights have been left on or when the kettle has finished boiling.

But as with all things, it could get much smarter. Having said we typically use under 400 watts watching TV, I also made the caveat ‘unless we’re cooking’. Given the system has a history of our power usage, it could feasibly determine suitable bands dynamically – so going over 1 kw between 6pm and 7pm isn’t necessarily a bad thing. Also, it chould only go red if the value goes over 1 kw for more than the time it takes to boil the kettle.

It all comes back to determining what is ‘unusual’.

12 comments

CurrentCostuino

19 Nov 2008

code

3 comments

With homecamp fast approaching I spent some time last weekend reviving some CurrentCost hacking I started in the summer.

One of the downsides of the data connection on the CurrentCost is the need for an always-on computer in order to capture the data. In practice, the existence of low powered machines like the MPC-L, or slug, make this a pretty inconsequential downside. That said, it was enough motivation to see what I could do with an Arduino and Ethernet Shield.

I have previously mentioned that part of the setup used to graph the power data online was to publish the data to a broker-in-the-sky using MQTT. To achieve this with an arduino it was clear that I needed one thing in particular – an arduino that is capable of connecting and publishing to an MQTT server.

With my ethernet shield in hand, I set about implementing enough of the MQTT spec to support basic publishing and subscribing. With that working, I looked into getting the arduino and CurrentCost talking. Thanks to Alexis, this was very straightforward, although I did rework the parsing of the data to take advantage of its somewhat-fixed length format.

Here’s the sketch I ended up with:

#include <Ethernet.h>
#include <PubSubClient.h>
#include <SoftwareSerial.h>

byte mac[] = {  0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
byte ip[] = { 172, 16, 0, 50 };
byte server[] = { 172, 16, 0, 103 };

PubSubClient client(server, 1883,0);

#define rx 6
#define tx 7

SoftwareSerial softSerial =  SoftwareSerial(rx, tx);

void setup()  {
  pinMode(rx, INPUT);
  pinMode(tx, OUTPUT);
  softSerial.begin(9650);
  Ethernet.begin(mac, ip);
  delay(500);
}

void connect() {
  while(!client.connect("nolcc")) {
    delay(10000);
  }
  delay(200);
}

int offset = -1;
void loop() {
  if (!client.connected()) {
    connect();
    offset = -1;
  }
  client.loop();

  // scan the data for <msg>
  while(offset == -1) {
    char c = ' ';
    while (c!='<') {
      c = softSerial.read();
    }
    if (softSerial.read()=='m' &&
        softSerial.read()=='s' &&
        softSerial.read()=='g' &&
        softSerial.read()=='>') {
      offset = 5;
    }
  }

  // skip to the current value
  while(offset < 156) {
    offset++;
    softSerial.read();
  }
  char current[5];
  int value = 0;
  for (int i=0;i<5;i++) {
    current[i] = softSerial.read();
  }
  offset += 5;

  // skip to the temperature value
  while(offset < 243) {
    offset++;
    softSerial.read();
  }
  char temp[5];
  for (int i=0;i<4;i++) {
    temp[i] = softSerial.read();
  }
  offset += 4;

  client.publish("current",(uint8_t*)current,5);
  client.publish("temp",(uint8_t*)temp,4);

  // scan for </msg>
  while(offset != -1) {
    char c = ' ';
    while (c!='<') {
      c = softSerial.read();
    }
    if (softSerial.read()=='/' &&
        softSerial.read()=='m' &&
        softSerial.read()=='s' &&
        softSerial.read()=='g' &&
        softSerial.read()=='>') {
      offset = -1;
    }
  }
}

It publishes each power reading to a topic called ‘current’ and each temperature reading to a topic called ‘temp’. This isn’t quite the right configuration for the existing broker setup, but it wouldn’t take much to modify it appropriately.

You’ll note this sketch isn’t much use without the library that does the MQTT bit. Luckily, I have been given permission to release the library to the world. I should stress at this point, in case it wasn’t obvious from the fact I am releasing this code on my personal site, that my employer has nothing to do with it.

You’ll find the Client for MQTT library here.

If you want an MQTT server to play with, go buy Lotus Expeditor to get your hands on a microbroker <gratuitous-plug />, or check out the links on mqtt.org for the Really Small Message Broker.

7 Jul 2008

Roo and I presented at Open Tech 2008 this weekend on the Current Cost hacking that has been going on in these parts. It was my first time presenting to quite such a large audience and all things considered, it was a lot of fun and went very well.

| View | Upload your own

Although I took some notes through the day of the other talks I went to, Roo has done a much better write-up than I would. So I’ll point you his way.

4 comments

older things »

knolleary

knolleary

stuff by nick o'leary