7 Apr 2011

I have been thinking about how MQTT could be integrated into Pachube, as a service that utilises their public API. With their Hackathon happening tomorrow, which I’m unable to attend, it felt like a good time to write up what I’ve done.

Basic Pachube concepts

The way Pachube structures data lends itself well to a topic hierarchy:

  • An Environment is a logical collection of datastreams. It can be used to represent a physical location, such as my house.
  • A Datastream represents a single sensor/device within a particular environment, such as living room temperature.
  • A Datapoint is a point-in-time value on a datastream

Data is sent to Pachube using HTTP POST requests. It can retrieved by either polling a datastream’s value, or by configuring a trigger. A trigger is a push notification that happens when a defined condition occurs. This notification exists as an HTTP POST from Pachube to the URL configured as a part of the trigger. The POST contains a json payload that contains the details of the notification. There are some limitations on data-rates around triggers; the minimum interval between the same trigger firing is 5 seconds.

All access to the API is controlled using API Keys. Each request to the API must be accompanied with either the X-PachubeApiKey HTTP header, or as a request parameter.
Each user has a master API key which gives full access to their account. Additional API keys can be created with different levels of access for a particular user.

The following scheme uses the API key as a part of the topic strings used. This relies on the broker having been configured to disallow any client to subscribe to the wild-carded topic “pb/#”. Beyond that, no more security is assumed (ie username/password, SSL etc), but of course could be added.

There are two components to this; an MQTT subscribing application that can bridge into Pachube and an HTTP listening application that can bridge from Pachube triggers into MQTT.

Publishing to Pachube

A datapoint value is published to the topic:

pb/<api key>/<environment id>/<datastream id>

which is received by the subscribing application, leading to it sending an HTTP PUT of the datapoint to the Pachube API:

http://api.pachube.com/v2/feeds/<environment id>/datastreams/<datastream id>.csv

with header property:

X-PachubeApiKey: <api key>

No pre-configuration is needed here – it relies on the provided API key having the correct permission to update the datastream. There are some questions here on how to handle failures to post (permission denied, service unavailable etc), as there is no mechanism to report failures back to the originating publisher.

Subscribing to Pachube

Subscribing to a datastream is slightly more involved as it requires an HTTP intermediary for Pachube to push its notifications to which can then be forwarded to the broker. There is also no way to automatically create the trigger when a client subscribes to a topic on the broker, so an additional configuration step is required of the client. It is also possible to configure multiple triggers for a single datastream, so we cannot use a simple one-to-one mapping of datastream to topic for the outbound flow. This causes an asymmetry between publishing and subscribing, but I think it is necessary.

A trigger is configured by publishing to the topic:

pb/<api key>/<environment id>/<datastream id>/trigger

with a payload that defines the trigger in json, for example:

   // As defined in the pachube API: 

This causes the appropriate HTTP PUT to the Pachube API to create the trigger.

The trigger is configured with the url of the HTTP intermediary that is able to react to the HTTP POSTs sent by Pachube by publishing the value to the corresponding MQTT topic. This means clients can subscribe to:

pb/<api key>/<topic>

to receive the updates as they are received.

As an aside, there could be an option to make the trigger feed ‘public’ on the broker, so a user’s API key doesn’t need to be shared. This would mean the trigger would get published to something like:


To unsubscribe, the trigger needs to be deleted on Pachube. This is done by publishing to the same topic as was used to create the trigger, but with a payload that includes a ‘delete’ command.

The bit at the end

I have implemented pretty much all of what I describe above and it works. Although there are some drawbacks with the approach:

  • The rate-limiting that Pachube applies to triggers (at least 5 seconds between firings) defines the minimum granularity for getting data out to MQTT – so potentially values could be missed.
  • Subscribing to a datastream is not as simple as subscribing to a topic.
  • No consideration is made of the additional meta-data that is provided with the HTTP requests – this is very much just about the raw data points

In an ideal world, the MQTT interface would sit much closer to the centre of Pachube and be more of a 1st class API citizen; but then I want to MQTT enable everything.

Unfortunately I don’t have the wherewithal to make it public facing for other’s to play with at the moment, which is a shame. If you’re really interested in seeing it in action, get in touch and I’ll see what can be done.

6 Apr 2011

I often spend time tweaking the theme of this site rather than write content for it, but I thought the latest tweak was worth a few words of mention.

I have tried to keep the theme quite clean – with all of the traditional sidebar content pushed right down at the bottom of the page. Previously, there was a simple division between the bottom content and the rest of the page; nothing more than a border-top CSS declaration:

Inspired by the new look Planet GNOME, I wanted to do something a bit more interesting down there – some sort of ragged edge, but I didn’t want to spend an age trying to draw it by hand.

Coincidentally, earlier today I was looking for an excuse to play with Processing and this seemed like just the thing. I very quickly got something producing a very regular pattern:

But that felt too boring – it needed some randomness.

These were way too severe. To emulate the grass-like effect, I moved to using curves, rather than straight lines.

After a couple attemps where I didn’t have my Bezier control points under control, I finally hit upon something I liked:

And that’s what you’ll find at the bottom of page. The nice thing with this approach is that I can easily tweak the script to produce variations and I don’t have to start from scratch each time.

For the record – and so I can find it when I want to tweak it – here’s the processing sketch that produced the final result:

update: stuck the code in a gist on github

void setup() {
  int x = 0;
  for (int i=0;i<200;i++) {
    int y = (int)random(15,30);

    int w = (int)random(5,15);
    x += w;
24 Jul 2009

Just a quick one to say I’ve made an update to the Arduino MQTT PubSubClient to fix a compile error with the latest version of the arduino software (0016).

As ever, get the updated MQTT Arduino client from here.

24 Mar 2009

Whilst playing around with some ideas for a project I have in mind, I have created something else along the way.

I needed the ability to draw lines between elements on a webpage, with some built-in intelligence on how to route the lines so they look sensible. I couldn’t find anything in jQuery, Dojo or elsewhere that did this type of thing out of the box, although I’ll admit I didn’t look too hard.

Having got something basically working, I decided to throw together a demo to help see if they did what I wanted. As is often the case, my little demo has taken a life of its own and has distracted me from the original project I was doing this for.

And so came to life the Logic Gate Thing.

Logic Gate Thing

A couple disclaimers to make: it has only been tested with Firefox on Linux and is reliant on Javascript. That said, I’d be happy to hear how other browsers get on with it.

Update: It should now work on WebKit based browsers; Safari, Epiphany, etc and Chrome. As Jamie points out in the comments, it does work in IE8 albeit with some slightly weird lines.

29 Jan 2009

I’ve made some updates to the MQTT library for the arduino ethernet shield and released version 1.1. You can find the library here. I’m thinking of moving it to github or google code once I can decide which to use – any suggestions? The code is now hosted on github.

This update includes:

  • smaller code footprint,
  • support for Will messages – particularly useful for embedded devices,
  • clarified licensing – this is released under the MIT License.