26 Sep 2013

Node-RED is the visual tool for wiring the Internet of Things that I have been working on through the year. What started as an educational exercise to get to grips with technologies such as Node.js and d3, soon turned into a tool that provided real value in client engagements.

One of our goals all along was to open-source the tool; to build a community of users and contributors. Following a whirlwind pass through the appropriate internal process, we got all the approvals we needed at the start of September and have released Node-RED as an Apache v2 licensed open source project on github.

You can find us here: nodered.org.

Following on from that, I have had the opportunity to speak at a couple of external events.

First up was the Wuthering Bytes technology festival in sunny/rainy Hebden Bridge. I did a completely unprepared, but well received, 5 minute lightning-talk at the end of the first day which generated a lot of interest. On the second day, there was a series of workshops, one of which was based around MQTT on Arduino – using my PubSubClient library. The workshop was organised by a contact of mine from the IoTLondon community, who was happy to see me there to help out. We ended up using Node-RED as well in the workshop as it makes it easy to visualise topics and move messages around. It was a great feeling to glance over the room and see people I’ve never met before using Node-RED.

The second event was last night, at the monthly London Node.js User Group (LNUG) Meetup. This time I had a 30 minute slot to fill, so went prepared with a presentation and demo. Again, it was well received and generated lots of good comments on twitter and afterwards in person. The talk was recorded and should be embedded below – starts at 29mins.

We’ve got a couple more talks lined up so far. Next up is the November meetup of IoTLondon, exact date tbc, and then ThingMonk in December.

22 Jan 2013

One of the parts of the Internet of Things that I feel often gets overlooked are the Things. There seems to be a new IoT gateway, hardware platform or consortium to “standardise” every week.

These are all honourable activities to move things either forward, or in the case of standardisation consortium, in circles.

But what about the Things? This is the part of IoT that appeals to me the most. Crafting a Thing that will appeal to a wide audience takes so much more that a bit of hardware and software. It needs designing to be an object that people willingly accept into their homes. This means much more than just the physical nature of the Thing, but at all levels that it is interacted with.

This was one of the inspirations behind my Orb project. But that, in its current state, has a long way to go. It remains the preserve of those who are prepared to get a soldering iron out and write some code.

I’ve known Alex for a few years now. She gave me my Arduino Ethernet shield back in the early days of Tinker and has been a good friend since. As someone with a design background, working in that space, she gets the Things idea better than most.

Likewise, Adrian proved his IoT credentials long ago with Bubblino. Soon you’ll be able to say he (co-)wrote the book on the subject.

They are working to produce a real Thing for the world to buy. The Good Night Lamp is based on an idea Alex had a few years ago and is conceptually simple; a pair of internet connected lamps, one big, one small. When the big one is turned on, then on comes the small one – where ever it is in the world.

That’s it. Nothing more, nothing less.

Imagine giving the big lamp to a loved one abroad; a wife, a boyfriend or an elderly grandparent. As they come and go, turning the big lamp on and off, so does the small lamp on the shelf in your living-room. It gives you a level of ambient intimacy; knowing the loved one is there, without being there.

In a work setting, perhaps it signals you’re available for the Skype chat, without resorting to email/IM.

That’s it. Nothing more. Nothing less.

Good Night Lamp on Kickstarter from Good Night Lamp on Vimeo.

The geek in me screams “but that’s easy to do – just plug an Arduino into a lamp and you’re done”. But that misses the point. This isn’t simply a lamp with an Arduino in it; it’s a Thing that has been designed.

Making Things and shipping them is hard and expensive. To help make it a reality they are currently running a Kickstarter campaign. It’s nearing the end and, as it stands, they’ve got quite a way to go.

If you like the idea, go and back the campaign. Even if they don’t reach their funding target, an unsuccessful Kickstarter project isn’t going to stop Alex and co making this Thing a reality.

15 Jan 2013

I made a thing. I wanted to make lots of things last year, but didn’t. That made me sad. It being a new year and all, it felt like a good time to start making.

Last summer, I was fortunate enough to be invited to BERG’s Little Printer Hack Day. I went along with not only all sorts of ideas to play with but also a migraine and ongoing waves of nausea through-out the day. Which wasn’t ideal. Despite that, it was a fun day and having managed to not pass out, I worked with Kass to make the ASCII Meterogram – a weather report from the Norwegian Meterological Institute and NRK.

A couple weeks ago, whilst contemplating doing a write-up of the year I stumbled over the code from the hack day and decided I ought to do something with it. Later the same day, for reasons I don’t recall, I found myself falling down the rabbit-hole of Maze generation algorithms on Wikipedia.

photo courtesy of Alice @ BERG

Not long later, I had a Little Printer publication that serves up a new random maze every day. There isn’t much to it really – just your standard recursive backtracker algorithm in a few lines of javascript. As you do.

The code’s up on GitHub. It’s a Django app, so if that’s your thing, feel free to have a play. Otherwise, have a look at the sample, or subscribe your own Little Printer through the BERG Cloud remote.

It feels good to make a thing. To make a thing, ship it and see, as of the moment I’m writing this, 30 people subscribed to receive a daily piece of the thing.

More making this year. More bad puns in blog post titles.

11 Nov 2012

I’ve just tagged a new release of the Arduino Client for MQTT – v1.9.1. This release includes an API change that will break existing sketches, something I’m very concious of doing – particular as the last release had such changes as well. But ultimately I decided this one was needed.

Previously, an instance of the client would be created with a call such as:

PubSubClient client(server, 1883, callback);

The library would then create an instance of EthernetClient under the covers. This was fine for hardware that used that class, but there are an increasing number of shields that use their own instance of the common Client API.

So, now the constructors for the client require an instance of Client to be passed in. For the vast majority of existing sketches, this just means changing the above line to something like:

EthernetClient ethClient;
PubSubClient client(server, 1883, callback, ethClient);

Amongst the other functional changes are the ability to publish a payload stored in PROGMEM, connect with a username/password and also the changes described in the previous post to not fragment MQTT packets over multiple TCP packets.

There is also now a regression test suite for the library. Built using python and Ino, the suite first verifies each of the examples compiles cleanly, and if an Arduino is connected, it will upload each sketch in turn and run unit tests against it. So far the focus has been on the scaffolding of the test suite, with only a couple actual tests in place for the example sketches. I’ve tried to write it in a way to be agnostic of the library it’s testing. There’s more work to do on that side of things, but ultimately it could well be reused by others. I imagine it’ll branch into its own repository when it gets to that point.

I’ve also given the project page a tidy and moved the API docs to their own page.

Finally, I’m considering closing the comments on the project page. with 75 comments already, it isn’t the best medium through which to provide support. I’m not sure pointing to the library’s issue tracker on github is necessarily the right thing either – it’s an issue tracker, not a support forum. Perhaps I should set up a dedicated Google group for the library – or is that overkill?

Feedback, as ever, is welcome.

28 Oct 2012

Cosm has had MQTT support for some time now and I’ve had my home energy usage bridged up there from my local RSMB quite happily. One thing I hadn’t played with properly was talking to Cosm directly from an Arduino – aside from a 5 minute proof-I-could-do-it.

I’m putting together a new release of the PubSubClient, and having had the occasional report of problems talking to Cosm with it, I decided to see where things stood. Sure enough, the connection was pretty unstable; often failing to connect, hanging and certainly not reliably sending or receiving messages.

Knowing the client was producing perfectly valid MQTT packets and works with other brokers, the problems had to either exist in the network connection to Cosm or with their broker implementation. The fact my RSMB bridge to Cosm work from the same network ruled out a fundamental network issue. Equally, there are plenty of MQTT brokers on-line these days to check it wasn’t a problem with the Arduino talking to a remote broker, rather than one on the same subnet.

Having debugged this sort of thing with other implementations, I had a good idea what the issue was; splitting MQTT packets over multiple TCP packets – something I knew the PubSubClient did.

When reading an MQTT packet from the network, it’s easy to assume that a read from the network will provide the next byte of the packet. When testing on local networks, that will mostly be true. But when dealing with internet-scale lag between TCP packets, the code doing the reading must handle the case when the next byte hasn’t arrived yet. This is what I think the Cosm broker is tripping over.

A quick test compared writing the packets to the network with multiple calls versus doing it with a single call and saw all of the reliability issues disappear. So I decided to update the client library to do just that.

If all you’re after is working code, then you can grab the latest library from GitHub. Note that this code includes an API change on each of the constructors – you need to provide an instance of EthernetClient rather than leave it to the library to create one. You can see what I mean in the updated examples.

In case you’re interested, here’s a run down of the changes I had to make to get this to work.

For the purposes of this exercise, an MQTT packet consists of three components; a single-byte header, between 1 and 4 bytes encoding the remaining length and finally the remaining payload bytes. The existing library provides a function that takes the header and payload, calculates the remaining length bytes and writes each of them to the network in turn – in other words, three separate network writes.

boolean PubSubClient::write(uint8_t header, uint8_t* buf, uint16_t length) {
   uint8_t lenBuf[4];
   uint8_t llen = 0;
   uint8_t digit;
   uint8_t pos = 0;
   uint8_t rc;
   uint8_t len = length;
   do {
      digit = len % 128;
      len = len / 128;
      if (len > 0) {
         digit |= 0x80;
      lenBuf[pos++] = digit;
   } while(len>0);

   rc = _client->write(header);
   rc += _client->write(lenBuf,llen);
   rc += _client->write(buf,length);
   lastOutActivity = millis();
   return (rc == 1+llen+length);

The crude way to reduce the network writes would be for the function to copy all of those bytes to a single buffer and then pass that single buffer to the network. But that would involve a lot of unnecessary copying of bytes around as well as the increased memory usage for the buffer.

   // Allocate yet more memory
   static uint8_t packet[MQTT_MAX_PACKET_SIZE];
   pos = 0;
   packet[pos++] = header;
   for (int i=0;i<llen;i++) {
      packet[pos++] = lenBuf[i];
   // Copy lots of bytes around   
   for (int i=0;i<length;i++) {
      packet[pos++] = buf[i];
   rc += _client->write(packet,pos);

There is already a buffer used for incoming packets which can be reused for the outgoing packets. The difficulty is the remaining length bytes; until the payload is constructed the code doesn’t know how long it will be, which means it does not know how many bytes are needed to encode the length. Without knowing how many bytes are used to encode the length, the client doesn’t know the offset into the buffer to start writing the payload.

The trick is knowing that there will be at most 4 bytes of length and one of header – so the payload can be written starting at the 6th byte of the buffer. Then, once the length bytes are calculated, they can be inserted into the buffer working back from the 5th byte and the header byte can be inserted ahead of the length bytes. This gets the MQTT packet into a single contiguous buffer, with minimal copying of bytes, which can be passed to the network – albeit starting at a known offset from the start of the buffer.

   // llen is the number of length bytes
   buf[4-llen] = header;
   for (int i=0;i<llen;i++) {
      buf[5-llen+i] = lenBuf[i];
   rc = _client->write(buf+(4-llen),length+1+llen);

Simple as that.

There is one place in the library that can’t be reduced to a single network write; the recently added publish_P function allows the message payload to be stored in PROGMEN. The benefits that brings would be entirely lost if they were then copied into regular memory before writing to the network. Not much to be done about that.