knolleary 2020-07-11T23:10:32+00:00 Nick O'Leary Automating Node-RED releases with GitHub Actions 2020-07-11T00:00:00+00:00 <p>When we do new releases of Node-RED there’s a mental checklist of tasks I run through.</p> <ul> <li>run the release build</li> <li>create the draft GitHub release</li> <li>attach the built zip to the release</li> <li>publish the modules to npm</li> <li>publish the GitHub release</li> <li>publish the release blog post</li> <li>post to the forum</li> <li>tweet about it</li> </ul> <p>Then about an hour later I remember to kick-off the Docker build and update the Node-RED webpage to reflect the new version number.</p> <p>For some reason, these last two steps often get overlooked - they just don’t form part of the critical path in my mind for getting the release published.</p> <p>They have always been ripe for automation as they both just involve updating the version number in some files. The problem was the effort to set that up has never really been worth the benefit.</p> <p>But then along came <a href="">GitHub Actions</a> and it all got much easier. This blog post details how I automated away those steps of the release process.</p> <h3 id="workflow-steps">Workflow steps</h3> <p>The full workflow is <a href="">defined here</a>. I’ve linked to the version of the workflow as of the 1.1.2 release just in case the latest version has completely changed by the time you come to read this.</p> <p>To begin with, we tell GitHub to trigger the workflow whenever we tag a new release.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="na">on</span><span class="pi">:</span> <span class="na">release</span><span class="pi">:</span> <span class="na">types</span><span class="pi">:</span> <span class="pi">[</span><span class="nv">published</span><span class="pi">]</span> </code></pre></div></div> <p>We then define the environment to run the workflow on and the steps it runs.</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">jobs</span><span class="pi">:</span> <span class="na">generate</span><span class="pi">:</span> <span class="na">name</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Update</span><span class="nv"> </span><span class="s">node-red-docker</span><span class="nv"> </span><span class="s">image'</span> <span class="na">runs-on</span><span class="pi">:</span> <span class="s">ubuntu-latest</span> <span class="na">steps</span><span class="pi">:</span> <span class="s">...</span> </code></pre></div></div> <p>The actual work to be done involves:</p> <ul> <li>checkout the <code class="language-plaintext highlighter-rouge">node-red</code> repository as it contains some scripts we need later</li> <li>checkout the <code class="language-plaintext highlighter-rouge">node-red-docker</code> repository, update some of its files and raise a pull-request on the repository with the changes</li> <li>checkout the <code class="language-plaintext highlighter-rouge"></code> repository, update its <code class="language-plaintext highlighter-rouge">index.html</code> file and raise a separate pull-request on the repository with that change</li> </ul> <h4 id="checking-out-repositories">Checking out repositories</h4> <p>The checkouts are easy to do with the <a href="">actions/checkout@v2</a> action. As we need to checkout multiple repositories, each is put into its own directory:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check out node-red repository</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span> <span class="na">with</span><span class="pi">:</span> <span class="na">path</span><span class="pi">:</span> <span class="s1">'</span><span class="s">node-red'</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check out node-red-docker repository</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span> <span class="na">with</span><span class="pi">:</span> <span class="na">repository</span><span class="pi">:</span> <span class="s1">'</span><span class="s">node-red/node-red-docker'</span> <span class="na">path</span><span class="pi">:</span> <span class="s1">'</span><span class="s">node-red-docker'</span> <span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Check out repository</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/checkout@v2</span> <span class="na">with</span><span class="pi">:</span> <span class="na">repository</span><span class="pi">:</span> <span class="s1">'</span><span class="s">node-red/'</span> <span class="na">path</span><span class="pi">:</span> <span class="s1">'</span><span class="s">'</span> </code></pre></div></div> <h4 id="updating-files">Updating files</h4> <p>With all of the repositories checked out, the next task is to update the necessary files.</p> <p>This needs some custom code and having looked at the various options such as creating a custom docker file, I went with the simple option of a pair of node.js scripts.</p> <p>That first needs the <a href="">actions/setup-node@v1</a> action to get Node 12 installed and ready to run:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">actions/setup-node@v1</span> <span class="na">with</span><span class="pi">:</span> <span class="na">node-version</span><span class="pi">:</span> <span class="s1">'</span><span class="s">12'</span> </code></pre></div></div> <p>The scripts themselves are:</p> <ul> <li><a href="">update-node-red-docker.js</a></li> <li><a href="">update-node-red-website.js</a></li> </ul> <p>I won’t bother with a line by line commentary, but they both follow the same structure.</p> <p>They begin with a bunch of common code that:</p> <ul> <li>grabs the new version number from the <code class="language-plaintext highlighter-rouge">node-red</code> repository’s <code class="language-plaintext highlighter-rouge">package.json</code> file.</li> <li>verifies the tag that triggered the action matches that version - a safety net to ensure we only do this on proper releases.</li> <li>verifies the version is for a stable release - we don’t want to update things for <code class="language-plaintext highlighter-rouge">x.y.z-beta</code> releases</li> </ul> <p>They then do their work on the files that need updating - mostly just search and replace of the version numbers - before saving the changes back.</p> <p>One thing to highlight from the docker update script is the line:</p> <div class="language-javascript highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nx">console</span><span class="p">.</span><span class="nx">log</span><span class="p">(</span><span class="s2">`::set-env name=newVersion::</span><span class="p">${</span><span class="nx">newVersion</span><span class="p">}</span><span class="s2">`</span><span class="p">);</span> </code></pre></div></div> <p>That syntax causes the <code class="language-plaintext highlighter-rouge">newVersion</code> environment variable to be set to the value of the <code class="language-plaintext highlighter-rouge">newVersion</code> variable in the script. This comes in handy in a later step.</p> <p>The scripts are run with:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">run</span><span class="pi">:</span> <span class="s">node ./node-red/.github/scripts/update-node-red-docker.js</span> <span class="pi">-</span> <span class="na">run</span><span class="pi">:</span> <span class="s">node ./node-red/.github/scripts/update-node-red-website.js</span> </code></pre></div></div> <h4 id="raising-pull-requests">Raising pull-requests</h4> <p>Finally, it’s time to raise pull requests on the corresponding repositories using the <a href="">peter-evans/create-pull-request@v2</a> action.</p> <p>Here’s the configuration of the step for the docker repo PR:</p> <div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="pi">-</span> <span class="na">name</span><span class="pi">:</span> <span class="s">Create Docker Pull Request</span> <span class="na">uses</span><span class="pi">:</span> <span class="s">peter-evans/create-pull-request@v2</span> <span class="na">with</span><span class="pi">:</span> <span class="na">token</span><span class="pi">:</span> <span class="s">${{ secrets.NR_REPO_TOKEN }}</span> <span class="na">committer</span><span class="pi">:</span> <span class="s">GitHub &lt;;</span> <span class="na">author</span><span class="pi">:</span> <span class="s">${{}} &lt;$;</span> <span class="na">path</span><span class="pi">:</span> <span class="s1">'</span><span class="s">node-red-docker'</span> <span class="na">commit-message</span><span class="pi">:</span> <span class="s1">'</span><span class="s">Bump</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">${{</span><span class="nv"> </span><span class="s">env.newVersion</span><span class="nv"> </span><span class="s">}}'</span> <span class="na">title</span><span class="pi">:</span> <span class="s1">'</span><span class="s">🚀</span><span class="nv"> </span><span class="s">Update</span><span class="nv"> </span><span class="s">to</span><span class="nv"> </span><span class="s">Node-RED</span><span class="nv"> </span><span class="s">$</span><span class="nv"> </span><span class="s">release'</span> <span class="na">body</span><span class="pi">:</span> <span class="pi">|</span> <span class="s">Updates the Node-RED Docker repo for the $ release.</span> <span class="s">Once this is merged, you will need to create a new release with the tag `v$`.</span> <span class="s">This PR was auto-generated by a GitHub Action. Any questions, speak to @knolleary</span> </code></pre></div></div> <p>Most of that should be fairly self explanatory, although there are a couple bits to highlight.</p> <p>First, note the <code class="language-plaintext highlighter-rouge">${{ env.newVersion }}</code> bit that picks up the environment variable set by the docker script so that the version can be included in the title and description of the pull-request.</p> <p>The second bit that needs highlighting is the <code class="language-plaintext highlighter-rouge">token</code> value. This uses a personal access token of a GitHub user that has write permission on the repository, which has been added as a secret to the node-red repository settings.</p> <p>I original set this up using a token for my account - but that would mean any action in the Node-RED repo would be able to do <em>anything</em> as me on <em>any</em> of the repositories I have write access to. So I created a NodeREDBot user, gave them write access to just these two repositories and generated an access token with the smallest set of permissions possible.</p> <p>I’ve seen GitHub acknowledge their personal access tokens need finer grained scopes down to individual repositories, but until that happens, we’ll stick with the bot user.</p> <h3 id="done-for-now">Done, for now</h3> <p>Now, whenever I tag a new release, within a couple minutes pull requests arrive on the docker and website repositories with all the updates that are needed.</p> <p>As my first foray into GitHub actions, it didn’t take long to get this all strung together. It was very satisfying to see the PRs arrive when I did the 1.1.1 and <a href="">1.1.2</a> releases this week.</p> <p>Automating these little tasks makes such a difference. Having seen how easy it is to get started, I’m now looking for what else we could use them for.</p> <p>One idea is to automatically raise an issue on the website repository if any core issue or pull-request has a <code class="language-plaintext highlighter-rouge">needs-docs</code> label adding. Updating the docs is often overlooked by contributors and I often spend a week before a release working through the backlog of doc updates needed. Having a tracking issues raised through-out the development cycle should help us keep on top of it.</p> <p>We also have some <a href="">API docs on the website</a> that are generated from the source code, but a pain to update on the site. It’s probably less than an hours work to automate that now I have all these pieces in place.</p> Twenty Nineteen 2020-01-02T00:00:00+00:00 <p>So, 2019 happened. Herewith some thoughts about the last year and reflecting on what’s to come.</p> <p>tl;dr: The world is literally on fire, Brexit is happening, we’re led by liars, cheats and idiots, and I’m no nearer to 13st then I was a year ago. But despite it all being a mess, there’s still plenty to do every day and with some focussed effort, it will all be okay in the end. Except maybe Brexit - nothing good can come from that.</p> <hr /> <p>My biggest achievement this year was getting Node-RED 1.0 released. I’ve tried to write up my thoughts about that a few times and never quite managed to find the right words.</p> <p>It was certainly exciting to get to that point. It was very satisfying to hit publish on it. And I certainly felt pride in having done so.</p> <p>But the last three months since the release have been a bit more of a mixed bag.</p> <p>Once it had shipped, I found myself without a plan. It wasn’t for a lack of things to do - the backlog is large and healthy. But I had lost the focus the 1.0 roadmap provided.</p> <p>Aside from a few fixes, I’ve not really done anything significant in the code. That isn’t to say I haven’t been busy. I’ve been working on documentation, creating some more video tutorials as well as putting together the workshop I ran at NodeConfEU.</p> <p>It’s all useful stuff, but it has all felt like catching up, rather than moving things forward. Each time I’ve sat in front of my laptop to write code or design some new feature, I’ve struggled to make any meaningful progress. And that hasn’t been fun.</p> <p>The frustration only increases when I reflect on the fact we still haven’t made Node-RED a sustainable project. If I stop working on the backlog then it grinds to a halt. We do have some valued contributors who definitely help to make a difference, but they are typically focused on their own priorities, rather than those of the wider project.</p> <p>It’s the usual Open Source story - there are plenty of companies benefiting from our work, but very few willing or able to pay that back. It’s a hard problem and one that doesn’t have an easy answer. It just so happens I’m at the sharp end of that deal. In general, I’m okay with that, but there are certainly times when I wish we could do more.</p> <p>Ever since Dave and I created Node-RED, I’ve been in roles that have enabled me to justify spending time on it. For example, when I was in Emerging Technologies, it was a tool we continued to develop to use on client projects. Then, I was a Developer Advocate focussed on IoT - using Node-RED in most of our activities to help engage with developers. But in 2019, I moved to the Open Source Development organisation, where my job <em>became</em> Node-RED.</p> <p>It’s a great place to be. Not many people can claim to have been able to turn a side-project into a job within a large organisation like IBM.</p> <p>I did a talk at <a href="">Monki Gras in 2016</a> about how Node-RED came about. Towards the end I shared some of the lessons of getting things done. One of them was:</p> <blockquote> <p>When a side-project becomes a job, it risks becoming a chore and that then risks leading to burn out.</p> </blockquote> <p>I don’t <em>think</em> I’m burnt out, but I’ve certainly been close to the edge and at times, stared over it. It’s clear I need to find a way to change things up - and writing this post is part of that.</p> <p>So with all of that said, here’s what I think I want to do in 2020.</p> <h2 id="get-a-plan">Get a plan</h2> <p>Whilst I’ve clearly needed a break since shipping 1.0, it’s time to get things moving again. The first step is to come up with a plan - a new roadmap to help prioritise what we’ve got to do.</p> <h2 id="get-more-organised-with-my-time">Get more organised with my time</h2> <p>With so many plates to keep spinning around the Node-RED project, I need to better structure my time to ensure things keep moving forward proactively rather than constantly reacting to the next thing that crops up.</p> <p>When I’m back in the office next week, I’m going to block out time in my diary each week for some of the regular tasks I need to do. I’m going to keep TweetDeck closed so I don’t have a drip feed of the latest horrors of the world distracting me.</p> <h2 id="keep-a-record">Keep a record</h2> <p>I’m going to keep a list of what I get done each week so I can look back on my successes, rather than just remember the failures.</p> <h2 id="side-projects">Side projects</h2> <p>I need something that isn’t Node-RED. A hobby, a side-project, a thing I can distract my brain with.</p> <p>I’ve started a couple things recently that I’m enjoying playing with and want to get to the point where I write about them here as well.</p> <blockquote class="instagram-media" data-instgrm-captioned="" data-instgrm-permalink=";utm_campaign=loading" data-instgrm-version="12" style=" background:#FFF; border:0; border-radius:3px; box-shadow:0 0 1px 0 rgba(0,0,0,0.5),0 1px 10px 0 rgba(0,0,0,0.15); margin: 1px; max-width:540px; min-width:326px; padding:0; width:99.375%; width:-webkit-calc(100% - 2px); width:calc(100% - 2px);"><div style="padding:16px;"> <a href=";utm_campaign=loading" style=" background:#FFFFFF; line-height:0; padding:0 0; text-align:center; text-decoration:none; width:100%;"> <div style=" display: flex; flex-direction: row; align-items: center;"> <div style="background-color: #F4F4F4; border-radius: 50%; flex-grow: 0; height: 40px; margin-right: 14px; width: 40px;"></div> <div style="display: flex; flex-direction: column; flex-grow: 1; justify-content: center;"> <div style=" background-color: #F4F4F4; border-radius: 4px; flex-grow: 0; height: 14px; margin-bottom: 6px; width: 100px;"></div> <div style=" background-color: #F4F4F4; border-radius: 4px; flex-grow: 0; height: 14px; width: 60px;"></div></div></div><div style="padding: 19% 0;"></div> <div style="display:block; height:50px; margin:0 auto 12px; width:50px;"><svg width="50px" height="50px" viewBox="0 0 60 60" version="1.1" xmlns="" xmlns:xlink=""><g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"><g transform="translate(-511.000000, -20.000000)" fill="#000000"><g><path d="M556.869,30.41 C554.814,30.41 553.148,32.076 553.148,34.131 C553.148,36.186 554.814,37.852 556.869,37.852 C558.924,37.852 560.59,36.186 560.59,34.131 C560.59,32.076 558.924,30.41 556.869,30.41 M541,60.657 C535.114,60.657 530.342,55.887 530.342,50 C530.342,44.114 535.114,39.342 541,39.342 C546.887,39.342 551.658,44.114 551.658,50 C551.658,55.887 546.887,60.657 541,60.657 M541,33.886 C532.1,33.886 524.886,41.1 524.886,50 C524.886,58.899 532.1,66.113 541,66.113 C549.9,66.113 557.115,58.899 557.115,50 C557.115,41.1 549.9,33.886 541,33.886 M565.378,62.101 C565.244,65.022 564.756,66.606 564.346,67.663 C563.803,69.06 563.154,70.057 562.106,71.106 C561.058,72.155 560.06,72.803 558.662,73.347 C557.607,73.757 556.021,74.244 553.102,74.378 C549.944,74.521 548.997,74.552 541,74.552 C533.003,74.552 532.056,74.521 528.898,74.378 C525.979,74.244 524.393,73.757 523.338,73.347 C521.94,72.803 520.942,72.155 519.894,71.106 C518.846,70.057 518.197,69.06 517.654,67.663 C517.244,66.606 516.755,65.022 516.623,62.101 C516.479,58.943 516.448,57.996 516.448,50 C516.448,42.003 516.479,41.056 516.623,37.899 C516.755,34.978 517.244,33.391 517.654,32.338 C518.197,30.938 518.846,29.942 519.894,28.894 C520.942,27.846 521.94,27.196 523.338,26.654 C524.393,26.244 525.979,25.756 528.898,25.623 C532.057,25.479 533.004,25.448 541,25.448 C548.997,25.448 549.943,25.479 553.102,25.623 C556.021,25.756 557.607,26.244 558.662,26.654 C560.06,27.196 561.058,27.846 562.106,28.894 C563.154,29.942 563.803,30.938 564.346,32.338 C564.756,33.391 565.244,34.978 565.378,37.899 C565.522,41.056 565.552,42.003 565.552,50 C565.552,57.996 565.522,58.943 565.378,62.101 M570.82,37.631 C570.674,34.438 570.167,32.258 569.425,30.349 C568.659,28.377 567.633,26.702 565.965,25.035 C564.297,23.368 562.623,22.342 560.652,21.575 C558.743,20.834 556.562,20.326 553.369,20.18 C550.169,20.033 549.148,20 541,20 C532.853,20 531.831,20.033 528.631,20.18 C525.438,20.326 523.257,20.834 521.349,21.575 C519.376,22.342 517.703,23.368 516.035,25.035 C514.368,26.702 513.342,28.377 512.574,30.349 C511.834,32.258 511.326,34.438 511.181,37.631 C511.035,40.831 511,41.851 511,50 C511,58.147 511.035,59.17 511.181,62.369 C511.326,65.562 511.834,67.743 512.574,69.651 C513.342,71.625 514.368,73.296 516.035,74.965 C517.703,76.634 519.376,77.658 521.349,78.425 C523.257,79.167 525.438,79.673 528.631,79.82 C531.831,79.965 532.853,80.001 541,80.001 C549.148,80.001 550.169,79.965 553.369,79.82 C556.562,79.673 558.743,79.167 560.652,78.425 C562.623,77.658 564.297,76.634 565.965,74.965 C567.633,73.296 568.659,71.625 569.425,69.651 C570.167,67.743 570.674,65.562 570.82,62.369 C570.966,59.17 571,58.147 571,50 C571,41.851 570.966,40.831 570.82,37.631"></path></g></g></g></svg></div><div style="padding-top: 8px;"> <div style=" color:#3897f0; font-family:Arial,sans-serif; font-size:14px; font-style:normal; font-weight:550; line-height:18px;"> View this post on Instagram</div></div><div style="padding: 12.5% 0;"></div> <div style="display: flex; flex-direction: row; margin-bottom: 14px; align-items: center;"><div> <div style="background-color: #F4F4F4; border-radius: 50%; height: 12.5px; width: 12.5px; transform: translateX(0px) translateY(7px);"></div> <div style="background-color: #F4F4F4; height: 12.5px; transform: rotate(-45deg) translateX(3px) translateY(1px); width: 12.5px; flex-grow: 0; margin-right: 14px; margin-left: 2px;"></div> <div style="background-color: #F4F4F4; border-radius: 50%; height: 12.5px; width: 12.5px; transform: translateX(9px) translateY(-18px);"></div></div><div style="margin-left: 8px;"> <div style=" background-color: #F4F4F4; border-radius: 50%; flex-grow: 0; height: 20px; width: 20px;"></div> <div style=" width: 0; height: 0; border-top: 2px solid transparent; border-left: 6px solid #f4f4f4; border-bottom: 2px solid transparent; transform: translateX(16px) translateY(-4px) rotate(30deg)"></div></div><div style="margin-left: auto;"> <div style=" width: 0px; border-top: 8px solid #F4F4F4; border-right: 8px solid transparent; transform: translateY(16px);"></div> <div style=" background-color: #F4F4F4; flex-grow: 0; height: 12px; width: 16px; transform: translateY(-4px);"></div> <div style=" width: 0; height: 0; border-top: 8px solid #F4F4F4; border-left: 8px solid transparent; transform: translateY(-4px) translateX(8px);"></div></div></div></a> <p style=" margin:8px 0 0 0; padding:0 4px;"> <a href=";utm_campaign=loading" style=" color:#000; font-family:Arial,sans-serif; font-size:14px; font-style:normal; font-weight:normal; line-height:17px; text-decoration:none; word-wrap:break-word;">In side project news, I&#39;ve figured out exporting from blender to three.js, including animations. Need to tweak the swing of the arms whilst walking. Pleased with the random head scratch whilst standing still. #blender #threejs</a></p> <p style=" color:#c9c8cd; font-family:Arial,sans-serif; font-size:14px; line-height:17px; margin-bottom:0; margin-top:8px; overflow:hidden; padding:8px 0 7px; text-align:center; text-overflow:ellipsis; white-space:nowrap;">A post shared by <a href=";utm_campaign=loading" style=" color:#c9c8cd; font-family:Arial,sans-serif; font-size:14px; font-style:normal; font-weight:normal; line-height:17px;"> Nick O&#39;Leary</a> (@knolleary) on <time style=" font-family:Arial,sans-serif; font-size:14px; line-height:17px;" datetime="2019-12-13T23:08:12+00:00">Dec 13, 2019 at 3:08pm PST</time></p></div></blockquote> <script async="" src="//"></script> <p>I started making a Raspberry Pi-scale R2-D2 using the 3D printer at work ages ago, but that ground to a halt. Having bought my own printer last year, it’s time I dusted this one off and made some progress.</p> <h2 id="write-something-here-more-often">Write something here more often</h2> <p>I’ve tried and failed this one many times. Writing is a muscle that needs exercising. I struggle with exercise.</p> <hr /> <p>So there we go, thanks for reading this far. Remember to say Hi and have a good 2020.</p> Moving Node-RED to a monorepo with multiple modules 2019-03-21T00:00:00+00:00 <p>There are two logical parts of Node-RED; the runtime where flows run and the editor where flows are edited. Ever since the start of the project, these two parts have been bundled together in a single blob of npm module.</p> <p>With the 0.20 release we’ve just published, the internals of Node-RED have now been split apart into 6 separate npm modules, along with the original <code class="language-plaintext highlighter-rouge">node-red</code> module that now has the task of pulling those modules back together so the user doesn’t know what we’ve done.</p> <p>This post describes how we went about doing that and some of the challenges we faced along the way. If you want to see it for yourself, the code is <a href="">here</a>.</p> <p>This blog post was presented at LNUG in September 2019.</p> <iframe width="560" height="315" src="" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <h3 id="one-vs-many-repositories">One-vs-Many repositories.</h3> <p>When I started looking at how to structure the code to support this approach, I had to decide whether to keep all the code in one repo or to split it into one repo per module.</p> <p>Splitting it out would have made it clear what code belongs to each module and make it easier for new developers to follow the structure.</p> <p>But it would also greatly increase the administrative burden; multiple repositories to manage, with multiple issue lists and the need to carefully co-ordinate pull-requests when a new feature spanned multiple modules.</p> <p>Keeping the code in one place made the most sense. The question was then how to do that in practice.</p> <h3 id="tooling">Tooling</h3> <p>I looked around at other projects that maintain multiple modules in a single repository. There seemed to be a split between ones that chose to use established tooling, such as <a href="">Lerna</a> and those that rolled their own solution.</p> <p>Not wanting to needlessly reinvent the wheel, I spent some time playing with Lerna.</p> <blockquote> <p>Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.</p> </blockquote> <p>The problem I found was Lerna has grown over time and does a lot. That isn’t necessarily a bad thing, but I found it hard to visualise how we’d migrate into using it. I got too caught up bouncing between the different options it provides without settling on one approach.</p> <p>I also found <a href="">this post</a> from Nolan Lawson on why PouchDB moved away from Lerna that gave some hands-on perspective - albeit from a while ago now.</p> <p>Ultimately I decided I wanted to understand the code structure and consequences of the split, rather than instant delegate that to the tooling. It wouldn’t preclude us from adopting Lerna in the future - but we’d be able to do that better informed.</p> <h3 id="laying-out-the-code">Laying out the code</h3> <p>The existing code structure was already split between the node.js based runtime and the browser-based editor:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── editor // All editor src and resources ├── nodes // The default core nodes ├── red // The node.js runtime code └── test // All test material </code></pre></div></div> <p>The primary target for repackaging was all of the node.js code under the <code class="language-plaintext highlighter-rouge">red</code> directory. The code was already reasonable well componentised under there, but it wasn’t perfect. It was littered with <code class="language-plaintext highlighter-rouge">require</code> statements with relative paths that made assumptions about where particular files were.</p> <p>The main challenge was figuring out a code layout that would allow the <code class="language-plaintext highlighter-rouge">require</code> statements to be updated to the new module structure, whilst still just working when run in the development environment.</p> <p>This is where I came across an approach linked to in Nolan’s post - <a href="">the “Alle” model</a>.</p> <p>First a quick detour into how node does module loading. When you call <code class="language-plaintext highlighter-rouge">require</code> with a relative path, node loads that file directly. For example, given a pair of files in adjacent directories:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>. ├── a │   └── index.js └── b └── index.js </code></pre></div></div> <p>The code in <code class="language-plaintext highlighter-rouge">a/index.js</code> can use the following code to load <code class="language-plaintext highlighter-rouge">b/index.js</code>:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const moduleB = require("../b/index.js"); </code></pre></div></div> <p>Now, lets say <code class="language-plaintext highlighter-rouge">a</code> and <code class="language-plaintext highlighter-rouge">b</code> are properly formed npm modules - so they include a <code class="language-plaintext highlighter-rouge">package.json</code> file. Rather than require using a relative path, we want to require using the name of the module:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>const moduleB = require("b"); </code></pre></div></div> <p>When you pass the name of a module to <code class="language-plaintext highlighter-rouge">require</code>, node will check the current directory for a <code class="language-plaintext highlighter-rouge">node_modules</code> directory and look in there for a module with that name. If it doesn’t find one, it then checks the parent directory for <code class="language-plaintext highlighter-rouge">node_modules</code> and so on until it reaches the root of the filesystem.</p> <p>We could take advantage of that in layout out the code - by adding a <code class="language-plaintext highlighter-rouge">node_modules</code> directory in the structure:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>. └── node_modules ├── a │   └── index.js └── b └── index.js </code></pre></div></div> <p>Now, when module <code class="language-plaintext highlighter-rouge">a</code> does <code class="language-plaintext highlighter-rouge">require("b")</code>, node will search up the directory structure find the <code class="language-plaintext highlighter-rouge">node_modules</code> directory and then find module <code class="language-plaintext highlighter-rouge">b</code> in there.</p> <p>Putting this into practice led to a structure of:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>├── packages │   └── node_modules │   ├── @node-red │   │   ├── editor-api │   │   ├── editor-client │   │   ├── nodes │   │   ├── registry │   │   ├── runtime │   │   └── util │   └── node-red └── test ├── editor ├── node_modules │   └── nr-test-utils ├── nodes └── unit ├── @node-red │   ├── editor-api │   ├── registry │   ├── runtime │   └── util └── node-red └── lib </code></pre></div></div> <p>You can see the seven new modules under <code class="language-plaintext highlighter-rouge">packages/node_modules</code> directory. They can now <code class="language-plaintext highlighter-rouge">require</code> each other just as they will when properly installed with npm.</p> <p>The test material was also restructured to match the layout. You may also spot the same trick was used there to make a module called <code class="language-plaintext highlighter-rouge">nr-test-utils</code> available to the test material.</p> <p>This module provides two functions: <code class="language-plaintext highlighter-rouge">require</code> and <code class="language-plaintext highlighter-rouge">resolve</code>. They can be used by the test material to require a particular file from the source tree without having to hardcode the relative path from the test material tree into the source tree.</p> <h3 id="getting-github-to-not-ignore-node_modules">Getting GitHub to not ignore node_modules</h3> <p>One downside of this approach is that many IDEs are told to ignore <code class="language-plaintext highlighter-rouge">node_modules</code> directories as they don’t typically contain code a developer is expected to edit.</p> <p>Adding some rules to <code class="language-plaintext highlighter-rouge">.gitignore</code> to not ignore these directories seemed to fix it for my preferred editor, atom.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node_modules !packages/node_modules !test/**/node_modules </code></pre></div></div> <p>We also found that GitHub would not generate diffs when showing changes to any files under those directories, so we had to add <code class="language-plaintext highlighter-rouge">.gitattributes</code> file containing the following:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>/packages/node_modules/** linguist-generated=false </code></pre></div></div> <p>That works on the desktop view, but the mobile view still suppresses diffs - not found a solution for that yet.</p> <h3 id="managing-dependencies">Managing dependencies</h3> <p>Each of the module directories has its own <code class="language-plaintext highlighter-rouge">package.json</code> listing its dependencies as normal. There is also a <code class="language-plaintext highlighter-rouge">package.json</code> file at the top level of the project that lists all dependencies (including development dependencies). This means we don’t have to run <code class="language-plaintext highlighter-rouge">npm install</code> in each module directory - in fact, we actively avoid doing that because those dependencies include references to our other modules that npm won’t be able to install itself.</p> <p>This does mean there is an overhead around managing the dependencies.</p> <p>Any new dependency needs adding in two places; the top-level package.json file so it gets installed in the development environment and the module’s own package.json file so it gets installed when the published module is installed.</p> <p>As soon as you have the same piece of information in two places, you raise the risk of them getting out of sync.</p> <p>The ‘Alle’ method I linked to earlier talks about automating the generation of the individual package.json files - something we haven’t adopted.</p> <p>Instead, to help manage this, a new script was added, <code class="language-plaintext highlighter-rouge">scripts/verify-package-dependencies.js</code> that checks that every dependency listed in individual module package.json files is also listed in the top-level package.json and that the version specifier matches.</p> <p>The default set of tests now includes this check, so a build will fail if a mismatch is found. The script can also be run with <code class="language-plaintext highlighter-rouge">--fix</code> to automatically update the versions in the module package.json file to match the top level one.</p> <p>The one scenario this doesn’t catch is when a new dependency is added to the top-level file, but not to the individual module. The unit tests will still pass because the module is installed at the top level - but the published module will be missing it. We’ll have to be careful around that until we plug the gap.</p> <h3 id="managing-versions">Managing versions</h3> <p>Another important design decision was how to manage the version numbers of the individual modules. For example, if a fix was needed in one module would we publish a new version of just that module, or would we bump the version of all the modules.</p> <p><code class="language-plaintext highlighter-rouge">npm</code> makes it easy to take either approach. If we wanted to be able to publish modules individually, we could set the dependency version numbers to <code class="language-plaintext highlighter-rouge">0.20.x</code> so they would always get the latest version available in a given minor release. The alternative would be to set them to a specific version to tie all the modules at a particular level.</p> <p>The problem with not keeping the versions closely aligned is what happens when a user hits a problem. The main goal of this entire refactoring was to hide the internal details of the split.</p> <p>If a user hits a problem today, they can tell us what version of Node-RED they have installed with a single number. It simply wouldn’t meet our goal if they had to provide the versions of all seven modules so would know exactly what they had installed. It would also be confusing to say they need to update their install to get a fix but the <code class="language-plaintext highlighter-rouge">node-red</code> module version doesn’t change.</p> <p>So for now we’re going to keep all the modules in sync.</p> <p>To help with that task, another script was added, <code class="language-plaintext highlighter-rouge">scripts/set-package-version.js</code> that can be used to update all of the individual <code class="language-plaintext highlighter-rouge">package.json</code> files with the right version number.</p> <h3 id="building-a-release">Building a release</h3> <p>We already had a build task, <code class="language-plaintext highlighter-rouge">grunt release</code>, that took the source tree and built the module that can be published to npm along with a zip file to upload to the GitHub release.</p> <p>That task has been updated to now build the 7 individual modules, this time packed as tgz files.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>.dist ├── modules │   ├── node-red-0.20.3.tgz │   ├── node-red-editor-api-0.20.3.tgz │   ├── node-red-editor-client-0.20.3.tgz │   ├── node-red-nodes-0.20.3.tgz │   ├── node-red-registry-0.20.3.tgz │   ├── node-red-runtime-0.20.3.tgz │   └── node-red-util-0.20.3.tgz └── </code></pre></div></div> <p>Those tgz files can be published one at a time to npm - ensuring the <code class="language-plaintext highlighter-rouge">node-red</code> module is done last. Currently that’s a manual task and something that is ripe for automating in the future.</p> <h3 id="what-next">What next?</h3> <p>On reflection, having published the 0.20 release and a handful of subsequent maintenance releases, I’m pretty happy with the approach we took. Aside from a few more bits of task automation we could add, I think we have created a project structure that is well defined and easy to work with.</p> <p>The main success has been that we did this without a single issue from a user related to the module structure or packaging. Users have no idea we made these changes - unless they read the release notes where we keep talking about it.</p> Ctrl-Shift-r 2019-03-14T00:00:00+00:00 <p>For both of you who follow this blog, you’ll notice it’s had a pretty major face-lift.</p> <p>There’s also a strong chance I’ve completely broken the RSS feed, so it’s entirely possible I’m typing into the void.</p> <p>I got bored of the old theme and have wanted to give it an big update for a while. But I simply couldn’t be bothered to go back to hacking around WordPress themes.</p> <p>I also decided to shift from WordPress to a statically hosted site. I prefer writing in Markdown and not getting distracted by all the bells and whistles of an editor. I realise it isn’t for everyone, but pushing some markdown to a git repo is a natural part of my daily workflow already, so it made sense to move over to GitHub Pages.</p> <p>The main challenge was getting all of the existing content out of WordPress.</p> <p>I found <a href="">this</a> script that can convert a WordPress export into Markdown files. It also tries to download any uploaded content the posts link to. It wasn’t perfect and took some tweaking, but it got me to a workable state. Thankfully I’m not a prolific writer, so an hour of going through each post and doing some final tidy up was all it took to give things a final polish.</p> <p>As for the theme itself, some trawling around found <a href="">this theme</a> that I liked the look of and it didn’t take much to customise it a bit to my tastes.</p> <p>The final piece was replicating some of the non-blog content I was also hosting on there. Most of that has copied over, but a <a href="/paper-graphs">couple custom php scripts</a> are now defunct and I’ll think about what to do with them another day.</p> <p>If you do read this post, please do say ‘Hi’ through some means. It’d be nice to know you exist.</p> Deploying Node-RED applications to devices using Balena 2018-09-27T00:00:00+00:00 <p>My last couple of posts have focused on creating a managed Node-RED deployment pipeline to IBM Cloud. There’s still more to do in that series, but for this one, I’m taking a bit of a detour to the edge of the network.</p> <p>One of the strengths of Node-RED is that it runs on devices just as happily as it does in cloud environments. This post looks at how we can replicate the deployment pipeline model from the first post, but this time target devices running in remote locations.</p> <p>To do this, we’re going to use <a href="">Balena</a> - a platform for managing fleets of connected devices, that makes updating them as easy as doing a <code class="language-plaintext highlighter-rouge">git push</code>. I’ve been meaning to play with Balena for ages and having got this working in no time today, I’m a tiny bit in love.</p> <p><em>2020-02: When I wrote the original version of this blog post, Balena were known as Not long after I published it, they renamed themselves to Balena. I’ve finally got around to updating this post to reflect the change.</em></p> <hr /> <h3 id="getting-started">Getting started</h3> <p>Before we begin, you’ll need:</p> <ul> <li>A <a href="">Balena</a> account - their free plan lets you create one application with up to 10 devices</li> <li>A pair of Raspberry Pis</li> </ul> <p>For this guide, I’m going to use two Raspberry Pis; one as the ‘development’ machine and one as a target device to push updates to via Balena. You could just as easily use your laptop as the development machine to get started.</p> <hr /> <h3 id="create-a-new-node-red-project">Create a new Node-RED project</h3> <p>As with the <a href="/2018/06/01/creating-a-node-red-deployment-pipeline-to-ibm-cloud/">previous post</a> you’ll need to enabled the Projects feature in Node-RED.</p> <p>On the ‘development’ Pi, edit your <code class="language-plaintext highlighter-rouge">settings.js</code> file to set <code class="language-plaintext highlighter-rouge">editorTheme.projects.enabled</code> to true. You may find there’s already an <code class="language-plaintext highlighter-rouge">editorTheme</code> entry in the file with a <code class="language-plaintext highlighter-rouge">menu</code> property - you’ll need to add in the <code class="language-plaintext highlighter-rouge">projects</code> property alongside that:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> editorTheme: { projects: { enabled: true }, menu: { ... } } </code></pre></div></div> <p>Then restart Node-RED using <code class="language-plaintext highlighter-rouge">node-red-stop &amp;&amp; node-red-start</code>. If you aren’t using a Pi, you’ll need to use whatever platform-appropriate means you have to restart.</p> <p>At this point you can either follow the previous post to create a GitHub repository for your project and clone it locally, or you can create an entirely local project. This guide doesn’t make use of GitHub, but the option is there if you want.</p> <p>The most important thing is to make a note of the key you choose to encrypt your credentials file with - you’ll need that later.</p> <hr /> <h3 id="turn-the-project-into-a-deployable-application">Turn the project into a deployable application</h3> <p>Once again, we need to edit some of the project files so it can be deployed as a standalone application.</p> <p>The <code class="language-plaintext highlighter-rouge">package.json</code> file is updated as before - make sure to leave the <code class="language-plaintext highlighter-rouge">node-red</code> section alone if you’ve picked different flow file names when creating the project:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>{ "name": "node-red-demo-1", "description": "A Node-RED Project", "version": "0.0.1", "dependencies": { "node-red": "0.19.*", "node-red-node-pi-sense-hat": "&gt;0.0.18" }, "node-red": { "settings": { "flowFile": "flow.json", "credentialsFile": "flow_cred.json" } }, "scripts": { "start": "node --max-old-space-size=160 ./node_modules/node-red/red.js --userDir . --settings ./settings.js flow.json" } } </code></pre></div></div> <p>I’ve included the <code class="language-plaintext highlighter-rouge">node-red-node-pi-sense-hat</code> nodes as the demo I’m building beyond this guide uses that particular accessory.</p> <h4 id="add-a-settings-file">Add a settings file</h4> <p>The <code class="language-plaintext highlighter-rouge">settings.js</code> file can be created with the following:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>module.exports = { credentialSecret: process.env.NODE_RED_CREDENTIAL_SECRET, httpAdminRoot: false } </code></pre></div></div> <p>By setting <code class="language-plaintext highlighter-rouge">httpAdminRoot</code> to <code class="language-plaintext highlighter-rouge">false</code>, the editor and admin apis will be disabled.</p> <h4 id="add-a-dockerfile">Add a Dockerfile</h4> <p>Balena uses docker images as the unit of deployment. To that end, we need to add a <code class="language-plaintext highlighter-rouge">Dockerfile</code> to build our application:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>FROM balenalib/raspberrypi3-node:8-slim # use apt-get if you need to install dependencies, # for instance if you need ALSA sound utils, just uncomment the lines below. #RUN apt-get update &amp;&amp; apt-get install -yq \ # alsa-utils libasound2-dev &amp;&amp; \ # apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/* RUN apt-get update &amp;&amp; apt-get install -yq \ python3=3.4.2-2 sense-hat raspberrypi-bootloader i2c-tools build-essential \ libssl-dev libffi-dev libyaml-dev python3-dev python3-pip python-rpi.gpio &amp;&amp; \ pip3 install sense-hat rtimulib pillow # Defines our working directory in container WORKDIR /usr/src/app # Copies the package.json first for better cache on later pushes COPY package.json package.json # This install npm dependencies on the Balena build server, # making sure to clean up the artifacts it creates in order to reduce the image size. RUN JOBS=MAX npm install --unsafe-perm &amp;&amp; npm cache clean --force &amp;&amp; rm -rf /tmp/* RUN apt-get remove build-essential libssl-dev libffi-dev libyaml-dev python3-dev python3-pip \ &amp;&amp; apt-get autoremove &amp;&amp; apt-get clean &amp;&amp; rm -rf /var/lib/apt/lists/* # This will copy all files in our root to the working directory in the container COPY . ./ # Enable systemd init system in container ENV INITSYSTEM on # server.js will run when container starts up on the device CMD ["npm", "start"] </code></pre></div></div> <p>I won’t go into all the details of the Dockerfile. It took a bit of trial and error to get the right dependencies installed for the SenseHAT.</p> <h3 id="commit-and-push-changes">Commit and push changes</h3> <p>At this point, all of the necessary changes have been made to the project files and you should commit the changes. You can do this either via the git command-line, or from within Node-RED.</p> <h4 id="committing-via-the-command-line">Committing via the command-line</h4> <p>From within the project directory, <code class="language-plaintext highlighter-rouge">~/.node-red/projects/&lt;name-of-project&gt;</code>, run the commands:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git add package.json Dockerfile settings.js git commit -m "Update project files" </code></pre></div></div> <h4 id="committing-via-node-red">Committing via Node-RED</h4> <p>Within Node-RED, open up the <code class="language-plaintext highlighter-rouge">history</code> sidebar tab. You should see the changed files in the ‘Local files’ section. If you don’t, click the refresh button to update the view. When you hover over each file a <code class="language-plaintext highlighter-rouge">+</code> button will appear on the right - click that button to move the file down to the ‘Changes to commit’ section.</p> <p>Once they are all staged, click the ‘commit’ button, enter a commit message and confirm.</p> <p>Switch to the <code class="language-plaintext highlighter-rouge">Commit History</code> section and you should see two commits in the list - the initial <code class="language-plaintext highlighter-rouge">Create project files</code> commit and the commit you’ve just done.</p> <h3 id="setting-up-balena">Setting up Balena</h3> <p>Next we’re going to get the second Pi setup as a managed device. Balena provide a great <a href="">getting started tutorial</a> that you should follow up to and including the ‘Provision your device’ step.</p> <p>That should get to the point where the Pi shows as connected in your Balena dashboard.</p> <h3 id="deploying-your-application">Deploying your application</h3> <p>Before we can get deploy the application, we need to:</p> <ul> <li>setup some SSH keys so our development Pi is able to push changes to Balena,</li> <li>tell Node-RED where to push the application,</li> <li>configure your device with the credential key</li> </ul> <h4 id="generate-ssh-keys">Generate SSH keys</h4> <p>Open up the Node-RED settings dialog (from the main menu) and switch to the ‘Git config’ tab. Click the ‘add key’ button, give it a name, and optionally a passphrase, then click ‘generate key’. After a few seconds you’ll be shown you new public key, which you should copy to your clipboard.</p> <p>Over in your <a href="">Balena preferences</a> you can then add that public key to your account.</p> <h4 id="configure-your-git-remote">Configure your git remote</h4> <p>Balena provides a remote git repository you push your application to in order to trigger a deployment. In your applications dashboard page you should see a text input with a <code class="language-plaintext highlighter-rouge">git remote</code> command in. The full command will look like:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git remote add balena &lt;user&gt;;user&gt;/&lt;app&gt;.git </code></pre></div></div> <p>You can either run that command in the directory <code class="language-plaintext highlighter-rouge">~/.node-red/projects/&lt;name-of-project&gt;</code>, or you can add it via the Node-RED editor. To do it via the editor, open up the Project settings dialog by clicking the <code class="language-plaintext highlighter-rouge">...</code> button next to the project name in the Info sidebar tab and then switch to the Settings tab. Click ‘add remote’, give it a name of <code class="language-plaintext highlighter-rouge">balena</code> and copy in the url from the command.</p> <h4 id="tell-your-device-its-credential-key">Tell your device its credential key</h4> <p>The final step is to give your device the key to decrypt its credentials file. We already setup the <code class="language-plaintext highlighter-rouge">settings.js</code> file to look for the <code class="language-plaintext highlighter-rouge">NODE_RED_CREDENTIAL_SECRET</code> environment variable. In the Balena application dashboard you should add a new Environment Variable with that name and the value you gave when generating your project.</p> <h3 id="push-your-application">Push your application</h3> <p>At last we can push the project to Balena. In the Commit History section of the Node-RED Project History sidebar tab, click the button with two arrows. Click the ‘Remote: none’ button to pick which remote to push to. It may prompt you to pick an ssh key at this point - pick the one you generated earlier. If you originally cloned the project from github, you’ll need to pick the ‘<code class="language-plaintext highlighter-rouge">balena</code>’ remote. Finally click ‘push’.</p> <p>If all goes well, your project will be pushed to Balena and its docker image built.</p> <p>Now, confession time. In putting this guide together, I’ve hit a couple usability issues with the git integration in Node-RED. When pushing to a remote git repository, you don’t get any feedback - just a spinning animation. That’s not normally much of an issue as it is fairly quick, but in the Balena case, when you push it runs the build and provides the full log in return. That’s invaluable in figuring out what you got wrong in the Dockerfile - but Node-RED doesn’t show you any of it. It can also take a minute or two - longer the first time you do it, but quicker in subsequent pushes.</p> <p>I did find myself resorting to pushing from the command line, using <code class="language-plaintext highlighter-rouge">git push balena master</code> in the project directory so I could see that output. Something for us to improve in the future.</p> <p>Assuming your push worked, you should see the update arrive in the Balena dashboard and be able to track its deployment to your device. The dashboard also shows you the application logs <em>and</em> lets you ssh into your device from the browser. It really is a delightful experience.</p> <p>All being well, you should see Node-RED startup in the device logs.</p> <h3 id="developer-workflow">Developer workflow</h3> <p>With everything in place, you should now have a developer workflow that consists of:</p> <ol> <li>developing and testing on a local Raspberry Pi</li> <li>committing changes, giving you full version control</li> <li>optionally pushing those changes to GitHub, or any other hosted git service</li> <li>deploying the application to your managed devices, with a simple push to the Balena remote.</li> </ol> <p>One thing worth looking at is to enable the ‘Delta updates’ feature on Balena. This keeps the size of any update to a bare minimum - which if its mostly just your flow file can be a huge saving compared to pushing a complete docker layer. I’m not entirely sure why this feature isn’t enabled by default, but you can find out how to turn it on <a href="">here</a>.</p> <h3 id="next-steps">Next steps</h3> <p>With this basic workflow in place, we can start thinking about how it would work with multiple devices. The great thing with Balena is that it’ll push your application to all of the connected devices. That does pose some challenges for us. For example, if we want each device to connect to Watson IoT Platform with its own set of credentials, we need a mechanism to give each device its own details <em>and</em> be able to make use of that in the Node-RED flow configuration. Thankfully you can go a long way with a few environment variables. But that’s for another post.</p> Connecting to IBM Cloud services from Node-RED 2018-06-05T00:00:00+00:00 <p>In the <a href="/2018/06/01/creating-a-node-red-deployment-pipeline-to-ibm-cloud/">first part of this series</a> I showed how to create a deployment pipeline from Node-RED running locally to Node-RED running in IBM Cloud.</p> <p>It got the basic pieces into place to let you deploy a version controlled Node-RED application to the cloud. The next task is to connect some other IBM Cloud services to the application.</p> <h4 id="ibm-cloud-aware-nodes">IBM Cloud-aware nodes</h4> <p>The existing Node-RED boilerplate comes with some extra nodes that are IBM Cloud-aware. They are able to automatically detect available instances of their respective services using the <code class="language-plaintext highlighter-rouge">VCAP_SERVICES</code> environment variable that Cloud Foundry provides.</p> <p>One such collection of nodes are for the Cloudant database service, which we’re going to add to our Node-RED application.</p> <p>The challenge is how to develop against those nodes when running locally - outside of the IBM Cloud environment.</p> <hr /> <h3 id="setting-up-cloudant">Setting up Cloudant</h3> <h4 id="create-a-cloudant-service-instance">Create a Cloudant service instance</h4> <p>Open up the <a href="">IBM Cloud catalog</a> and select the ‘Cloudant NoSQL DB’ service. Create a new instance, making sure you select the same region as your Node-RED application.</p> <h4 id="bind-cloudant-to-your-node-red-application">Bind Cloudant to your Node-RED application</h4> <p>Go to the dashboard page for your Node-RED application and select the ‘Connections’ tab. Find your newly created Cloudant service in the list and click ‘connect’.</p> <p><img src="/blog/content/2018/06/Application_Details_-_IBM_Cloud-1-1024x579.png" alt="" /></p> <p>It will prompt you to restage the application which will take a couple of minutes to complete.</p> <p>Once that’s done, go back to the ‘Runtime’ tab on the IBM Cloud dashboard and the environment variables section. You will see a section for <code class="language-plaintext highlighter-rouge">VCAP_SERVICES</code> - this is the environment variable that the platform uses to pass the application all of the details it needs to access the connected services. You should see an entry for our newly created cloudant instance - if you don’t, make sure the restage has completed and reload the page.</p> <p>Beneath the credentials is an ‘export’ button - clicking that will download a copy to a file called <code class="language-plaintext highlighter-rouge">&lt;your-app-name&gt;_vcap.json</code>.</p> <p>Copy that file into your Node-RED user directory, <code class="language-plaintext highlighter-rouge">~/.node-red</code> - do <em>not</em> put it under version control.</p> <p>Edit your local <code class="language-plaintext highlighter-rouge">settings.js</code> file - this is the one in <code class="language-plaintext highlighter-rouge">~/.node-red</code> <em>not</em> the one in your project directory.</p> <p>Add the following just above the <code class="language-plaintext highlighter-rouge">module.exports</code> line and then restart Node-RED. Make sure to replace <code class="language-plaintext highlighter-rouge">&lt;your-app-name&gt;_vcap.json</code> with the actual name of the file you downloaded.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>var fs = require("fs"); var path = require("path"); // Load and export IBM Cloud service credentials process.env.VCAP_SERVICES = fs.readFileSync(path.join(__dirname,"&lt;your-app-name&gt;_vcap.json")); </code></pre></div></div> <p>Your local Node-RED now has access to your service credentials in the same way as your Node-RED in IBM Cloud does.</p> <h4 id="install-the-ibm-cloud-enabled-cloudant-nodes">Install the IBM Cloud-enabled Cloudant nodes</h4> <p>Open up the Palatte Manager from the drop-down menu in Node-RED. Go to the ‘Install’ tab, search for <code class="language-plaintext highlighter-rouge">node-red-node-cf-cloudant</code> and click install.</p> <p>Once installed, you’ll have a new pair of Cloudant nodes in the ‘storage’ section of the palette. Drag one into your workspace and double-click to edit it. The ‘Service’ property should have the name of your Cloudant service listed. If it doesn’t, check you’ve follow the steps to get your <code class="language-plaintext highlighter-rouge">VCAP_SERVICES</code> setup correctly.</p> <p>Close the edit dialog but <em>do not</em> delete the node - we’ll come back to this a bit later.</p> <h4 id="add-the-cloudant-nodes-to-the-project">Add the Cloudant nodes to the project</h4> <p>Having installed the nodes locally, we need to add them to our project’s <code class="language-plaintext highlighter-rouge">package.json</code> file so they also get installed when deploying to the cloud. We can do this within Node-RED by going to the ‘information’ sidebar tab and clicking the button next to the project name. This opens up the Project Settings dialog.</p> <p>Go to the ‘Dependencies’ tab where you’ll see a list of the modules our project depends on. This is a combination of modules already listed in <code class="language-plaintext highlighter-rouge">package.json</code> and modules which provide nodes we have in our flow. At this point you should have two entries <code class="language-plaintext highlighter-rouge">node-red</code> and <code class="language-plaintext highlighter-rouge">node-red-node-cf-cloudant</code>.</p> <p>Ignore the offer to remove <code class="language-plaintext highlighter-rouge">node-red</code> from the project as we need that, but do click the ‘add to project’ button next to the Cloudant module.</p> <p><img src="/blog/content/2018/06/BP8A11Ltw2.gif" alt="" /></p> <h4 id="commit-changes">Commit changes</h4> <p>If you switch back to the ‘history’ tab you should now have two entries in the ‘Local files’ section - <code class="language-plaintext highlighter-rouge">manifest.yml</code> and <code class="language-plaintext highlighter-rouge">package.json</code>. If you click on either filename it will show you a diff of what has changed in the file. Check the changes look correct then click the ‘+ all’ button to prepare both files for committing and then commit them. Switch to the ‘Commit History’ tab and push the changes up to GitHub.</p> <p>Wait for the Travis build to redeploy you application and then reload it your browser. You should now have the Cloudant nodes available in the palette and, as before, when you add one to your workspace and edit it, your Cloudant service will be selected.</p> <h3 id="separating-development-and-production">Separating development and production</h3> <p>At this point, an application built locally will use the same Cloudant instance as the one running in IBM Cloud.</p> <p>If we consider the local Node-RED as a <em>development</em> environment and the IBM Cloud instance as the <em>production</em> environment, then they really should use separate instances.</p> <p>This can be achieved by creating a second Cloudant instance to treat as the <em>development</em> instance. Rather than connect it to your Node-RED application, you can generate a set of credentials from the instance dashboard page.</p> <p><img src="/blog/content/2018/06/Service_Details_-_IBM_Cloud-1024x512.png" alt="" /></p> <p>Update the <code class="language-plaintext highlighter-rouge">&lt;your-app-name&gt;_vcap.json</code> file with the new credentials and after restarting Node-RED, your local instance will now be accessing the separate instance.</p> <hr /> <h3 id="next-steps">Next Steps</h3> <p>This post has shown how to connect IBM Cloud services to you Node-RED application with separate development and production instances. It’s another important step to creating production-ready applications with Node-RED in IBM Cloud.</p> <p>In the next part of this series, we’ll look at how to start building a simple application using this setup. That’s what I said in the previous post, but I really mean it this time.</p> Creating a Node-RED deployment pipeline to IBM Cloud 2018-06-01T00:00:00+00:00 <p><a href="">Node-RED</a> has been available in the <a href="">IBM Cloud catalog</a> since the early days of what was then known as IBM Bluemix. Provided as a boilerplate application, it gives a really quick and easy way to get started with both Node-RED and the wide range of services available in the catalog.</p> <p>The boilerplate is optimised for ease of use and getting started. Applications running in the Cloud Foundry part of IBM Cloud cannot treat their local file system as persistent storage - any time the application is restarted the file system is wiped back to its original state. This is why the Node-RED boilerplate comes with an instance of the Cloudant database service; giving it somewhere to store its flows.</p> <p>It also means that any nodes that are installed using the Palette Manager in the editor have to be dynamically reinstalled whenever the application restarts. This is not ideal as it takes longer to restart, exposes the application to random network/npm failures and also risks memory issues as Node-RED tries to reinstall multiple things at once.</p> <p>The better solution is to enable the Continuous Deployment feature and edit the application’s <code class="language-plaintext highlighter-rouge">package.json</code> file to explicitly add any additional modules. That’s also good as it means your application is version controlled and can be easily restored.</p> <p>Except that isn’t entirely true. The underlying Node-RED application might be version controlled, but the most valuable part, the flows, are still held in Cloudant.</p> <p>In an ideal world, you’d have all of your application assets under a single source of version control. It should be possible to deploy that application to separate development, test and production environments. It should all fit in with more traditional developer workflows.</p> <p>This is the first in a series of posts that will show how you can create just such a workflow.</p> <p>You’ll be able to develop a Node-RED application on a local machine, pushing changes to a GitHub repository and have them deploy automatically to IBM Cloud using Travis CI.</p> <hr /> <h3 id="getting-started">Getting started</h3> <p>Before we begin, you’ll need:</p> <ul> <li>A <a href="">GitHub</a> account - it’s free!</li> <li>A <a href="">Travis CI</a> account - sign-up using your GitHub account - it’s free!</li> <li>An <a href="">IBM Cloud</a> account - sign-up for a <a href="">Lite Account</a>; it’s free, doesn’t require a credit card, never expires and gives you enough resources to get started</li> </ul> <p>You’ll also need <a href="">Node-RED</a> installed locally.</p> <hr /> <h3 id="create-a-new-node-red-project">Create a new Node-RED project</h3> <p>Node-RED introduced the Projects feature in the <a href="">0.18 release</a>. It allows you to manage your flows in a git repository along with all the other pieces you need to create a redistributable Node-RED application.</p> <h4 id="enabling-the-node-red-projects-feature">Enabling the Node-RED projects feature</h4> <p>In the 0.18 release, the Projects feature needs to be enabled. Edit your <code class="language-plaintext highlighter-rouge">settings.js</code> file and update the <code class="language-plaintext highlighter-rouge">editorTheme</code> setting to change the <code class="language-plaintext highlighter-rouge">projects.enabled</code> flag to <code class="language-plaintext highlighter-rouge">true</code>. If you don’t have an <code class="language-plaintext highlighter-rouge">editorTheme</code> setting, add one in:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code> editorTheme: { projects: { enabled: true } } </code></pre></div></div> <p>You can see how it should look in our <a href="">default settings file</a> - although the default is set to disable the feature, so if you copy it, make sure you change it to <code class="language-plaintext highlighter-rouge">true</code>.</p> <p>When you restart Node-RED, you’ll be shown a welcome screen that introduces the projects feature.</p> <h4 id="create-a-new-github-repository">Create a new GitHub repository</h4> <p>To create our Node-RED project, we’re going to first create a new repository on GitHub for the project.</p> <p>Login to <a href="">GitHub</a>, click the <code class="language-plaintext highlighter-rouge">New repository</code> option under the <code class="language-plaintext highlighter-rouge">+</code> menu in the header.</p> <p><img src="/blog/content/2018/06/knolleary__Nick_O_Leary_-1024x193.png" alt="" /></p> <p>Give your repository a name and leave all of the options as they are - in particular, do <strong>not</strong> tick the <code class="language-plaintext highlighter-rouge">Initialize this repository with a README</code> option. Then click ‘Create repository’.</p> <p>On the repository page, copy the git url to your clipboard as we’ll need it in the next step.</p> <p><img src="/blog/content/2018/06/knolleary_node-red-demo-1024x152.png" alt="" /></p> <h4 id="clone-the-repository-into-a-new-project">Clone the repository into a new project</h4> <p>Back in Node-RED, select the option to create a new project by cloning a repository. When prompted, paste in the git url from the previous step.</p> <p>Once you create the project, you’ll get a message saying it’s empty and it will offer to create a default set of project files - an offer you should accept.</p> <p><img src="/blog/content/2018/06/My_Node-RED-1024x374.png" alt="" /></p> <p>It will then prompt you for the name of the flow file to use - we’ll use <code class="language-plaintext highlighter-rouge">flow.json</code>. Next it will ask about encrypting your flow credentials - something you must enable as you will be publishing your flow to GitHub. Provide an encryption key and make a note of it for later on.</p> <p>With that done, you’ll now have your project ready to start wiring up your flows.</p> <hr /> <h3 id="modify-the-project-to-run-on-ibm-cloud">Modify the project to run on IBM Cloud</h3> <p>In order to deploy your Node-RED project as a Cloud Foundry application on IBM Cloud we need to add some extra files and update an existing one. These changes need to be made outside of Node-RED in a text editor of your choice.</p> <p>First we need to find the project files. Node-RED stores them in a directory under the runtime user directory. By default, that will be <code class="language-plaintext highlighter-rouge">~/.node-red/projects/&lt;name-of-project&gt;</code>.</p> <h4 id="update-packagejson">Update package.json</h4> <p>The project already has a default <code class="language-plaintext highlighter-rouge">package.json</code> file that needs some updates:</p> <ul> <li>add <code class="language-plaintext highlighter-rouge">node-red</code> in the dependencies section - this will ensure Node-RED gets installed when the application is deployed.</li> <li>add a <code class="language-plaintext highlighter-rouge">scripts</code> section to define a <code class="language-plaintext highlighter-rouge">start</code> command - this is how IBM Cloud will run the application. We’ll look at this in a bit more detail in a moment.</li> <li> <p>add an <code class="language-plaintext highlighter-rouge">engines</code> section to define what version of Node.js we want to run with. You could leave this out and just get whatever the current Node.js buildpack defaults to, but it is better to be explicit.</p> <p>{ “name”: “node-red-demo-1”, “description”: “A Node-RED Project”, “version”: “0.0.1”, “dependencies”: { “node-red”: “0.18.*” }, “node-red”: { “settings”: { “flowFile”: “flow.json”, “credentialsFile”: “flow_cred.json” } }, “scripts”: { “start”: “node –max-old-space-size=160 ./node_modules/node-red/red.js –userDir . –settings ./settings.js flow.json” }, “engines”: { “node”: “8.x” } }</p> </li> </ul> <p>Lets take a closer look at the start command:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>node --max-old-space-size=160 (1) ./node_modules/node-red/red.js (2) --userDir . (3) --settings ./settings.js (4) flow.json (5) </code></pre></div></div> <ol> <li>As we’re running with a fixed memory limit, this argument is used to tell <code class="language-plaintext highlighter-rouge">node</code> when it should start garbage collecting.</li> <li>With <code class="language-plaintext highlighter-rouge">node-red</code> listed as an npm dependency of the project, we know exactly where it will get installed and where the <code class="language-plaintext highlighter-rouge">red.js</code> main entry point it.</li> <li>We want Node-RED to use the current directory as its user directory</li> <li>Just to be sure, we point at the settings file it should use - something we’ll add in the next step</li> <li>Finally we specify the flow file to use.</li> </ol> <p>With the current version of Node-RED, 0.18, you should restart Node-RED after editing this file. - it doesn’t know the file has changed and may overwrite any changes you’ve made if you modify the project within the editor later.</p> <h4 id="add-a-settings-file">Add a settings file</h4> <p>We need a settings file to configure Node-RED for the IBM Cloud environment. Create a file called <code class="language-plaintext highlighter-rouge">settings.js</code> in the project directory and copy in the following:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>module.exports = { uiPort: process.env.PORT, credentialSecret: process.env.NODE_RED_CREDENTIAL_SECRET, adminAuth: { type: "credentials", users: [], default: { permissions: "read" } } } </code></pre></div></div> <p>This tells Node-RED to listen on the port IBM Cloud gives us from the <code class="language-plaintext highlighter-rouge">PORT</code> environment variable. It also sets the key used to decrypt the credentials - this time coming from the <code class="language-plaintext highlighter-rouge">NODE_RED_CREDENTIAL_SECRET</code> environment variable. That lets us provide the key to the application without having to hardcode it in the version controlled files. We’ll sort that out in a later step of this post.</p> <p>Finally it configures the editor to be in read-only mode. In a future post we’ll turn off the editor entirely, but leaving it running is useful at this stage to help verify your application is running.</p> <h4 id="add-a-manifest-file">Add a manifest file</h4> <p>The next file we need is the <code class="language-plaintext highlighter-rouge">manifest.yml</code> file used to deploy the application. Here’s a minimal file to start with. Make sure you change the <code class="language-plaintext highlighter-rouge">name</code> field to something unique for your project - <code class="language-plaintext highlighter-rouge">nr-demo</code> is already used and will cause your deploy to fail if you don’t change it.</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>applications: - name: nr-demo memory: 256MB instances: 1 </code></pre></div></div> <hr /> <h3 id="configure-travis">Configure Travis</h3> <p>Next we’re going to get Travis to watch our GitHub repository and trigger a build whenever we push changes to it.</p> <h4 id="enable-travis-for-your-repository">Enable Travis for your repository</h4> <p>Sign in to <a href="">Travis</a> and connect it to your GitHub account. Go to your profile page and enable Travis for your new repository. You may have to click the ‘Sync account’ button for it to show up.</p> <p><img src="/blog/content/2018/06/TYzxS6YyVG.gif" alt="" /></p> <h4 id="add-a-travisyml-file">Add a .travis.yml file</h4> <p>The project needs a file called <code class="language-plaintext highlighter-rouge">.travis.yml</code> to tell Travis what to do when it runs a build. A build consists of three phases: <em>install</em>, <em>script</em> and <em>deploy</em>. For the purposes of this exercise, we’re going to skip the <em>install</em> and <em>script</em> phases - they can be used in the future to run automated tests against the application.</p> <p>For the <em>deploy</em> phase we can use an integration Travis already has with IBM Cloud - albeit under the old brand name: <a href="">Bluemix CloudFoundry</a>.</p> <p>With all that in mind, copy the following into your <code class="language-plaintext highlighter-rouge">.travis.yml</code> file:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>language: node_js node_js: - "node" install: true script: echo "Skipping build" deploy: edge: true provider: bluemixcloudfoundry username: apikey organization: space: dev manifest: manifest.yml </code></pre></div></div> <p>You’ll need to set the <code class="language-plaintext highlighter-rouge">organization</code> and <code class="language-plaintext highlighter-rouge">space</code> fields to match your own account details. The <code class="language-plaintext highlighter-rouge">username</code> must be set to <code class="language-plaintext highlighter-rouge">apikey</code> and the next step is to get a password we can use.</p> <h4 id="generate-an-ibm-cloud-api-key">Generate an IBM Cloud API key</h4> <p>We need to generate an API key in our IBM Cloud account which we can use for the Travis deploy.</p> <p>Log in to the <a href="">IBM Cloud dashboard</a> and select <code class="language-plaintext highlighter-rouge">Manage -&gt; Security -&gt; Platform API Keys</code> from the menu in the header.</p> <p>Click the <code class="language-plaintext highlighter-rouge">Create</code> button, enter a sensible name for the key and click <code class="language-plaintext highlighter-rouge">Create</code>. The key will be generated and in the next dialog it will let you copy it to your clipboard. <em>Make sure you copy it</em> - once you close the dialog you will <em>not</em> be able to see it again and you’ll need to generate a new one.</p> <h4 id="add-the-encrypted-api-key-to-your-travisyml">Add the encrypted api key to your .travis.yml</h4> <p>Rather than paste this key into your <code class="language-plaintext highlighter-rouge">.travis.yml</code> file directly, Travis provides a way to encrypt the key first so it can be added safely.</p> <p>To do this, you must first install the Travis CLI. How exactly you do that will depend on your OS and whether you have ruby installed and whether your PATH is setup correctly and lots of other things that may trip you up along the way. Suffice to say, <em>if</em> you have ruby installed, it <em>should</em> be a simple case of running:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>gem install travis </code></pre></div></div> <p>You can then run:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>travis encrypt --add deploy.password </code></pre></div></div> <p>It will prompt you to paste in your api key, hit <code class="language-plaintext highlighter-rouge">enter</code> then <code class="language-plaintext highlighter-rouge">ctrl-d</code>. If you look in your .travis.yml file you should see a <code class="language-plaintext highlighter-rouge">password/secure</code> section added under the <code class="language-plaintext highlighter-rouge">deploy</code> section.</p> <p>If you get errors such as <code class="language-plaintext highlighter-rouge">travis: command not found</code> then you may need to update your <code class="language-plaintext highlighter-rouge">PATH</code> to include wherever <code class="language-plaintext highlighter-rouge">gem</code> installed the package.</p> <hr /> <h3 id="commit-and-push-changes">Commit and push changes</h3> <p>At this point, all of the necessary changes have been made to the project files. You can commit the changes and push them up to GitHub. You can do this either via the git command-line, or from within Node-RED.</p> <h4 id="committing-via-the-command-line">Committing via the command-line</h4> <p>From within the project directory, <code class="language-plaintext highlighter-rouge">~/.node-red/projects/&lt;name-of-project&gt;</code>, run the commands:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>git add package.json .travis.yml manifest.yml settings.js git commit -m "Update project files" git push </code></pre></div></div> <h4 id="committing-via-node-red">Committing via Node-RED</h4> <p>Within Node-RED, open up the <code class="language-plaintext highlighter-rouge">history</code> sidebar tab. You should see the four changed files in the ‘Local files’ section. If you don’t, click the refresh button to update the view. When you hover over each file a <code class="language-plaintext highlighter-rouge">+</code> button will appear on the right - click that button to move the file down to the ‘Changes to commit’ section.</p> <p>Once all four are staged, click the ‘commit’ button, enter a commit message and confirm.</p> <p>Switch to the <code class="language-plaintext highlighter-rouge">Commit History</code> section and you should see two commits in the list - the initial <code class="language-plaintext highlighter-rouge">Create project files</code> commit and the commit you’ve just done.</p> <p>Click the remote button - the one with up/down arrows in - and click <code class="language-plaintext highlighter-rouge">push</code> in the dialog. This will send the changes up to GitHub.</p> <p><img src="/blog/content/2018/06/InGueV8Ebg.gif" alt="" /></p> <hr /> <h3 id="watch-your-build">Watch your build</h3> <p>If you go back to Travis, you should see the commit trigger a new build against your repository. If all is well, two to three minutes later the build should pass and you should be able to open <code class="language-plaintext highlighter-rouge">http://&lt;name-of-app&gt;</code> and be welcomed by the Node-RED editor.</p> <p>If it fails, check the build log to see what went wrong.</p> <hr /> <h3 id="tell-your-application-your-credential-secret">Tell your application your credential secret</h3> <p>Now that your application has been created on the IBM Cloud, one final step is to tell your application the key it should use to decrypt your credentials file.</p> <p>Go to the <a href="">IBM Cloud dashboard page</a> for your newly deployed application. On the ‘runtime’ page, go to the ‘environment variable’ section and add a variable called <code class="language-plaintext highlighter-rouge">NODE_RED_CREDENTIAL_SECRET</code> set to whatever credential key you set when you created your Node-RED project right at the start of this whole exercise.</p> <p><img src="/blog/content/2018/06/Application_Details_-_IBM_Cloud-1024x672.png" alt="" /></p> <p>Click the ‘save’ button and your application will be restarted, now with this variable set.</p> <hr /> <h3 id="next-steps">Next steps</h3> <p>If you’ve got this far, well done. You now have a pipeline going from Node-RED on your local machine to Node-RED running in the IBM Cloud. Whenever you make changes locally, commit and push them to GitHub, your application on IBM Cloud will be restaged thanks to Travis.</p> <p>In the next part of this series, we’ll look at how to start building a simple application using this setup.</p> Getting Node-RED to One Million downloads 2018-03-16T00:00:00+00:00 <p>In our recent project <a href="">update blog post</a>, I marked the fact Node-RED has recently hit 1 million downloads. It’s a big milestone to reach for the project and a good opportunity to reflect back on how we’ve got here.</p> <p>There are a couple of threads I want to explore.</p> <ul> <li>Open by default</li> <li>Low Code application development</li> </ul> <h3 id="open-by-default">Open by default</h3> <p>When we started Node-RED, it was a tool to help us do our day job, but our day job wasn’t to spend time writing development tools - it was to build real solutions for clients as part of our Emerging Technologies group. That had some consequences on how we approached developing Node-RED. It meant that we weren’t adding features speculatively - everything we did was in reaction to a real need to do something. We were also limited in how much time we could spend on it during the day - evenings and weekends were much more the norm.</p> <p>As we discussed how to push it to a wider audience we knew that open sourcing it was the only route we wanted to take. The alternative was for it to remain a proprietary piece of code that would likely sit on a shelf and only get used by us - which wasn’t really an option.</p> <p>I <a href="">spoke at MonkiGras a couple years ago</a> about our experiences going through the open-sourcing process at IBM. It was a straight-forward process and very much reflected a growing attitude of being open by default - something that has continued to flourish and become an important part of our culture.</p> <p>Being an open source project has been absolutely instrumental to the success we’ve seen, but it was never a guarantee of success. A ‘build it and they will come’ approach may have worked in Field of Dreams, but with any brand new open source project, it takes hard work to spread the word and get people engaged with it.</p> <p>It also takes deliberate attention to develop in the open. Discussions that would previously have happened over a coffee in front of a whiteboard need to happen in an open forum. The long list of ideas written on sticky notes need to be visible to the community. Communication is key and I think this is an area we can continue to improve on.</p> <h3 id="low-code-application-development">Low Code application development</h3> <p>Node-RED embodies a Low Code style of application development; where developers can quickly create meaningful applications without having to write reams of code. The term Low Code was coined by the Forrester Research in a <a href="">report published in 2014</a> - but it clearly embodies a style of development that goes back further back then that.</p> <p>There are a number of benefits to Low Code application development, all of which we’ve seen first hand with Node-RED.</p> <p><strong>It reduces the time taken to create a working application</strong></p> <p>This allows the real value to be realised much quicker than with traditional development models.</p> <p><strong>It is accessible to a wide range of developers and non-developers</strong></p> <p>Above all else, this is one of the most important benefits we’ve seen. Anyone who understands a domain-specific problem, such as a business analyst, a linguist or a building engineer, will know the discrete steps needed to solve it. Node-RED gives them the tools to express those steps within a flow and build the solution for themselves.</p> <p><strong>The visual nature helps users to see their application</strong></p> <p>“<em>Show, don’t tell</em>” is a powerful concept. We often see Node-RED get used to demo capabilities of APIs, such as the Watson cognitive services. It’s so effective because the visualisation of your application logic shows the art of the possible without having to explain every semi-colon, bracket and brace. Not everyone thinks in lines of code; the visual representation of application logic is much more relatable.</p> <p>This is all evident in the way Node-RED is often used as part of the <a href="">code patterns</a> my colleagues produce on <a href="">IBM Code</a>.</p> <h3 id="low-code-but-i-want-to-write-code">Low Code? But I <em>want</em> to write code</h3> <p>Low Code platforms may open up application development to a wider audience of developers, but they still have their critics in those who prefer to be able to tinker with the underlying code.</p> <p>This is where the open by default approach of Node-RED brings us an advantage. Node-RED isn’t a closed platform that acts entirely as a black-box. Anyone is able to look under the covers and see what’s going on, to provide feedback or to suggest changes.</p> <p>If someone finds a node that doesn’t do quite what they need, they can easily work with the author to add the desired features, or choose to create their own node.</p> <p>I think one of the most important things we did in the project at the very start was to make it possible for anyone to self-publish their own nodes. We chose not to become gate-keepers to what the community could add to Node-RED. The fact there are over 1300 3rd party nodes today, a number that climbs steadily, is testament to that decision.</p> <h3 id="getting-to-the-next-million">Getting to the next million</h3> <p>As we look to the future of the project, there is a lot still to come. The <a href="">roadmap to 1.0</a> opens up so much potential for new uses of Node-RED that we’re all eager to get there as soon as we can. We continue to see more companies adopt Node-RED as part of their own developer experience. As the user community continue to grow, our real challenge is to grow the contributor community - helping drive the project forward.</p> <p>I’ve talked about Low Code in this post and not mentioned IoT once. We are still, and will remain, IoT at heart. But there’s a much broader framing of the project to consider beyond IoT. I have more personal flows doing simple web automation tasks than I do anything related to IoT. There’s a huge developer audience to tap into who may otherwise be put off by our focus on IoT. How that framing takes shape is something we need to think carefully about.</p> <p>Ultimately, Node-RED is an open source, low-code, event-driven, flow-based programming environment. It has a great community behind it and new users coming to it every day. I’m sure we’ll get to the next million much quicker than the first.</p> Week 10-30 2018-01-05T00:00:00+00:00 <blockquote> <p>hopefully it won’t be another 6 weeks before I write again</p> </blockquote> <p>Nope. That didn’t really work out did it. Let’s not kid ourselves, I’m not about to recap the last 20 weeks. For all sorts of reasons, it’s taken all my energy to keep the various plates spinning with little room for much else.</p> <p>A lot of my time has been spent heads down trying to make progress on the Node-RED roadmap we published in the summer. The content of the roadmap is good, but the timescales I set out for it were not. I knew they were ambitious and set myself and the community a challenge - but ultimately the weight of delivering the roadmap has fallen on my shoulders. And I’ve been feeling that weight heavily for a while.</p> <p>One of my goals for last year was to improve the sustainability of the project; to get to the point where I could go do something else and the project would continue to progress. The last few months have shown that, despite some positive signs, it’s not there yet. There are times when I feel frustrated at having to churn out code as it leaves very little time for doing all the other things I could/should be doing. But I also recognise that getting the roadmap done will unlock so much potential in the project.</p> <p>And this is one of my problems; having invested so much time, energy and emotion in the project over the last five years, when I talk to others about my goals and aspirations, I usually end up talking about them in relation to the project. I think I’m okay with that in the short term, but I need to remind myself that I’m not the project and the project isn’t me. I need to remember its okay to be selfish and that I’m allowed to have goals that don’t relate to Node-RED.</p> <p>So here we are with 2018.</p> <p>With all that said, my focus remains getting Node-RED to be a sustainable project and getting the roadmap delivered. Beyond that… time will tell.</p> Building Bleep, a TJ Bot 2017-08-11T00:00:00+00:00 <p>TJBot is an <a href="">open source DIY kit</a> from IBM Watson to create a Raspberry Pi powered robot backed by the IBM Watson cognitive services. First published late last year, the project provides all the design files needed to create the body of the robot as either laser-cut cardboard or 3D printed parts.</p> <p>It includes space for an RGB LED on top of its head, an arm driven by a servo that can wave, a microphone for capturing voice commands, a speaker so it can talk back and a camera to capture images.</p> <p><img src="/blog/content/2017/08/tjbot-1024x905.jpg" alt="" /></p> <p>There are an ever-growing set of tutorials and how-tos for building the bot and getting it hooked up to the IBM Watson services. All in all, its a lot of fun to put together. Having played with one at Node Summit a couple weeks ago, I finally decided to get one of my own. So this week I’ve mostly been monopolising the 3D printer in the Emerging Technologies lab.</p> <blockquote> <div style="padding:8px;"> <div style=" background:#F8F8F8; line-height:0; margin-top:40px; padding:50.0% 0; text-align:center; width:100%;"> <div style=" background:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAsCAMAAAApWqozAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAMUExURczMzPf399fX1+bm5mzY9AMAAADiSURBVDjLvZXbEsMgCES5/P8/t9FuRVCRmU73JWlzosgSIIZURCjo/ad+EQJJB4Hv8BFt+IDpQoCx1wjOSBFhh2XssxEIYn3ulI/6MNReE07UIWJEv8UEOWDS88LY97kqyTliJKKtuYBbruAyVh5wOHiXmpi5we58Ek028czwyuQdLKPG1Bkb4NnM+VeAnfHqn1k4+GPT6uGQcvu2h2OVuIf/gWUFyy8OWEpdyZSa3aVCqpVoVvzZZ2VTnn2wU8qzVjDDetO90GSy9mVLqtgYSy231MxrY6I2gGqjrTY0L8fxCxfCBbhWrsYYAAAAAElFTkSuQmCC); display:block; height:44px; margin:0 auto -44px; position:relative; top:-22px; width:44px;"></div></div> [A post shared by Nick O&#39;Leary (@knolleary)]( on <time style=" font-family:Arial,sans-serif; font-size:14px; line-height:17px;" datetime="2017-08-08T14:58:44+00:00">Aug 8, 2017 at 7:58am PDT</time> </div> <script async="" defer="" src="//"></script> </blockquote> <blockquote> <div style="padding:8px;"> <div style=" background:#F8F8F8; line-height:0; margin-top:40px; padding:50.0% 0; text-align:center; width:100%;"> <div style=" background:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAACwAAAAsCAMAAAApWqozAAAABGdBTUEAALGPC/xhBQAAAAFzUkdCAK7OHOkAAAAMUExURczMzPf399fX1+bm5mzY9AMAAADiSURBVDjLvZXbEsMgCES5/P8/t9FuRVCRmU73JWlzosgSIIZURCjo/ad+EQJJB4Hv8BFt+IDpQoCx1wjOSBFhh2XssxEIYn3ulI/6MNReE07UIWJEv8UEOWDS88LY97kqyTliJKKtuYBbruAyVh5wOHiXmpi5we58Ek028czwyuQdLKPG1Bkb4NnM+VeAnfHqn1k4+GPT6uGQcvu2h2OVuIf/gWUFyy8OWEpdyZSa3aVCqpVoVvzZZ2VTnn2wU8qzVjDDetO90GSy9mVLqtgYSy231MxrY6I2gGqjrTY0L8fxCxfCBbhWrsYYAAAAAElFTkSuQmCC); display:block; height:44px; margin:0 auto -44px; position:relative; top:-22px; width:44px;"></div></div> [A post shared by Nick O&#39;Leary (@knolleary)]( on <time style=" font-family:Arial,sans-serif; font-size:14px; line-height:17px;" datetime="2017-08-08T16:33:54+00:00">Aug 8, 2017 at 9:33am PDT</time> </div> <script async="" defer="" src="//"></script> </blockquote> <p>After about 20 hours of printing, my TJBot was coming together nicely, but I knew I wanted to customise mine a bit. In part this was for me to figure out the whole process of designing something in Blender and getting it printed, but also because I wanted my TJBot to be a bit different.</p> <p>First up was a second arm to given him a more balanced appearance. In order to keep it simple, I decided to make it a static arm rather than try to fit in a second servo. I took the design of the existing arm, mirrored it, removed the holes for the servo and extended it to clip onto the TJBot jaw at a nice angle.</p> <p><img src="/blog/content/2017/08/arm_blend-1024x747.png" alt="" /></p> <p>The second customisation was a pair of glasses because, well, why not?</p> <p><img src="/blog/content/2017/08/Glasses_blend-1024x747.png" alt="" /> <img src="/blog/content/2017/08/Ear_blend-1024x747.png" alt="" /></p> <p>I designed them with a peg on their bridge which would push into a hole drilled into TJBot’s head. I also created a pair of ‘ears’ to push out from the inside of the head for the arms of the glasses to hook onto. I decided to do this rather than remodel the TJBot head piece because it was quicker to carefully drill three 6mm holes in the head than it was to wait another 12 hours for a new head to print.</p> <p>In fact, as I write this, I’ve only drilled the hole between the eyes as there appears to be enough friction for the glasses to hold with just that one fixing point.</p> <p><img src="/blog/content/2017/08/bleep-1024x768.jpg" alt="" /></p> <p>There some other customisations I’d like to play with in the future; this was enough for me to remember how to drive Blender without crying too much.</p> <p>Why call him ‘Bleep’? Because that’s what happens when you let your 3 and 7 year old kids come up with a name.</p> <p>I’ve created a <a href="">repository on GitHub</a> with the designs of the custom parts and where I’ll share any useful code and how-tos I come up with.</p> Week 4-9 2017-08-10T00:00:00+00:00 <p>Rather predictably, I’m already 6 weeks behind on my weekly notes of what I’m doing. So here’s a bumper catch up.</p> <p><strong>Weeks 4 + 5</strong> were focussed on three things; sorting out some fixes for issues that slipped into 0.17, writing up the Node-RED roadmap to take us to version 1.0 and preparing for upcoming events.</p> <p>The end of week 5 was the <a href="">FullStack</a> conference itself. I could only make it on the last day, when I was scheduled to speak. When I arrived, all the sessions of the day were written up on a whiteboard and everyone could vote on which talk they planned to see. The organisers then used that to assign the sessions to the different sized rooms. Slightly unfortunately the title of my talk had been lost somewhere along the way, so was simply listed as “Node-RED”. I got a roomful of 30 or so people; many of who had heard of Node-RED already. With the talk done, I couldn’t hang around for the other sessions as I had to get home ahead of a weekend away.</p> <p><strong>Week 6</strong> started with us publishing the <a href="">Node-RED roadmap</a> to version 1.0. It was great to get that shared and start to get feedback from the community on it. Its an ambitious roadmap - I said in the blog post the goal is to get there by the end of the year. I suspect it will by a bit later than that, but it sets out the tone and direction of the project nicely.</p> <p>One of the more interesting things I did this week was to spend a bit of time talking with a startup who were having some issues with Node-RED. They were using it to create bots for social-media engagement and had a high profile event coming up that they expected to drive particular high workloads through the system. I reviewed their flows and provided some feedback on areas that were potential bottlenecks. A few days later I heard back that they had implemented some of the suggestions and were happy it was handling the workload they needed it to.</p> <p><strong>Week 7</strong> was mostly spent in San Francisco for Node Summit. On Monday I took the opportunity to have a catch up over breakfast with Matt Biddulph, followed by an impromptu Node-RED talk as his new company. I then spent the rest of the day at the Galvanize office where IBM were hosting an IoT workshop in the afternoon, followed by a meetup in the evening at which I did my Node-RED talk.</p> <p>Tuesday was the Day Zero part of NodeSummit where I did my full Node-RED talk to a packed room and got to see lots of other great talks. If anything, there was too much good stuff - the afternoon session had 6 back-to-back 30 minute talks with no break. For a jet lagged Brit who hadn’t had a decent cup of tea for a few days, I definitely faded out.</p> <p>The week ended somewhat abruptly the next day when I found myself having to rearrange flights to get home two days early. I have to say a big thanks to American Express Global Business Travel who showed a great deal of empathy and great customer service when they agreed to refund the $900 fee Delta Airlines charged me without any empathy or good customer service to change the ticket.</p> <p><strong>Week 8</strong> was a bit all over the place.</p> <p><strong>Week 9</strong> was back to normal somewhat, but most notably came the announcement that Hitachi are adopting Node-RED as the tooling for their Lumada IoT platform. We’ve been working with Hitachi for the last few months, discussing their requirements and how Node-RED can fit with what they are doing. The roadmap we published a couple weeks ago has certainly been influenced by their involvement and I look forward to their contribution growing.</p> <p>I also kept the Emerging Technologies’ 3D printer busy building a <a href="">TJ Bot</a> - more on that soon.</p> <p>And that brings us to today - Toby’s 7th birthday. We’ve just watched Star Wars (A New Hope) for the first time - although he knows it backwards thanks to the various Lego versions he’s already seen. And I’ve taken the opportunity to get these notes written up.</p> <p>Next week is a holiday, so there won’t be any updates, but hopefully it won’t be another 6 weeks before I write again.</p> Week 1-3 2017-06-30T00:00:00+00:00 <p>I’ve liked the idea of writing week notes for ages, but I’ve never managed to get started. When you’re busy with work, without taking the time to reflect on what you’ve actually done, you can look up and realise half the year has already passed. Hi July!</p> <p>In the past, there has always been too much I can’t talk about publicly which has made it hard to write anything meaningful. But now I have a new job, a new purpose and a new found energy to write here more.</p> <p>Between my Developer Advocacy role and running the Node-RED project there will be plenty I can write about. There will undoubtedly be things I can’t talk about directly, but that shouldn’t get in the way.</p> <p>The main thing I need to remember is who I’m writing these week notes for. I’m sorry, dear reader, but it isn’t you. Whilst I hope you find them interesting, if you chose to read them, they are primarily here for me to record just what it is I spend my time doing. Sometimes there will be a story to tell, sometimes there will be a collection of bullet points. The main thing is to get that rhythm going.</p> <p>Given the slightly blurred lines of when I formally moved from ETS to the Digital Business Group, I’m going to call Monday June 12th as the start of Week 1. Which does mean I’m already 3 weeks behind on my notes. So lets catch things up.</p> <p><strong>Week 1</strong> was spent juggling three things; trying to get Node-RED 0.17 finished and released, preparing and delivering a talk on Node-RED Dashboard and preparing three separate workshops on Node-RED and IBM Watson IoT to be delivered the following week.</p> <p>We try to do regular releases with Node-RED, but inevitably, things get in the way and more time passes than we’d like between them. 0.17 was overdue but there were a small number of outstanding issues that needed squashing. I wanted to get them done and released before the workshops the following week as I knew they’d benefit from it. But in the end, it had to give way.</p> <p>The Node-RED Dashboard talk was an internal call with a community of technical sales folk to introduce them to Dashboard and show them it can do. Whilst we have a standard set of slides for the main Node-RED project, we didn’t have an equivalent set for Dashboard. So I spent a day pulling those together. (Note to self: we really should make our standard intro slides available publicly somewhere)</p> <p>The workshops were part of large education event for the senior technical architect community across the whole of Europe. I was running the IoT stream and had three workshops to prepare to take the participants through the whole process of using Node-RED on a device, getting it connected via Watson IoT into Bluemix and then controlling it using the Watson Conversation service. The material I produced was a set of high-level exercises that addressed each step in the process. I have to admit I wasn’t sure on how high-level or how detailed I had to make the material, but decided I’d be able to improvise once I got a better sense of the participants at the event. Which leads me neatly on to…</p> <p><strong>Week 2</strong> was spent in Lisbon, Portugal, at the Future Skills event to deliver the workshops. 250 Technical Architects from across Europe converged on this event to participate in one of four streams; Cognitive, Blockchain, Data Science and IoT. In all, we had about 60 colleagues in the IoT stream. Across the first day and a half, we delivered the workshops and they then had to take what they had learnt and produce a demo/project/poc that was relevant to their clients. With all the usual challenges of unusable wifi at big events it was great to see a genuine variety of projects at the end.</p> <p>My main takeaway was the workshop material I had produced was too ambitious for the time available. It also left too much for the participants to figure out for themselves. A good start for the material, but needs more work to be properly reusable.</p> <p><strong>Week 3</strong> was back in Hursley focusing on 0.17. The remaining issues had been mostly resolved and the last task was to get the release notes written and some final documentation updates done. For a release that had over 300 commits, that took some time. Some releases have obvious headline features - this one was much more a sum-of-its-parts type release. That made it a challenge to decide what things to pull out into the release blog post. But I got there eventually and 0.17 was released today.</p> <p>Being back in Hursley also meant catching up on things that I’d had to put to one side whilst in Lisbon. Conversations around our plans for developer advocacy activities, discussions with other teams to ensure we’re working in the same direction and the usual time spent help out colleagues with their random questions about Node-RED that pepper my time.</p> <p>With the move to the DBG group, I knew I’d still spend a lot of my time on Node-RED, but it was the pieces around the edge that were going to change. Based on the conversations this week, there’s lots of interesting things to get done.</p> <p>Finally, I got confirmation that I’m speaking at Node Summit in San Francisco at the end of July and also just managed to submit both a talk and workshop proposal for NodeConfEU in November before the deadline today.</p> <p>Next week is more planning; with Node-RED 0.17 I’m working on the plan for the next release and beyond. I also need to start preparing my talk for FullStack conference two weeks from today. And it looks like we have a couple issues to tidy up and get a 0.17.1 released sooner rather than later.</p> <p>That’s it. First set of week notes done, spanning three weeks and almost hitting 1000 words. Expect a few scrawled bullet points next time.</p> Time for a change 2017-06-16T00:00:00+00:00 <p>Last September, as I marked my 15 years with IBM, I ended a post wondering what the next 5 years would hold in store. Perhaps a bit sooner than I had planned, it’s time to find out.</p> <p>I’ve been in IBM’s Emerging Technologies Services group for close to 6 years now. In that time I’ve worked on a host of projects, including:</p> <ul> <li>writing embedded MQTT code for smart energy meters</li> <li>connecting a Scalextric to a mind-reading headset</li> <li>6 weeks spent in a 7th floor conference room writing custom Dojo UI widgets (not a highlight, but you’ve got to pay the bills)</li> <li>wiring up some Arduinos and Pis in an Ice Cream factory (a highlight)</li> </ul> <p>And of course, the main thing I’ve focused on for the last 3 years, Node-RED.</p> <p>The client-funded nature of the work ETS does has meant I’ve only been able to work on Node-RED because someone has been willing to pay me to do so. I’ve been incredibly fortunate to have been in that position; being able to turn a side-project into a job.</p> <p>I’ve enjoyed my time in ETS, but the time has come to make a change.</p> <p>I have now moved over to IBM Digital Business Group to be a Developer Advocate, focussing on IoT. This is a new part of IBM where all of our work around developer advocacy is being brought together in one place.</p> <p>Making our products, APIs and services attractive to developers is vital to the future of the business. Long gone are the days that software is sold to the CIO. Developers are far more empowered than they ever were to make their own technology choices. Go read <a href="">Stephen O’Grady’s The New Kingmakers</a> for a far more insightful take on how this has been the trend for quite some time now.</p> <p>In my new role I’ll be working to help developers engage with our technology. Whether that’s through hackathons, conferences, blog posts, stack overflow questions, videos, Coursera courses - really anything and anywhere that developers are.</p> <p>So what does this mean for my time on Node-RED?</p> <p>Well, truth be told, the group I now work for is the group who was previously funding my time on Node-RED. So whilst I have changed departments, the primary focus of my work stays exactly what it has been: leading the Node-RED project and working hard to grow the community and eco-system around it.</p> <p>Everything changes. Nothing changes. On to the next 5 years.</p> 15 Years an IBMer 2016-09-04T00:00:00+00:00 <p>In a previous post, I <a href="/2011/09/03/10-years-an-ibmer/">wrote</a> about how I ended up at IBM. I finished that post with a question: Will I still be here in another 10 years? Well, I’m half-way to finding out, but I’m no closer to an answer.</p> <p>The last 5 years in the Emerging Technologies group have flown by. The last three years of which have been pretty much dominated by my work on <a href="">Node-RED</a>. It’s been a lot of hard work, but it has been incredibly satisfying work. I’m very fortunate to be in this position; not everyone gets to turn a side-project they created into a job.</p> <p>That said, as I spoke about in my <a href="">Monkigras talk</a> this year, when a side-project becomes a job, it risks becoming a chore. I have to continually make sure I strike a balance between the fun and the not-so-fun parts. I don’t always get that balance right which can make it harder work than it should be.</p> <p>So what of the next 5 years? No doubt Node-RED will continue to be my focus for now, but that doesn’t stop me wondering what else I could be doing.</p> A Conversational Internet of Things - ThingMonk talk 2014-12-04T00:00:00+00:00 <p><em>Earlier this year, Tom Coates wrote a <a href="">blog post</a> about his session at this year’s O’Reilly Foo Camp. Over tea with colleagues, we talked about some of the ideas from the post and how some of our research work might be interesting when applied to them.</em></p> <p><em>One thing led to another and I found myself talking about it at ThingMonk this year. What follows is a slightly expanded version of my talk.</em></p> <hr /> <p><img src="/blog/content/2014/12/ciot-0-300x168.png" alt="ciot-0" /></p> <p><img src="/blog/content/2014/12/ciot-2-300x168.png" alt="ciot-2" /></p> <p><strong>Humanising Things</strong></p> <p>We have a traditional of putting human faces on things. Whether it’s literally seeing faces on the Things in our everyday lives, such as the <a href="">drunk octopus</a> spoiling for a fight, or possibly the most <a href="">scary drain pipe</a> ever.</p> <p>Equally, we have a tendency to put a human persona onto things. The advent of Twitter brought an onslaught of Things coming online. It seemingly isn’t possible for me to do a talk without at least a fleeting mention of Andy Standford-Clark’s twittering ferries; where regular updates are provided for where each ferry is.</p> <p><img src="/blog/content/2014/12/ciot-5-300x168.png" alt="ciot-5" /></p> <p>One of the earliest Things on Twitter was Tower Bridge. Tom Armitage, who was working near to the bridge at the time, wrote some code that grabbed the schedule for the bridge opening and closing times, and created the account to relay that information.</p> <p><img src="/blog/content/2014/12/ciot-6-300x168.png" alt="ciot-6" /></p> <p>One key difference between the ferries and the bridge is that the ferries are just relaying information, a timestamp and a position, whereas the bridge is speaking to us in the first-person. This small difference immediately begins to bring a more human side to the account. But ultimately, they are simple accounts that relay their state with whomever is following them.</p> <p>This sort of thing seems to have caught on particularly with the various space agencies. We no longer appear able to send a robot to Mars, or land a probe on a comet without an accompanying twitter account bringing character to the events.</p> <p><img src="/blog/content/2014/12/ciot-7-300x168.png" alt="ciot-7" /></p> <p>There’s always a sense of excitement when these inanimate objects start to have a conversation with one another. The conversations between the philae lander and its orbiter were particularly touching as they waved goodbye to one another. Imagine, the lander, which was launched into space years before Twitter existed, chose to use its last few milliamps of power to send a final goodbye.</p> <p><img src="/blog/content/2014/12/ciot-8-300x168.png" alt="ciot-8" /></p> <p>But of course as soon as you peek behind the curtain, you see someone running Tweetdeck, logged in and typing away. I was watching the live stream as the ESA team were nervously awaiting to hear from philae. And I noticed the guy in the foreground, not focused on the instrumentation as his colleagues were, but rather concentrating on his phone. Was he the main behind the curtain, preparing Philae’s first tweet from the surface? Probably not, but for the purposes of this talk, let’s pretend he was.</p> <p><img src="/blog/content/2014/12/ciot-9-300x168.png" alt="ciot-9" /></p> <p>The idea of giving Things a human personality isn’t a new idea. There is a wealth of rigorous scientific research in this area.</p> <p>One esteemed academic, Douglas Adams, tells us about the work done by the The Sirius Cybernetics Corporation, who invented a concept called Genuine People Personalities (“GPP”) which imbue their products with intelligence and emotion.</p> <p>He writes:</p> <blockquote> <p>Thus not only do doors open and close, but they thank their users for using them, or sigh with the satisfaction of a job well done. Other examples of Sirius Cybernetics Corporation’s record with sentient technology include an armada of neurotic elevators, hyperactive ships’ computers and perhaps most famously of all, Marvin the Paranoid Android. Marvin is a prototype for the GPP feature, and his depression and “terrible pain in all the diodes down his left side” are due to unresolved flaws in his programming.</p> </blockquote> <p>In a related field, we have the <a href="">Talkie Toaster</a> created by Crapola, Inc and seen aboard Red Dwarf. The novelty kitchen appliance was, on top of being defective, only designed to provide light conversation at breakfast time, and as such it was totally single-minded and tried to steer every conversation to the subject of toast.</p> <p><img src="/blog/content/2014/12/ciot-13-300x168.png" alt="ciot-13" /></p> <p><strong>Seam[less’ful]ness</strong></p> <p>In this era of the Internet of Things, we talk about a future where our homes and workplaces are full of connected devices, sharing their data, making decisions, collaborating to make our lives ‘better’.</p> <p>Whilst there are people who celebrate this invisible ubiquity and utility of computing, the reality is going to much more messy.</p> <p><a href="">Mark Weiser</a>, Chief Scientist at Xerox PARC, coined the term “ubiquitous computing” in 1988.</p> <blockquote> <p>Ubiquitous computing names the third wave in computing, just now beginning. First were mainframes, each shared by lots of people. Now we are in the personal computing era, person and machine staring uneasily at each other across the desktop. Next comes ubiquitous computing, or the age of calm technology, when technology recedes into the background of our lives.</p> </blockquote> <p>Discussion of Ubiquitous Computing often celebrated the idea of seamless experiences between the various devices occupying our lives. But in reality, Mark Weiser advocated for the opposite; that seamlessness was undesirable and self-defeating attribute of such a system.</p> <p>He preferred a vision of “Seamfulness, with beautiful seams”</p> <p><img src="/blog/content/2014/12/ciot-15-300x168.png" alt="ciot-15" /></p> <p>The desire to present a single view of the system, with no joins, is an unrealistic aspiration in the face of the cold realities of wifi connectivity, battery life, system reliability and whether the Cloud is currently turned on.</p> <p>Presenting a user with a completely monolithic system gives them no opportunity to connect with and begin to understand the constituent parts. That is not it say this information is needed to all users all of the time. But there is clearly utility to some users some of the time.</p> <p>When you come home from work and the house is cold, what went wrong? Did the thermostat in the living room break and decide it was the right temperature already? Did the message from the working thermostat fail to get to the boiler? Is the boiler broken? Did you forgot to cancel the entry in your calendar saying you’d be late home that day?</p> <p>Without some appreciation of the moving parts in a system, how can a user feel any ownership or empowerment when something goes wrong with it. Or worse yet, how can they avoid feeling anything other than intimidated by this monolithic system that simply says “I’m Sorry Dave, I’m afraid I can’t do that”.</p> <p>Tom Armitage <a href="">wrote up his</a> talk from Web Directions South and published it earlier this week, just as I was writing this talk. He covers a lot of what I’m talking about here so much more eloquently than I am - go read it. One piece his post pointed me at that I hadn’t seen was <a href="">Techcrunch’s recent review of August’s Smart Lock</a>.</p> <p><img src="/blog/content/2014/12/ciot-16-300x168.png" alt="ciot-16" /></p> <p>Tom picked out some choice quotes from the review which I’ll share here:</p> <blockquote> <p>“…much of the utility of the lock was negated by the fact that I have roommates and not all of them were willing or able to download the app to test it out with me […] My dream of using Auto-Unlock was stymied basically because my roommates are luddites.”</p> </blockquote> <blockquote> <p>“Every now and then it didn’t recognize my phone as I approached the door.”</p> </blockquote> <blockquote> <p>“There was also one late night when a stranger opened the door and walked into the house when August should have auto-locked the door.”</p> </blockquote> <p>This is the reason for having beautiful seams; seams help you understand the edges of a devices sphere of interaction, but should not be so big to trip you up. Many similar issues exists with IP connected light bulbs. When I need to remember which app to launch on my phone depending on which room I’m walking into, and which bulbs happen to be in there, the seams have gotten too big.</p> <p>In a recent <a href="">blog post</a>, Tom Coates wrote about the idea of a chatroom for the house - go read it.</p> <blockquote> <p>Much like a conference might have a chatroom, so might a home. And it might be a space that you could duck into as you pleased to see what was going on. By turning the responses into human language you could make the actions of the objects less inscrutable and difficult to understand.</p> </blockquote> <p><img src="/blog/content/2014/12/ciot-17-300x168.png" alt="ciot-17" /></p> <p>This echoes back to the world of Twitter accounts for Things. But rather than them being one-sided conversations presenting raw data in a more consumable form, or Wizard-of-Oz style man-behind-the-curtain accounts, a chatroom is a space where the conversation can flow both ways; both between the owner and their devices, but also between the devices themselves.</p> <p>What might it take to turn such a chatroom into a reality?</p> <p><img src="/blog/content/2014/12/ciot-18-300x168.png" alt="ciot-18" /></p> <p><strong>Getting Things Talking</strong></p> <p>Getting Things connected is no easy task.</p> <p>We’re still in the early days of the protocol wars.</p> <p>Whilst I have to declare allegiance to the now international OASIS standard MQTT, I’m certainly not someone who thinks one protocol will rule them all. It pains me whenever I see people make those sorts of claims. But that’s a talk for a different day.</p> <p>Whatever your protocol of choice, there are an emerging core set that seem to be the more commonly talked about. Each with its strengths and weaknesses. Each with its backers and detractors.</p> <p><img src="/blog/content/2014/12/ciot-19-300x168.png" alt="ciot-19" /></p> <p>What (mostly) everyone agrees on is the need for more than just efficient protocols for the Things to communicate by. A protocol is like a telephone line. It’s great that you and I have agreed on the same standards so when I dial this number, you answer. But what do we say to each other once we’re connected? A common protocol does not mean I understand what you’re trying to say to me.</p> <p>And thus began the IoT meta-model war.</p> <p>There certainly a lot of interesting work being done in this area.</p> <p>For example, HyperCat, a consortium of companies coming out of a Technology Strategy Board funded Demonstrator project in the last year or so.</p> <p><img src="/blog/content/2014/12/ciot-21-300x168.png" alt="ciot-21" /></p> <blockquote> <p>HyperCat is an open, lightweight JSON-based hypermedia catalogue format for exposing collections of URIs. Each HyperCat catalogue may expose any number of URIs, each with any number of RDF-like triple statements about it. HyperCat is simple to work with and allows developers to publish linked-data descriptions of resources.</p> </blockquote> <p>URIs are great. The web is made of them and they are well understood. At least, they are well understood by machines. What we’re lacking is the human view of this world. How can this well-formed, neatly indented JSON be meaningful or helpful to the user who is trying to understand what is happening.</p> <p>This is by no means a criticism of HyperCat, or any of the other efforts to create models of the IoT. They are simply trying to solve a different set of problems to the ones I’m talking about today.</p> <p><img src="/blog/content/2014/12/ciot-23-300x168.png" alt="ciot-23" /></p> <p><strong>Talking to Computers</strong></p> <p>We live in an age where the talking to computers is becoming less the reserve of science fiction.</p> <p>Siri, OK Google, Cortana all exist as ways to interact with the devices in your pocket. My four year old son walks up to me when I have my phone out and says: “OK Google, show me a picture of the Octonauts” and takes over my phone without even having to touch it. To him, as to me, voice control is still a novelty. But I wonder what his 6 month old sister will find to be the intuitive way of interacting with devices in a few years time.</p> <p>The challenge of Natural Language Parsing, NLP, is one of the big challenges in Computer Science. Correctly identifying the words being spoken is relatively well solved. But understanding what those words mean, what intent they try to convey, is still a hard thing to do.</p> <p>To answer the question “Which bat is your favourite?” without any context is hard to do. Are we talking to a sportsman with their proud collection of cricket bats? Is it the zoo keeper with their colony of winged animals. Or perhaps a comic book fan being asked to chose between George Clooney and Val Kilmer.</p> <p>Context is also key when you want to hold a conversation. The English language is riddled with ambiguity. Our brains are constantly filling in gaps, making theories and assertions over what the other person is saying. The spoken word also presents its own challenges over the written word.</p> <p><img src="/blog/content/2014/12/ciot-28-300x168.png" alt="ciot-28" /></p> <p>“Hu was the premiere of China until 2012”</p> <p>When said aloud, you don’t know if I’ve asked you a question or stated a fact. When written down, it is much clearer.</p> <p><img src="/blog/content/2014/12/ciot-29-300x168.png" alt="ciot-29" /></p> <p>In their emerging technology report for 2014, Gartner put the Internet of Things at the peak of inflated expectation. But if you look closely at the curve, up at the peak, right next to IoT, is NLP Question Answering. If this was a different talk, I’d tell you all about how IBM Watson is solving those challenges. But this isn’t that talk.</p> <p><img src="/blog/content/2014/12/ciot-30-300x168.png" alt="ciot-30" /></p> <p><strong>A Conversational Internet of Things</strong></p> <p>To side step a lot of the challenges of NLP, one area of research we’re involved with is that of Controlled Natural Language and in particular, Controlled English.</p> <p>CE is designed to be readable by a native English speaker whilst representing information in a structured and unambiguous form. It is structured by following a simple but fully defined syntax, which may be parsed by a computer system.</p> <p>It is unambiguous by using only words that are defined as part of a conceptual model.</p> <p>CE serves as a language that is both understandable by human and computer system - which allows them to communicate.</p> <p>For example,</p> <pre>there is a thermometer named t1 that is located in the room r1</pre> <p>A simple sentence that establishes the fact that a thermometer exists in a given room.</p> <pre>the thermometer t1 can measure the environment variable temperature</pre> <p>Each agent in the system builds its own model of the world that can be used to define concepts such thermometer, temperature, room and so on. As the model is itself defined in CE, the agents build their models through conversing in CE.</p> <pre>there is a radiator valve v1 that is located in the room r1 the radiator valve v1 can control the environment variable temperature</pre> <p>It is also able to using reasoning to determine new facts.</p> <pre>the room r1 has the environment variable temperature that can be measured and that can be controlled</pre> <p>As part of some research work with Cardiff University, we’ve been looking at how CE can be extended to a conversational style of interaction.</p> <p>These range from exchanging facts between devices - the tell</p> <pre>the environment variable temperature in room r1 has value "21"</pre> <p>Being able to ask question - ask-tell</p> <pre>for which D1 is it true that ( the device D1 is located in room V1 ) and ( the device D1 can measure the environment variable temperature ) and ( the value V1 == "r1")</pre> <p>Expanding on and explaining why certain facts are believed to be true:</p> <pre>the room r1 has the environment variable temperature that can be measured and that can be controlled because the thermometer named t1 is located in the room r1 and can measure the environment variable temperature and the radiator valve v1 is located in the room r1 and can control the environment variable temperature</pre> <p>The fact that the devices communicate in CE means the user can passively observe the interactions. But whilst CE is human readable, it isn’t necessarily human writeable. So some of the research is also looking at how to bridge from NL to CE using a confirm interaction:</p> <pre>NL: The thermometer in the living room has moved to the dining room CE: the thermometer t1 is located in the room r2</pre> <p>Whilst the current research work is focused on scenarios for civic agencies - for example managing information exchange in a policing context, I’m interested in applying this work to the IoT domain.</p> <p>With these pieces, you can begin to see how you could have an interaction like this:</p> <pre> User: I will be late home tonight. House: the house will have a state of occupied at 1900 User: confirmed House: the room r1 has a temperature with minimum allowable value 20 after time 1900 the roomba, vc1, has a clean cycle scheduled for time 1800 </pre> <p>Of course this is still quite dry and formal. It would be much more human, more engaging, if the devices came with their own genuine people personality. Or at least, the appearance of one.</p> <pre> User: I will be late home tonight. House: Sorry to hear that, shall I tell everyone to expect you back by 7? User: yes please Thermometer: I'll make sure its warm when you get home Roomba: *grumble* </pre> <p>I always picture the Roomba as being a morose, reticent creature who really hates its own existence. We have one in the kitchen next to our lab at work, set to clean at 2am. If we leave the door to the lab open, it heads in and, without fail, maroons itself on a set of bar stools we have with a sloped base. Some might call that a fault in its programming, much like Marvin, but I like to think its just trying to find a way to end it all.</p> <p>This is all some way from having a fully interactive chat room for your devices. But the building blocks are there and I’ll be exploring them some more.</p> Went to Designcamp, got the T-Shirt 2014-11-12T00:00:00+00:00 <p>Last month, I was fortunate enough to fly off to Austin with a group of colleagues for a week long IBM Design Thinking camp. It was an opportunity to get away from the day job, with laptops all-but banned, and have a deep-dive into what IBM Design is about and how it can be applied.</p> <p>As a relatively new effort within the company, IBM Design sets out to bring a focus back to where it should be; the human-experience of our products and services. This isn’t just about making pretty user interfaces; it is the entire experience of our products.</p> <p>As an engineer, the temptation is always there to create shiny new features. But no matter how shiny it is, if it isn’t what a user needs, then it’s a waste of effort. The focus has to be on what the user wants to be able to do. This is something I’ve always tried to do with Node-RED; we often get suggestions for features that, once you start picking at them, are really solutions looking for a problem. Once you work back and identify the problem, we’re often able to identify alternative solutions that are even better.</p> <p><a href="" title="P1070048 by Nick O&#x27;Leary, on Flickr"><img src="" alt="P1070048" /></a></p> <p>It’s often just a matter of asking the right question; At Designcamp, the very first exercise we were asked to do was to draw a new type of vase. Everyone drew something that looked vaguely vase-like. Then (spoilers…) we were asked to draw a better way to display flowers. At this point we got lots of decidedly un-vase-like ideas that were much more imaginative. It’s the difference between asking for a feature and asking for an idea. The former presupposes a lot about the nature of the answer, the latter is focused on not just the <strong>what</strong>, but also the <strong>why</strong>.</p> <p>This relentless focus on the user isn’t a new idea. GDS, who are doing incredible things with government services, have it as their very first <a href="">Design Principle</a>. But it is refreshing to see this focus being brought to bear within a transformation of how the entire company operates.</p> <p>Oh, and of course being in Austin, we got to screen print our own IBM Designcamp T-Shirts to commemorate the visit.</p> <p><a href="" title="Go to Designcamp, screen print your own t-shirt. Obvs. by Nick O&#x27;Leary, on Flickr"><img src="" alt="Go to Designcamp, screen print your own t-shirt. Obvs." /></a></p> <p>Lots more photos from the week over on <a href="">flickr</a>.</p> Automated GitHub pull-request status updates with Node-RED 2014-04-19T00:00:00+00:00 <p>Having recently added a Travis build for the Node-RED repository, we get a nice green, or sometimes nasty red, merge button on pull-requests, along with a comment from Travis thanks to GitHub’s <a href="">Commit Status API</a>.</p> <p>From a process point of view, the other thing we require before merging a pull-request is to ensure the submitter has completed a Contributor License Agreement, or CLA. This has been a manual check against a list we maintain. But with GitHub’s recent addition of the <a href="">Combined Status api</a> I decided we ought to automate this check.</p> <p>And of course, why not implement it in a Node-RED flow.</p> <p>There are a few simple steps to take:</p> <ol> <li>know there has been a pull-request submitted, or that we want to manually trigger a check</li> <li>check the submitter against the list</li> <li>update the status accordingly</li> </ol> <p>Rather than poll the api to see when a new pull-request has arrived, GitHub allows you to register a Webhook to get an http POST when certain events occur. For this to work, we need an HTTP-In node to accept the request. As with any request, we need to make sure it is responded to properly, as well as keeping a log - handy for debugging as you go.</p> <p><img src="/blog/content/2014/04/nr-gh-flow-1.png" alt="nr-gh-flow-1" /></p> <p>With that in place, we can register the webhook on the repositories we want to run the checks on.</p> <ol> <li>From the repository page, find your way to the webhook page via Settings ➝ WebHooks &amp; Services ➝ Add webhook</li> <li>Enter the url of the http-in node</li> <li>Select ‘Let me select individual events.’ and then pick, as a minimum, ‘Pull Request’ and ‘Issue comment’</li> <li>Ensure the ‘Active’ option is ticked and add the webhook</li> </ol> <p>When the POST arrives from GitHub, the <code class="language-plaintext highlighter-rouge">X-GitHub-Event</code> http header identifies the type of event that triggered the hook. As we’ve got a couple different types registered for the hook we can use a Switch node to route the flow depending on what happened. We also need a function node to access the http header as the Switch node can’t do by itself.</p> <pre> // GitHub Event Type msg.eventType = msg.req.get('X-GitHub-Event'); return msg; </pre> <p><img src="/blog/content/2014/04/nr-gh-flow-2.png" alt="nr-gh-flow-2" /></p> <p>Skipping ahead a bit, to update the status of the commit requires an <a href="">http POST to the GitHub API</a>. This is easily achieved with an HTTP Request node, but it does need to be configured to point to the right pull-request. Pulling that information out of the ‘pull-request’ webhook event is easy as it’s all in there.</p> <pre> // PR Opened msg.login = msg.payload.pull_request.user.login; msg.commitSha = msg.payload.pull_request.head.sha; msg.repos = msg.payload.pull_request.base.repo.full_name; return msg; </pre> <p>It’s a bit more involved with the ‘issue-comment’ event which fires for all comments made in the repository. We only want to process comments against pull-requests that contain the magic text to trigger a CLA check to run.</p> <p>Even after identifying the comments we’re interested in, there’s more work to be done as they don’t contain the information needed to update the status. Before we can do that, we must first make another <a href="">http request to the GitHub API</a> to get the pull-request details.</p> <pre> // Filter PR Comments msg.repos = msg.payload.repository.full_name; if (msg.payload.issue.pull_request &amp;&amp; msg.payload.comment.body.match(/node-red-gitbot/i)) { msg.url = ""+msg.repos+ "/pulls/"+msg.payload.issue.number+ "?access_token=XXX"; msg.headers = { "user-agent": "node-red", "accept": "application/vnd.github.v3" }; return msg; } return null; </pre> <p>Taking the result of the request, we can extract the information needed.</p> <pre> // Extract PR details var payload = JSON.parse(msg.payload); msg.login = payload.user.login; msg.commitSha = payload.head.sha; msg.repos = payload.base.repo.full_name; return msg; </pre> <p><img src="/blog/content/2014/04/nr-gh-flow-4.png" alt="nr-gh-flow-4" /></p> <p>Now, regardless of which event has been triggered we have everything we need to check the CLA status; if <code class="language-plaintext highlighter-rouge">msg.login</code> is in the list of users who have completed a CLA, send a ‘success’ status, otherwise send a ‘failure’.</p> <pre> // Update CLA Status var approved = ['list','of','users']; var login = msg.login; var commitSha = msg.commitSha; var repos = msg.repos; msg.headers = { "user-agent": "node-red", "accept": "application/vnd.github.she-hulk-preview+json" }; msg.url = ""+repos+ "/statuses/"+commitSha+ "?access_token=XXX"; msg.payload = { state:"failure", description:"No CLA on record", context:"node-red-gitbot/cla", target_url:"" } if (approved.indexOf(login) &gt; -1) { msg.payload.state = "success"; msg.payload.description = "CLA check passed"; } return msg; </pre> <p><img src="/blog/content/2014/04/nr-gh-flow-3.png" alt="nr-gh-flow-3" /></p> <p>And that’s all it takes.</p> <p>One of the things I like about Node-RED is the way it forces you to break a problem down into discrete logical steps. It isn’t suitable for everything, but when you want to produce event-drive logic that interacts with on-line services it makes life easy.</p> <p>You want to be alerted that a pull-request has been raised by someone without CLA? Drag on a twitter node and you’re there.</p> <p>There are some improvements to be made.</p> <p>Currently the node has the list of users who have completed the CLA hard-coded in - this is okay for now, but will need to be pulled out into a separate lookup in the future.</p> <p>It also assumes the PR only contains commits by the user who raised it - if a team has collaborated on the PR, this won’t spot that. Something I’ll have to keep an eye on and refine the flow if it becomes a real problem.</p> <p>Some of the flow can already be simplified since we <a href="">added template support</a> for the HTTP Request node’s URL property.</p> <p>The final hiccup with this whole idea is the way GitHub currently present commit statuses on the pull-request page. They only show the most recent status update, not the combined status. If the CLA check updates first with a failure, then the Travis build passes a couple minutes later, the PR will go green thanks to the Travis success. The only way to spot the Combined Status is currently ‘failure’ is to dive into the API. I’ve written a little script to give me a summary of all PR’s against the repositories, which will do for now.</p> <p>Given the Combined Status API was only added as a preview last month, hopefully once it gets added to the stable API they will update how the status is presented to use it.</p> <p>A final note, I’ve chosen not to share the full flow json here - hopefully there’s enough above to let you recreate it, but if you really want it, <a href="">ping me</a>.</p> Changing Ghost permalinks without breaking everything 2014-01-26T00:00:00+00:00 <p>I’m a big fan of the <a href="">Ghost blogging platform</a>. Having heard <a href="">Hannah Wolfe</a> talk about it a couple of times, I was keen to try it out. In fact I purposefully held off setting up a blog for <a href="">Node-RED</a> until Ghost was available.</p> <p>It was painless to set up and get running on my Web Faction hosted server - even though at that point Web Faction hadn’t created their one-click installer for it.</p> <p>Ghost released their second stable version a couple weeks ago, which added, amongst many other new features, the ability for a post’s url to contain its date. This was a feature I hadn’t realised I was missing until I wrote a post on the blog called ‘Community News’. The intent was this would be a commonly used blog title as we post things we find people doing with Node-RED. However, without a date in the post, the urls would end up being <code class="language-plaintext highlighter-rouge">.../community-news-1</code>, <code class="language-plaintext highlighter-rouge">.../community-new-2</code> .. etc. Perfectly functional, but not to my taste.</p> <p>So I clicked to enable dated permalinks on the blog, and promptly found I had broken all of the existing post urls. I had half-hoped that on enabling the option, any already-published posts would not be affected - but that wasn’t the case; they all changed.</p> <p>To fix this, I needed a way to redirect anyone going to one of the old urls to the corresponding new one. I don’t have access to the proxy running in front of my web host, so that wasn’t an option.</p> <p>Instead, I added a bit of code to my installation of Ghost that deals with it.</p> <p>In the top level directory of Ghost is a file called <code class="language-plaintext highlighter-rouge">index.js</code>. This is what gets invoked to run the platform and is itself a very simple file:</p> <pre> var ghost = require('./core'); ghost(); </pre> <p>What this doesn’t reveal is that the call to <code class="language-plaintext highlighter-rouge">ghost()</code> accepts an argument of an instance of <a href="">express</a>, the web application framework that Ghost uses. This allows you to pass in an instance of Express that you’ve tweaked for your own needs - such as one that knows about the old urls and what they should redirect to:</p> <pre> var ghost = require('./core'); var express = require("express"); var app = express(); ghost(app); var redirects = express(); var urlMap = [ {from:'/version-0-2-0-released/', to:'/2013/10/16/version-0-2-0-released/'}, {from:'/internet-of-things-messaging-hangout/', to:'/2013/10/21/internet-of-things-messaging-hangout/'}, {from:'/version-0-3-0-released/', to:'/2013/10/31/version-0-3-0-released/'}, {from:'/version-0-4-0-released/', to:'/2013/11/14/version-0-4-0-released/'}, {from:'/version-0-5-0-released/', to:'/2013/12/21/version-0-5-0-released/'}, {from:'/community-news-14012014/', to:'/2014/01/22/community-news/'} ]; for (var i=0; i&lt;urlMap.length; i+=1) { var to = urlMap[i].to; redirects.all(urlMap[i].from,function(req,res) { res.redirect(301,to); }); } app.use(redirects); </pre> <p>It’s all a bit hard-coded and I don’t know how compatible this will be with future releases of Ghost, but at least it keeps the old urls alive. After all, <a href="">cool uris don’t change</a>.</p> Wiring the Internet of Things - ThingMonk talk 2013-12-05T00:00:00+00:00 <p><em>I spoke this week at the ThingMonk conference. Unlike other talks I’ve given, I actually wrote this one down, rather than my normal approach of throwing some slides together at the last minute.</em></p> <p><em>That has the added benefit of giving me a coherent(ish) written version I can post here. Below is the talk as I roughly planned it, albeit certainly not a word-for-word record of what I actually said.</em></p> <iframe width="560" height="315" src="" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <hr /> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-00-300x230.png" alt="node-red-thingmonk-slide-00" /></a></p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-01-300x230.png" alt="node-red-thingmonk-slide-01" /></a></p> <p>Andy Stanford-Clark, inventor of MQTT, sat here with us today, lives on the Isle of Wight, the south coast of the Isle of Wight.</p> <p>The nature of his job means he has to head to London for meetings fairly regularly. Getting to London from the South cost of the Isle of Wight in time for a morning meeting is no mean feat. To catch the 7.30 train from Southampton, he has to catch the 6.45am ferry from Cowes, for which he has leave home at 6am to catch.</p> <p>On one particular morning, he left home in the early morning sunshine, to arrived at the ferry port in a thick blanket of fog. And the ferries frequently get delayed in the fog.</p> <p>Helpfully, there was a phone line you could call to find out if the ferries were sailing. A helpful answering machine message telling you the state of play. Unfortunately, the guy who updates the message doesn’t arrive until 9, so it still said the ferries were running fine.</p> <p>Sat in his car, with the extra hour in bed he could have had on his mind, Andy did what he often does and started thinking.</p> <p>Surely there must be someway to know if the ferries are sailing.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-02-300x230.png" alt="node-red-thingmonk-slide-02" /></a></p> <p>Firing up his laptop, plugging in his 3G modem, he soon found this site - which shows, in realtime, all of the boats sailing in the Solent. This is based on the [AIS] radio transponders they all carry that broadcast their GPS position and identification.</p> <p>Being a frequent user of the Isle of Wight ferries, Andy was soon able to pick out all of the ferries he used and started scraping their position.</p> <p>A few lines of Perl later - even great minds have their flaws - and he had created a geofence - a virtual boundary - around the Cowes and Southampton ferry terminals. So whenever a ferry arrived or left a terminal, he could get a notification.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-03-300x230.png" alt="node-red-thingmonk-slide-03" /></a></p> <p>The obvious next step was to give each of the ferries their own Twitter account - allowing the ferries to tweet what they were doing.</p> <p>So a quick glance at his twitter stream would tell him if the ferries were sailing as expected - and give him back his hour in bed if they weren’t.</p> <p>As is often the case with innovation, Andy, sat there in the fog-bound car park in Cowes solved a problem personal to him and in doing so, created a Thing of use to a wider audience. So much so that, a few months later, looking at the Red Funnel website to check times for a friend he spotted a new section of the page listing the live arrival and departure times of each ferry.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-04-300x230.png" alt="node-red-thingmonk-slide-04" /></a></p> <p>Curious, he clicked through and it took him to the Twitter accounts he had set up - some one at Red Funnel had found what he’d done and rather than get him to shut it down, integrated it into their own site.</p> <p>Now, Andy being Andy, and it being April 1st that he spotted this, he logged into one of the accounts and sent a ferry to land-locked Milton Keynes.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-05-300x230.png" alt="node-red-thingmonk-slide-05" /></a></p> <p>A short while, and a couple phone conversations, later, Red Funnel formally adopted the system; ‘get into Social Media’ had been on their todo list for ages, but they hadn’t figured out what to do.</p> <hr /> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-06-300x230.png" alt="node-red-thingmonk-slide-06" /></a></p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-07-300x230.png" alt="node-red-thingmonk-slide-07" /></a></p> <p>The Internet of Things is not a single amorphous blob that you can stick a label on. Everyone has their own definition or angle on the subject.</p> <p>What I find fascinating is when an individual is able to bring together different sources of data or events and combine them in some new way. To create a new system or emergent behaviour that wasn’t part of the original intent.</p> <p>The AIS transponders are there for maritime safety, but now, indirectly, allow Red Funnel to provide a service they didn’t know they could.</p> <hr /> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-08-300x230.png" alt="node-red-thingmonk-slide-08" /></a></p> <p>As a long time Linux user I have a physical fight with every projector I have to plug into my laptop. It also means I have a tendency to see any new connected device as something of a challenge; will it work with my laptop - or does it depend on an iPhone app to do anything interesting.</p> <p>From a consumer point of view, the Apple Store iPhone Accessories category is full of high end, beautifully designed connected devices. And that’s how IoT is beginning to enter many people’s lives on a more concious level, albeit at the premium that the Apple experience elicits.</p> <p>But how open are any of these devices for them to used beyond the purposes they were designed.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-09-300x230.png" alt="node-red-thingmonk-slide-09" /></a></p> <p>Take for example, as I’m sure will have been invoked already, the NEST thermostat, and their recently announced Smoke &amp; Co2 detector. Taking one of the more mundane, invisible pieces of electronics in the home, to be a fully connected, intelligent device. On one side, I’m fascinated by what it takes to produce appealing connected devices. On the other, I want to know how much can I tinker with it.</p> <p>So far, the Nest is a complete black box - albeit a round, shiny black box. But, they have announced a developer API to come in the New Year.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-10-300x230.png" alt="node-red-thingmonk-slide-10" /></a></p> <p>By comparision, are the WeMo sockets - the remote controllable sockets made by Belkin. They don’t have a formal api, but the developer community has found ways in and there are open-source libraries out there already for them.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-11-300x230.png" alt="node-red-thingmonk-slide-11" /></a></p> <p>But of course have announced for the robustness and security of their system, they may have to ‘secure’ the open protocols that are being used.</p> <hr /> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-12-300x230.png" alt="node-red-thingmonk-slide-12" /></a></p> <p>The common theme here is how you can empower the individual developer to do something interesting. Open APIs and standards-based protocols - something I’m sure Ian will talk about later today.</p> <p>One of the way that openess manifests itself is within the node.js ecosystem.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-13-300x230.png" alt="node-red-thingmonk-slide-13" /></a></p> <p>One of the many strengths of Node.js, the server-side JavaScript environment, is the NPM repository. This is a repository of almost 50,000 modules that have been created by developers around the world to add all sorts of pieces of functionality into the environment.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-14-300x230.png" alt="node-red-thingmonk-slide-14" /></a></p> <p>A quick search in the repository finds at least 3 different modules written to control WeMo sockets, there’s even one for the NEST to query basic information from the Thermostat.</p> <p>Whatever the device, as time passes, the probability of there being at a relevant module within NPM increases to 1.</p> <p>This can pose its own challenges - there is no arbiter of quality to get a module into the repository; of those three Wemo modules, two of them are still version 0.0.2 and seem to have been orphaned off - so you do have to take some care picking the right modules to use.</p> <hr /> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-15-300x230.png" alt="node-red-thingmonk-slide-15" /></a></p> <p>The days of hardware hacking being the preserve of the highly skilled have long since gone.</p> <p>The arduino, mbed, beaglebone black all exist to make it easier for anyone to start wiring things together; to cross the physical/digital divide and make Things.</p> <p>When you look at the NEST as the current pinacle of connected thermostats, you also have to consider things like this:</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-16-300x230.png" alt="node-red-thingmonk-slide-16" /></a></p> <blockquote> <p>Built an ‘annoying home thermometer’ that plays Arduino ‘Twinkle Twinkle Little Star’ example on loop while room temp is above 25 degrees.</p> <p>@danlockton -</p> </blockquote> <p>It’s that freedom to play, to create something that joins A &amp; B, to experiment with what works for you as an individual.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-17-300x230.png" alt="node-red-thingmonk-slide-17" /></a></p> <p>As part of the Homesense research project, Russell Davies, along with Tom Taylor who spoke earlier, created this bike map. Centred on his flat in London, it has LEDs at each of the local Boris bike stations. Whenever there are more than 5 bikes available at a particular station, the light comes on. It means when he leaves in the morning, with a simple glance he knows whether to turn left or right to find the nearest available bike. There’s no screen here, no mobile app, no intrusive notification. A quick glance and he’s on his way.</p> <p>The ability for individuals to create things for themselves; to solve problems personal to them.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-18-300x230.png" alt="node-red-thingmonk-slide-18" /></a></p> <p>This is Bubblino - Adrian McEwens connected bubble machine that spews out bubbles when people mention it on twitter. Silly, delightful, playful.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-19-300x230.png" alt="node-red-thingmonk-slide-19" /></a></p> <p>A couple years ago, Andy Huntingdon, called this stage of IoT as the Geocities of Things. Geocities being that early space on the internet where may people cut their teeth in creating X-Files fan pages or unwiting tributes to the creator of the animated under-construction gif.</p> <p>It was a space you could create webpages without really knowing what you were doing. Clay Shirky famously thought sites like Geocities would never take off, but later came to realise that:</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-20-300x230.png" alt="node-red-thingmonk-slide-20" /></a></p> <blockquote> <p>Creating something personal, even of moderate quality, has a different kind of appeal than consuming something made by others, even something of high quality.</p> </blockquote> <p>Geocities allowed people to play, to experiement. To create sites that were beautiful in their eyes - if not anyone elses. It’s where a generation learnt the art, or otherwise, of web design.</p> <p>And with IoT today, there is an explosion in platforms, products and protocols that help solve some of the underlying hard technical challenges and provide a space for people to play.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-21-300x230.png" alt="node-red-thingmonk-slide-21" /></a></p> <p>But that explosion brings its own challenges. Each platform has its own set of apis. Each protocol has its own set of clients and servers to learn about. Every technology that makes something easier to do, brings another choice that a developer has to make.</p> <p>Just like the WeMo node.js modules I mentioned.</p> <p>There is never going to be a single, one-size-fits-all solution. Nor should there be.</p> <p><a href=""><img src="/blog/content/2013/12/node-red-thingmonk-slide-22-300x230.png" alt="node-red-thingmonk-slide-22" /></a></p> <p>The challenge of integration is the diversity of skills and knowledge needed to get things talking.</p> <p>Take for example, the challenge of Andy’s Twittering Ferries. It required someone with knowledge of the AIS transponders to create the site that plotted the positions for Andy to find. It required the ability to scrape the data from the page and parse out the individual ferry’s position and compare to the geo-fences. It required the ability to get through Twitter’s OAuth authentication flows to get the tweet posted.</p> <p>For all the security benefits brought by OAuth in replacing the exchange of usernames and passwords, it has probably been the root cause of more than its fair share of headaches as developers try to figure out which token goes where and is signed with what and when.</p> <p>And that’s a common problem. As a developer, you spend more time on HOW to integrate with something than you do on WHAT you want to do with it.</p> <hr /> <p><em>At which point I introduce Node-RED, and demo what can be done with it</em></p> Node-RED - a visual tool for wiring the Internet of Things 2013-09-26T00:00:00+00:00 <p>Node-RED is the visual tool for wiring the Internet of Things that I have been working on through the year. What started as an educational exercise to get to grips with technologies such as Node.js and d3, soon turned into a tool that provided real value in client engagements.</p> <p>One of our goals all along was to open-source the tool; to build a community of users and contributors. Following a whirlwind pass through the appropriate internal process, we got all the approvals we needed at the start of September and have released Node-RED as an Apache v2 licensed open source project on github.</p> <p>You can find us here: <a href=""></a>.</p> <p>Following on from that, I have had the opportunity to speak at a couple of external events.</p> <p>First up was the <a href="">Wuthering Bytes</a> technology festival in sunny/rainy Hebden Bridge. I did a completely unprepared, but well received, 5 minute lightning-talk at the end of the first day which generated a lot of interest. On the second day, there was a series of workshops, one of which was based around MQTT on Arduino - using my <a href="">PubSubClient library</a>. The workshop was organised by a contact of mine from the <a href="">IoTLondon</a> community, who was happy to see me there to help out. We ended up using Node-RED as well in the workshop as it makes it easy to visualise topics and move messages around. It was a great feeling to glance over the room and see people I’ve never met before using Node-RED.</p> <p>The second event was last night, at the monthly London Node.js User Group (LNUG) Meetup. This time I had a 30 minute slot to fill, so went prepared with a presentation and demo. Again, it was well received and generated lots of good comments on twitter and afterwards in person. The talk was recorded and should be embedded below - starts at 29mins.</p> <iframe width="560" height="315" src="" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <p>We’ve got a couple more talks lined up so far. Next up is the November meetup of IoTLondon, exact date tbc, and then <a href="">ThingMonk</a> in December.</p> The Good Night Lamp 2013-01-22T00:00:00+00:00 <p>One of the parts of the Internet of Things that I feel often gets overlooked are the Things. There seems to be a new IoT gateway, hardware platform or consortium to “standardise” every week.</p> <p>These are all honourable activities to move things either forward, or in the case of standardisation consortium, in circles.</p> <p>But what about the Things? This is the part of IoT that appeals to me the most. Crafting a Thing that will appeal to a wide audience takes so much more that a bit of hardware and software. It needs designing to be an object that people willingly accept into their homes. This means much more than just the physical nature of the Thing, but at all levels that it is interacted with.</p> <p>This was one of the inspirations behind my <a href="/orb/">Orb project</a>. But that, in its current state, has a long way to go. It remains the preserve of those who are prepared to get a soldering iron out and write some code.</p> <p>I’ve known <a href="">Alex</a> for a few years now. She gave me my <a href="">Arduino Ethernet shield</a> back in the early days of Tinker and has been a good friend since. As someone with a design background, working in that space, she gets the Things idea better than most.</p> <p>Likewise, <a href="">Adrian</a> proved his IoT credentials long ago with <a href="">Bubblino</a>. Soon you’ll be able to say he (co-)wrote the <a href="">book on the subject</a>.</p> <p>They are working to produce a real Thing for the world to buy. The <a href="">Good Night Lamp</a> is based on an idea Alex had a few years ago and is conceptually simple; a pair of internet connected lamps, one big, one small. When the big one is turned on, then on comes the small one - where ever it is in the world.</p> <p>That’s it. Nothing more, nothing less.</p> <p><img src="/blog/content/2013/01/gnl_web_what.jpg" alt="" title="gnl_web_what" /></p> <p>Imagine giving the big lamp to a loved one abroad; a wife, a boyfriend or an elderly grandparent. As they come and go, turning the big lamp on and off, so does the small lamp on the shelf in your living-room. It gives you a level of ambient intimacy; knowing the loved one is there, without being there.</p> <p>In a work setting, perhaps it signals you’re available for the Skype chat, without resorting to email/IM.</p> <p>That’s it. Nothing more. Nothing less.</p> <iframe src="" width="500" height="281" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe> <p><a href="">Good Night Lamp on Kickstarter</a> from <a href="">Good Night Lamp</a> on <a href="">Vimeo</a>.</p> <p>The geek in me screams “but that’s easy to do - just plug an Arduino into a lamp and you’re done”. But that misses the point. This isn’t simply a lamp with an Arduino in it; it’s a Thing that has been designed.</p> <p>Making Things and shipping them is hard and expensive. To help make it a reality they are currently running a <a href="">Kickstarter campaign</a>. It’s nearing the end and, as it stands, they’ve got quite a way to go.</p> <p>If you like the idea, go and back the campaign. Even if they don’t reach their funding target, an unsuccessful Kickstarter project isn’t going to stop Alex and co making this Thing a reality.</p> A-Mazing Little Printer 2013-01-15T00:00:00+00:00 <p>I made a thing. I wanted to make lots of things last year, but didn’t. That made me sad. It being a new year and all, it felt like a good time to start making.</p> <p>Last summer, I was fortunate enough to be invited to BERG’s <a href="">Little Printer Hack Day</a>. I went along with not only all sorts of ideas to play with but also a migraine and ongoing waves of nausea through-out the day. Which wasn’t ideal. Despite that, it was a fun day and having managed to not pass out, I worked with <a href="">Kass</a> to make the <a href="">ASCII Meterogram</a> - a weather report from the Norwegian Meterological Institute and NRK.</p> <p>A couple weeks ago, whilst contemplating doing a write-up of the year I stumbled over the code from the hack day and decided I ought to do something with it. Later the same day, for reasons I don’t recall, I found myself falling down the rabbit-hole of <a href="">Maze generation algorithms</a> on Wikipedia.</p> <p><img src="/blog/content/2013/01/maze.jpg" alt="" /></p> <p>photo courtesy of <a href="">Alice @ BERG</a></p> <p>Not long later, I had a Little Printer publication that serves up a new random maze every day. There isn’t much to it really - just your standard <a href="">recursive backtracker</a> algorithm in a few lines of javascript. As you do.</p> <p>The code’s up on <a href="">GitHub</a>. It’s a Django app, so if that’s your thing, feel free to have a play. Otherwise, have a look at the <a href="">sample</a>, or subscribe your own Little Printer through the <a href="">BERG Cloud remote</a>.</p> <p>It feels good to make a thing. To make a thing, ship it and see, as of the moment I’m writing this, 30 people subscribed to receive a daily piece of the thing.</p> <p>More making this year. More bad puns in blog post titles.</p> Updated Arduino Client for MQTT 2012-11-11T00:00:00+00:00 <p>I’ve just tagged a new release of the Arduino Client for MQTT - <a href="">v1.9.1</a>. This release includes an API change that will break existing sketches, something I’m very concious of doing - particular as the last release had such changes as well. But ultimately I decided this one was needed.</p> <p>Previously, an instance of the client would be created with a call such as:</p> <pre>PubSubClient client(server, 1883, callback);</pre> <p>The library would then create an instance of <code class="language-plaintext highlighter-rouge">EthernetClient</code> under the covers. This was fine for hardware that used that class, but there are an increasing number of shields that use their own instance of the common <code class="language-plaintext highlighter-rouge">Client</code> API.</p> <p>So, now the constructors for the client require an instance of <code class="language-plaintext highlighter-rouge">Client</code> to be passed in. For the vast majority of existing sketches, this just means changing the above line to something like:</p> <pre>EthernetClient ethClient; PubSubClient client(server, 1883, callback, ethClient);</pre> <p>Amongst the other functional changes are the ability to publish a payload stored in <code class="language-plaintext highlighter-rouge">PROGMEM</code>, connect with a username/password and also the changes described in the <a href="/2012/10/28/talking-to-cosm-from-an-arduino-using-mqtt/">previous post</a> to not fragment MQTT packets over multiple TCP packets.</p> <p>There is also now a regression test suite for the library. Built using python and <a href="">Ino</a>, the suite first verifies each of the examples compiles cleanly, and if an Arduino is connected, it will upload each sketch in turn and run unit tests against it. So far the focus has been on the scaffolding of the test suite, with only a couple actual tests in place for the example sketches. I’ve tried to write it in a way to be agnostic of the library it’s testing. There’s more work to do on that side of things, but ultimately it could well be reused by others. I imagine it’ll branch into its own repository when it gets to that point.</p> <p>I’ve also given the <a href="">project page</a> a tidy and moved the API docs to their <a href="">own page</a>.</p> <p>Finally, I’m considering closing the comments on the project page. with 75 comments already, it isn’t the best medium through which to provide support. I’m not sure pointing to the library’s <a href="">issue tracker</a> on github is necessarily the right thing either - it’s an issue tracker, not a support forum. Perhaps I should set up a dedicated Google group for the library - or is that overkill?</p> <p>Feedback, as ever, is welcome.</p> Talking to Cosm from an Arduino using MQTT 2012-10-28T00:00:00+00:00 <p>Cosm has had MQTT support for some time now and I’ve had my home energy usage bridged up there from my local RSMB quite happily. One thing I hadn’t played with properly was talking to Cosm directly from an Arduino - aside from a 5 minute proof-I-could-do-it.</p> <p>I’m putting together a new release of the PubSubClient, and having had the occasional report of problems talking to Cosm with it, I decided to see where things stood. Sure enough, the connection was pretty unstable; often failing to connect, hanging and certainly not reliably sending or receiving messages.</p> <p>Knowing the client was producing perfectly valid MQTT packets and works with other brokers, the problems had to either exist in the network connection to Cosm or with their broker implementation. The fact my RSMB bridge to Cosm work from the same network ruled out a fundamental network issue. Equally, there are plenty of MQTT brokers on-line these days to check it wasn’t a problem with the Arduino talking to a remote broker, rather than one on the same subnet.</p> <p>Having debugged this sort of thing with other implementations, I had a good idea what the issue was; splitting MQTT packets over multiple TCP packets - something I knew the PubSubClient did.</p> <p>When reading an MQTT packet from the network, it’s easy to assume that a read from the network will provide the next byte of the packet. When testing on local networks, that will mostly be true. But when dealing with internet-scale lag between TCP packets, the code doing the reading must handle the case when the next byte hasn’t arrived yet. This is what I think the Cosm broker is tripping over.</p> <p>A quick test compared writing the packets to the network with multiple calls versus doing it with a single call and saw all of the reliability issues disappear. So I decided to update the client library to do just that.</p> <p>If all you’re after is working code, then you can grab the latest library from <a href="">GitHub</a>. Note that this code includes an API change on each of the constructors - you need to provide an instance of <code class="language-plaintext highlighter-rouge">EthernetClient</code> rather than leave it to the library to create one. You can see what I mean in the updated <a href="">examples</a>.</p> <p>In case you’re interested, here’s a run down of the changes I had to make to get this to work.</p> <p>For the purposes of this exercise, an MQTT packet consists of three components; a single-byte header, between 1 and 4 bytes encoding the remaining length and finally the remaining payload bytes. The existing library provides a function that takes the header and payload, calculates the remaining length bytes and writes each of them to the network in turn - in other words, three separate network writes.</p> <pre> boolean PubSubClient::write(uint8_t header, uint8_t* buf, uint16_t length) { uint8_t lenBuf[4]; uint8_t llen = 0; uint8_t digit; uint8_t pos = 0; uint8_t rc; uint8_t len = length; do { digit = len % 128; len = len / 128; if (len &gt; 0) { digit '= 0x80; } lenBuf[pos++] = digit; llen++; } while(len&gt;0); rc = _client-&gt;write(header); rc += _client-&gt;write(lenBuf,llen); rc += _client-&gt;write(buf,length); lastOutActivity = millis(); return (rc == 1+llen+length); } </pre> <p>The crude way to reduce the network writes would be for the function to copy all of those bytes to a single buffer and then pass that single buffer to the network. But that would involve a lot of unnecessary copying of bytes around as well as the increased memory usage for the buffer.</p> <pre> // Allocate yet more memory static uint8_t packet[MQTT_MAX_PACKET_SIZE]; pos = 0; packet[pos++] = header; for (int i=0;i&lt;llen;i++) { packet[pos++] = lenBuf[i]; } // Copy lots of bytes around for (int i=0;i&lt;length;i++) { packet[pos++] = buf[i]; } rc += _client-&gt;write(packet,pos); </pre> <p>There is already a buffer used for incoming packets which can be reused for the outgoing packets. The difficulty is the remaining length bytes; until the payload is constructed the code doesn’t know how long it will be, which means it does not know how many bytes are needed to encode the length. Without knowing how many bytes are used to encode the length, the client doesn’t know the offset into the buffer to start writing the payload.</p> <p>The trick is knowing that there will be at most 4 bytes of length and one of header - so the payload can be written starting at the 6th byte of the buffer. Then, once the length bytes are calculated, they can be inserted into the buffer working back from the 5th byte and the header byte can be inserted ahead of the length bytes. This gets the MQTT packet into a single contiguous buffer, with minimal copying of bytes, which can be passed to the network - albeit starting at a known offset from the start of the buffer.</p> <pre> // llen is the number of length bytes buf[4-llen] = header; for (int i=0;i&lt;llen;i++) { buf[5-llen+i] = lenBuf[i]; } rc = _client-&gt;write(buf+(4-llen),length+1+llen); </pre> <p>Simple as that.</p> <p>There is one place in the library that can’t be reduced to a single network write; the recently added <code class="language-plaintext highlighter-rouge">publish_P</code> function allows the message payload to be stored in <code class="language-plaintext highlighter-rouge">PROGMEN</code>. The benefits that brings would be entirely lost if they were then copied into regular memory before writing to the network. Not much to be done about that.</p> Moving energy readings from MySQL to redis 2012-08-30T00:00:00+00:00 <p>I’ve been running a site for the <a href="">Chale Community Project</a> for a while now that allows members of the community monitor their energy usage online.</p> <p><em><code class="language-plaintext highlighter-rouge">tl;dr;</code> I changed some code that was using MySQL to store meter readings to use redis instead. The rest of this post describes some of the design decisions needed to make the shift.</em></p> <p>The energy readings are published using a version of my <a href="/2011/04/04/currentcost-mqtt-bridge/">modified CurrentCost bridge</a> over MQTT. As well as the whole house reading, a meter might have a number of individual appliance monitors attached. Each of these readings is published to its own topic in a very simple hierarchy:</p> <pre>sm00/CF35D16315BF93EC053E4EFFC614E3E944C2A626/1/2</pre> <p>The first two topic levels identify the meter and the second two identify the individual devices attached.</p> <p>With readings arriving for each device every 6 seconds, I started with simply dumping them into MySQL - with each level of the topic as a column in the table, thrown in with a timestamp. As this granularity of data isn’t needed for the site, a job could then run every 10 minutes to average the readings into 10 minute slots and put them into another table. This was good enough at the time, but it wasn’t a long term solution.</p> <p>Aside from some serious normalisation of the database (adding the 40 character meter ID in every row turned out not to be the most efficient thing to do), I wanted to move to using redis for the live data. There were a few reasons for this, but primarily it was a combination of wanting to lighten the load on the (shared) MySQL server, as well as to do something more interesting with redis.</p> <p>The shift from relational table to key-value store required a change in mindset as to how to store the readings. The main requirement was to minimise the work needed to store a reading as well as the work needed to calculate the average reading for each device.</p> <p>Redis provides a list data type which, when using the receiving topic as the key, is a perfect fit for storing all of the readings for a particular device. But that causes problems in generating the average as the list will contain all values for the device, not just the values for a particular 10 minute timeslot.</p> <p>The solution is to simply add the timeslot to the key. The timeslot is the current time, such as <code class="language-plaintext highlighter-rouge">2012-08-20 15:21:34</code> with the last three digits set to 0 - <code class="language-plaintext highlighter-rouge">2012-08-20 15:20:00</code>.</p> <p>This leads to keys such as:</p> <pre>20120820152000:sm00:CF35D16315BF93EC053E4EFFC614E3E944C2A626:1:2 20120820153000:sm00:CF35D16315BF93EC053E4EFFC614E3E944C2A626:1:2</pre> <p>That gets the readings stored pretty efficiently, already split into a list-per-device-per-timeslot.</p> <p>The task of averaging the lists could then just look at the current time, generate the timeslot string for the <em>previoius</em> timeslot, retrieve all keys that begin with that string and process all the lists. But there are two problems with this approach.</p> <p>First, redis doesn’t recommend using its <code class="language-plaintext highlighter-rouge">KEYS pattern</code> command in production. I doubt this system would ever scale to the point where the performance hit of that command became an issue, but still, it’s worth sticking to the appropriate best practices.</p> <p>The second issue is that it makes a big assumption that the only timeslot that needs to be processed is the previous one. If the task fails to run for any reason, it needs to process all of the outstanding timeslots.</p> <p>This needed one of the other data types available in redis; the sorted set. This allows you to have an ordered list of values that are guaranteed to be unique. When each reading arrives, the timeslot value is put into a sorted set called, imaginatively, ‘<code class="language-plaintext highlighter-rouge">timeslots</code>’. As it is a set, it doesn’t matter that the same timeslot value will be put into the set thousands of times - it will only result in a single entry. As it is sorted, the task doing the averaging can process the outstanding timeslots in order.</p> <p>Finally, a normal set is used to store all of the reading keys that have been used for a particular timeslot. As this is per-timeslot, the key for the set is ‘<code class="language-plaintext highlighter-rouge">readings:</code>’ appended with the timeslot.</p> <p>So, with that in place, the averaging task takes the follow steps.</p> <ol> <li>Gets the list of timeslots that there are readings for from the sorted set <code class="language-plaintext highlighter-rouge">timeslots</code>.</li> <li> <p>For each timeslot, that is not the current one:</p> <div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. Remove the timeslot from the set 2. Get the list of reading keys from list `readings:_timeslot_` 3. For each reading key, retrieve the list, calculate the average, insert it into the MySQL table and delete the list 4. Delete the `readings:_timeslot_` list </code></pre></div> </div> </li> </ol> <p>And that is what I’ve had running for a few weeks now. No grand conclusion.</p> Orb: The Base 2012-08-23T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em> <img src="/blog/content/2012/08/orb-08.png" alt="" title="orb-08" /> The base of my <a href="/2008/11/25/diy-ambient-orb-redux/">original Orb</a> was cut out of a thick piece of wood using a circle cutting drill bit. A pair of slightly smaller circular bits were then used to create a recess in the top of the base and finally a hole drilled through the centre. This was all it needed to glue the BlinkM in place with the lead running out of the base to the off-board arduino.</p> <p>This approach wasn’t going to work for the new Orb, which needs a hollow base to house all of the boards. It also needed to be something that others could easily replicate.</p> <p>I discounted 3D printing early on, as I wanted the base to be wooden. This left some sort of laser cut thing. There are a number of on-line services for getting things laser cut, such as <a href="">Ponoko</a>. They offer a wide range of materials, including 3mm thick plywood - ideal for this.</p> <p>To figure out what that base was going to look like, I started by glueing a stack of cardboard together and literally carved out the shape I wanted. This let me work out what each of the individual 3mm layers needed to look like. It also helped to identify how to make the base in such a way as to allow the PCB to be accessed; I didn’t want it to be a permanently sealed unit.</p> <div style="float: left; clear: both; margin-right: 20px;">![Orb Base Layers](/blog/content/2012/08/g3986-1.png "Orb Base Layers")</div> <div style="text-align: center;">[![Prototype base](]( "Prototype base by knolleary, on Flickr")</div> <p>The base consists of 12 layers, with 3 in the bottom ‘half’ and 9 in the top (the diagram over on the left omits most of the layers). A pair of nuts, shown in red on the diagram, are sandwiched between layers in the top half, which allow the two sections to be held together using screws. The bottom of the base also includes holes for mounting the PCB stack to it.</p> <p>With the layers drawn up in inkscape, it was time to get them made. A chance conversation with <a href="">Adrian McEwen</a> at a <a href="">#iotlondon</a> meetup enabled me to take advantage of <a href="">DoES Liverpool’s</a> laser cutter.</p> <p>Adrian kindly took some photos of the LASER in action.</p> <p><a href="" title="Lasers! by knolleary, on Flickr"><img src="" alt="Lasers!" /></a> <a href="" title="Lasers! by knolleary, on Flickr"><img src="" alt="Lasers!" /></a> <a href="" title="Lasers! by knolleary, on Flickr"><img src="" alt="Lasers!" /></a> <a href="" title="Lasers! by knolleary, on Flickr"><img src="" alt="Lasers!" /></a></p> <p>Proving yet again, atoms are hard, this first pass wasn’t a complete success. I had made the space in the centre of the base layers the exact same size as the PCBs - so exact that the PCBs didn’t quite fit. A quick rework of the design and another pass with the lasers and I had all the bits I needed.</p> <p><a href="" title="Orb Base by knolleary, on Flickr"><img src="" alt="Orb Base" /></a></p> <p>Some glueing and sanding later, and the base looks quite good - although I don’t appear to have a more recent photo of it to hand. It needs some more work as the design didn’t properly take into account how the orb actually attaches to the base - I have vague thoughts about the liberal use of a hot-glue gun to achieve this currently. I’ve also had some thoughts about changing the style of power-supply connector, which will change the size of opening needed in the side.</p> <p>But all in all, a significant step forward as it now looks more like something people would be prepared to have sat in their home.</p> Orb: The Bottom Board 2012-08-01T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em> <img src="/blog/content/2012/08/orb-07.png" alt="" title="orb-07" /> With the Orb needing to be wireless, one of the early decisions was what sort of wireless to use.</p> <p>The first option to consider was Wi-Fi. This would allow the Orb to connect straight to the Internet via the home router - no intermediary is needed. The downside is that the Orb doesn’t have any sort of UI that would allow configuration of the Wi-Fi settings. Some Arduino WiFi shields will read the configuration from an SD card - but that is not at all a nice user experience. Nor is there a lot of space available on the bottom board to fit all of the hardware in. It is also an expensive option. Likewise Bluetooth and ZigBee weren’t really a good fit.</p> <p>That left a 433Mhz, or similar, radio link. I had played around with some cheap radio transceivers before and they seemed a good way to go. But then I discovered the RFM12B module via <a href="">JeeLabs</a>. Jean-Claude has written a lot about the module on his blog and has created an Arduino library for driving it - perfect for the Orb. Another benefit is that the <a href="">Nanode</a> board comes with this module - making it nicely interoperable.</p> <p>The downside of the module is that it runs at 3.3V - but the rest of the Orb runs at 5V. I considered squeezing in the appropriate voltage regulators to provide both voltages from an unregulated supply. But I decided it would be easier to offload some of that by requiring a regulated 5V power supply. That way, the bottom board only needs a regulator to provide 3.3V to the radio, as well as a set of voltage dividers on each of the signal lines between the radio and the ATmega328. It also opens the option of using a micro-USB port as has become standard with mobile phone chargers - although that’s for the future.</p> <p>Although I said the Orb has no configuration interface, I decided it needed something. A common pattern with radio-linked devices, such as the CurrentCost meters, is that you press and hold a switch to put the device into a pairing-mode. This mode allows you to not only establish a link between the device and a base station, but also to link multiple devices with a single base, whilst maintaining their individual addressability. To this end, I included a surface-mount switch on the <em>underside</em> of this board so it can be accessed through the bottom of the Orb housing.</p> <p>As ever, the schematics are on <a href="">github</a>.</p> <p><a href="" title="PCB #3 by knolleary, on Flickr"><img src="" alt="PCB #3" /></a> <a href="" title="PCB #3 - built by knolleary, on Flickr"><img src="" alt="PCB #3 - built" /></a></p> <p>And that’s the final board. All soldered, stacked and ready to do its thing. What’s next? Making a base to house these boards in. But that’s for another day.</p> <div style="text-align: center; clear: both;">[![Stacked](]( "Stacked by knolleary, on Flickr")</div> Orb: The Middle Board 2012-07-30T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em> <img src="/blog/content/2012/07/orb-06.png" alt="" title="orb-06" /> A couple of years ago, I bought a <a href="">Component Bundle</a> from oomlout - a kit of parts to build your own Arduino-compatible board. This proved to be a great lesson in what an Arduino actually is and how it can ultimately boil down to a small handful of components.</p> <p>As with the other boards, the Eagle files are on <a href="">github</a>. I’ve also included pngs of both the schematic and board for those without Eagle.</p> <div style="text-align: center; clear: both;">![](</div> <p>Here’s a quick run through of that handful of components. <code class="language-plaintext highlighter-rouge">C1</code>, <code class="language-plaintext highlighter-rouge">C2</code> and the 16MHz Crystal keep the ATmega328 at the heart of the board ticking. <code class="language-plaintext highlighter-rouge">C3</code> is a smoothing capacitor between <code class="language-plaintext highlighter-rouge">Vcc</code> and <code class="language-plaintext highlighter-rouge">GND</code>. <code class="language-plaintext highlighter-rouge">C4</code> smooths the reset line from the FTDI header - in comparing various Arduino-clone schematics, I couldn’t find a consensus on whether it’s actually needed. As it’s easier to solder a jumper across unused gaps than it is to retro-fit an additional component, I decided to play safe and added it in. Finally, <code class="language-plaintext highlighter-rouge">R1</code> and the switch allow the board to be manually reset.</p> <div style="text-align: center; clear: both;">[![Eagling](]( "Eagling by knolleary, on Flickr")</div> <p>Laying this board out in Eagle was when I discovered how addictive it can be manually routing all of the connections. There’s an art in making the mess of connections aesthetically pleasing; completely pointless for the functionality of the board, but satisfying to get just so. I also discovered how to add a ground plane to the board - making the bottom layer much easier to layout.</p> <div style="text-align: center; clear: both;">[![PCB #2](]( "PCB #2 by knolleary, on Flickr") [![PCB #2 - half built](]( "PCB #2 - half built by knolleary, on Flickr") [![PCBs #1 &amp; #2](]( "PCBs #1 &amp; #2 by knolleary, on Flickr")</div> <p>After another three-week wait, the boards arrived and the first soldered together. Thankfully, the lessons learnt with the LED boards, and the hours spent making minute tweaks to the layout meant it worked first time. Which, in hindsight, was very pleasing as I hadn’t actually verified the circuit on a breadboard and had going straight into production with a theoretical circuit.</p> Bridging MQTT from RSMB to Cosm 2012-06-09T00:00:00+00:00 <p><a href="">Cosm</a>, formerly known as Pachube, has had MQTT support for a while now and I’ve been meaning to do something with it.</p> <p>One of the strengths of a publish/subscribe system, such as MQTT, is the decoupling of the producers and consumers of messages. I don’t have to modify the code that reads my energy usage from the CurrentCost meter each time I think of a new use of the data - I just add a new, self-contained, subscriber.</p> <p>So to get my energy usage data into a datastream on Cosm, I could simply write a client that subscribes to the <code class="language-plaintext highlighter-rouge">house/cc/power</code> topic on my home broker and republishes it to the appropriate topic on the Cosm broker.</p> <p>But even that’s far more work than is really needed - why write code when the broker can do it for me? RSMB lets you create bridge connections to other brokers and define how topics should be mapped between them. Now, I’ve done this plenty of times, but I still get confused with the topic-mapping syntax RSMB uses in its config file. So, for future reference, here’s how to do it.</p> <pre> connection cosm try_private false address 1883 username <COSM_API_KEY> clientid knolleary topic "" out house/cc/power /v2/feeds/62907/datastreams/0.csv &lt;/pre&gt; The `topic` line shows how you can achieve a one-to-one mapping between different topics - the non-obvious part being the need to have the empty "" as the topicString. If you are using Mosquitto, an additional parameter is needed in the `topic` line to specify the QoS to use: <pre> topic "" out 0 house/cc/power /v2/feeds/62907/datastreams/0.csv </pre> Going the other way, subscribing to a feed on Cosm, is just as easy: <pre> topic "" in inbound/cosm /v1/feeds/XXXXXX/datastreams/0.csv </pre> Or, if you are using Mosquitto: <pre> topic "" in 0 inbound/cosm /v1/feeds/XXXXXX/datastreams/0.csv </pre> Note the use of `/v1/` in the remote topic string this time around. As [documented](, this gets you just the raw data value, rather than the comma-separated time-stamp and data value returned by the `/v2/` api. Of course, depending on what you're doing, you may want the time-stamp. Equally, you may want the full JSON or <shuddder> XML formats - in which case you would change the `.csv` to `.json` or `.xml` respectively. In a future post, I'll look more at how the topic-mapping in RSMB (and Mosquitto) can be used to good effect in scaling an MQTT system. </shuddder></COSM_API_KEY></pre> Queuing with MQTT 2012-04-11T00:00:00+00:00 <p>A question has come up a couple times on the MQTT mailing list asking how it can be used for workload distribution; given tasks being published to a topic, how can they be distributed fairly amongst the subscribers.</p> <p>The short answer is that you can’t - it isn’t how things work in a publish/subscribe system. With MQTT, each subscriber is treated equal and every message published to a topic will go to every subscriber.</p> <p>Despite the natural urge to shoehorn MQTT into any solution, the correct answer is to use the right tool for the job. In this case, something that provides proper queuing semantics to ensure each message is only consumed once - something like <a href="">Celery</a> for example.</p> <p>That said, let’s have a look at one way you could shoehorn MQTT into this particular scenario.</p> <p>First, lets define the scenario. Say, for example, you have a <a href="/2012/04/04/streaming-tweets-to-mqtt/">stream of tweets being published to MQTT</a>. Specifically a stream of all tweets that contain the text “4sq”. Why would you do this? Well, as explained by <a href="">Matt Biddulph</a> in his <a href="">Where 2012 workshop</a>, this gives you a feed of all check-ins on foursquare that have been shared on Twitter. Each tweet contains a url which gives you more metadata about the check-in. To do anything meaningful with the tweets, you need to retrieve the url, which, as it uses the foursqure url-shortener, requires two http requests.</p> <p>The rate at which the tweets arrive, 25,000 an hour Matt suggests, makes it impractical to do anything but the most basic of operations on them. It certainly isn’t practical to do the two http requests to retrieve the foursquare information for each tweet in real-time.</p> <p>There isn’t any real magic needed here; a single subscriber to the topic dumps the messages onto a queue of some form that can then have as many active consumers as makes sense for the scenario.</p> <p>A couple of years ago, I would have jumped straight at dumping the tweets into a MySQL table - how very 2010. These days, it’s so much easier to use something like redis.</p> <pre> import redis import mosquitto // ... // exercise for the reader to: // - create MQTT client, connect it and subscribe to the appropriate topic // - create a redis client and connect it // ... // on_message callback for the mosquitto client def onmqttmessage(client,message): redis.lpush(message.topic,message.payload) </pre> <p>The consumers then become simple redis clients that <code class="language-plaintext highlighter-rouge">[redis.brpop(topic)](</code> to get the next tweet to process. If you want to get extra fancy, then <code class="language-plaintext highlighter-rouge">[redis.brpoplpush(topic,processing_queue)](</code> and a housekeeping thread lets you make sure nothing gets lost if a consumer dies unexpectedly.</p> <p>Looking at this, you may ask why bother with MQTT at all? Why not just replace the MQTT publishing bits in the twitter streaming code with the redis putting bits? Well, I have used the word “shoehorn” three times in this post.</p> Orb: Fading the LEDs 2012-04-08T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em></p> <p>The Hello-World of Arduino sketches is making the on-board LED blink on and off. The equivalent for the orb is getting the three LEDs cycle through their three colours.</p> <pre> // The nine LED pins int LEDS[] = {7,6,5,A0,A1,A2,A3,A4,A5}; void setup() { // Set them all to be outputs for (int i=0; i&lt;9; i++) { pinMode(LEDS[i],OUTPUT); } } void loop() { // Toggle each LED in turn for (int i=0; i&lt;9; i++) { digitalWrite(LEDS[i],HIGH); delay(300); digitalWrite(LEDS[i],LOW); } } </pre> <p>Toggling the individual LEDs on and off like this is trivial but the Orb needs to be able to fade smoothly between colours. The normal approach to do this would be to use the Arduino’s PWM pins and let the hardware deal with it. But with nine pins to control that isn’t an option so the Orb has to resort to software PWM.</p> <p>So, what is PWM and how can it be done in software? Pulse-width modulation is where an LED is rapidly turned on and off, many times a second. The greater the ratio of off to on, the dimmer the LED appears. If done fast enough, the eye doesn’t see any flicker.</p> <p>But therein lies a problem; the <code class="language-plaintext highlighter-rouge">digitalWrite</code> command is slow. Arduino is a great environment for hiding some of the underlying complexity of the hardware, but it occasionally does this at the cost of efficiency.</p> <p>To speed things up, the Orb has to go back to basics and manipulate the pins directly.</p> <p>All of the IO pins on an Arduino are grouped into one of three <a href="">port registers</a>. By toggling the appropriate bits in the registers, the pins can be controlled.</p> <p>Using the same pin assignments as the previous example, the following macros can be used to control the LEDs.</p> <pre> // Bit-masks for each pin of each LED #define l1rm 0x80 #define l1gm 0x40 #define l1bm 0x20 #define l2rm 0x01 #define l2gm 0x02 #define l2bm 0x04 #define l3rm 0x08 #define l3gm 0x10 #define l3bm 0x20 #define LEDA(r1,g1,b1) PORTD = (PORTD&amp;0x03)'(r1*l1rm)'(g1*l1gm)'(b1*l1bm) #define LEDB(r2,g2,b2) PORTC = (PORTC&amp;0xF8)'(r2*l2rm)'(g2*l2gm)'(b2*l2bm) #define LEDC(r3,g3,b3) PORTC = (PORTC&amp;0xC7)'(r3*l3rm)'(g3*l3gm)'(b3*l3bm) </pre> <p>With this is place, we can do a first pass at PWM. For each LED, we’ll define a brightness between 0 and 255. In each loop of the arduino code, a global counter is incremented. If the counter is less then an LED’s brightness, the LED should be on, otherwise it should be off. Which looks something like this:</p> <pre> byte brightness[] = {0,128,255, 0,128,255, 0,0,0 }; byte counter = 0; void loop() { LEDA((brightness[0]&gt;counter)?1:0, (brightness[1]&gt;counter)?1:0, (brightness[2]&gt;counter)?1:0); LEDB((brightness[3]&gt;counter)?1:0, (brightness[4]&gt;counter)?1:0, (brightness[5]&gt;counter)?1:0); LEDC((brightness[6]&gt;counter)?1:0, (brightness[7]&gt;counter)?1:0, (brightness[8]&gt;counter)?1:0); counter++; } </pre> <p>The problem with this approach is that it assumes <code class="language-plaintext highlighter-rouge">loop</code> will get called with an absolute regularity to keep things smooth. The code above would probably work fine, but once you add in handling of the radio to control things, then that regularity could be lost.</p> <p>I came across a solution for this in a <a href="">blog post</a> from the marvellous JeeLabs:</p> <blockquote> <p>What occurred to me, is that you could re-use a hardware counter which is always running in the ATmega when working with the Arduino libraries: the <code class="language-plaintext highlighter-rouge">TIMER-0</code> millisecond clock!</p> <p>It increments every 4 µs, from 0 to 255, and wraps around every 1024 µs. So if we take the current value of the timer as the current time slot, then all we need to do is use that same map as in the original rgbAdjust sketch to set all I/O pins!</p> </blockquote> <p>This introduces another idea that can make the code more efficient; rather than evaluate the brightness levels every time around the loop, create an array representing the 256 time-slots with the appropriate port register values pre-calculated.</p> <p>With the arrays in place, the code in the <code class="language-plaintext highlighter-rouge">loop</code> function simply becomes:</p> <pre> void loop() { PORTD = (PORTD&amp;0x03)'portDSlots[TCNT0]; PORTC = (PORTC&amp;0xC0)'portCSlots[TCNT0]; } </pre> <p>Setting up the slots can be done with the following code:</p> <pre> void setupSlots(int r1,int g1,int b1,int r2,int g2,int b2,int r3,int g3,int b3) { memset(portDSlots,0,256); memset(portCSlots,0,256); for (int i=0;i&lt;256;i++) { portDSlots[i] = ((r1&gt;i)?l1rm:0)'((g1&gt;i)?l1gm:0)'((b1&gt;i)?l1bm:0); portCSlots[i] = ((r2&gt;i)?l2rm:0)'((g2&gt;i)?l2gm:0)'((b2&gt;i)?l2bm:0)' ((r3&gt;i)?l3rm:0)'((g3&gt;i)?l3gm:0)'((b3&gt;i)?l3bm:0); } } </pre> <p>Fading can then be achieved by incrementally stepping to the desired brightness levels.</p> <p>As before, the full code is available on <a href="">github</a>.</p> <div style="text-align: center;"><iframe src="" width="500" height="281" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe></div> Streaming Tweets to MQTT 2012-04-04T00:00:00+00:00 <p>For a demo I’m putting together at work, I needed to have tweets available on an MQTT topic. The whys and wherefores of the demo might be something I’ll write about in the future, but for now, this post is about getting the tweets into MQTT.</p> <p>To be honest, it is pretty trivial. The Twitter Streaming API is <a href="">well documented</a> and can be used to get a stream of tweets based on whatever search terms you choose. As each tweet is received, it just needs to be published with an MQTT client to the topic of choice.</p> <p>It’s barely worth a blog post.</p> <p>I’ve put the code up on <a href="">github</a>.</p> Orb: The LED Board 2012-04-02T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em> <img src="/blog/content/2012/04/orb-03.png" alt="" title="orb-03" /> The top board in the Orb is the most simple - all it needs is three RGB LEDs and their resistors.</p> <p>A quick <a href="">search on ebay</a> found a good supply of LEDs similar to those used on the BlinkMs. The datasheet on the auction listing gave me all the information I needed to start laying out the PCB.</p> <p>I considered writing a more detailed post about using Eagle, but there are plenty of tutorials out there already (<a href="">here’s</a> the Sparkfun one that got me started).</p> <p>The Eagle files for the board are on <a href="">github</a>.</p> <p>With that done, I looked around at options for getting the PCB made. <a href="">Olimex</a> had been recommended to me by a colleague, but to be honest I found their long list of design rules quite overwhelming for a first timer. I eventually settled on <a href="">SeeedStudio’s Fusion PCB service</a> - who provide their own Eagle design rules file for you to run against your design. It worked out at about £1 for each board - which seemed pretty reasonable and you get 10 of them to play with.</p> <p>The only downside was the wait - it took just over three weeks for the boards to arrive. Rapid prototyping for those not in a hurry.</p> <p><a href="" title="PCB #1 by knolleary, on Flickr"><img src="" alt="PCB #1" /></a> <a href="" title="PCB #1 by knolleary, on Flickr"><img src="" alt="PCB #1" /></a> <a href="" title="PCB #1 by knolleary, on Flickr"><img src="" alt="PCB #1" /></a></p> <p>When they did eventually arrive, it didn’t take me long to find the faults in the design. Despite my hours staring at the layout in Eagle, there were a couple things wrong. First, I realised I had missed a connection on the board layout - this was before I learnt enough about the Ratsnest tool which would have spotted the mistake. The second fault was with the custom Eagle part I had made for the LEDs - the outline was rotated 90°, putting GND somewhere it shouldn’t be. I didn’t spot this until I was figuring out what had caused the <a href="">LEDs to burn-out</a> on my first fully soldered board. Thankfully both of these could be resolved without having to get another batch of boards made - a jumper wire replaces the missing connection and I need to remember to ignore the printed outline when placing the LEDs.</p> <p>With these faults worked around, I had a working LED board. It only took a handful of wires and an Arduino to see it in action. Cue the video.</p> <div style="text-align: center;"><iframe src="" width="500" height="281" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe></div> Orb: Introduction 2012-03-27T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em></p> <p>The original Orb was a simple thing; a BlinkM on the end of a long piece of telephone cable plugged into an Arduino, controlled by the low-powered Linux server behind my TV that’s always on.</p> <p><img src="/blog/content/2012/03/orb-01.png" alt="" title="orb-01" /></p> <p>That isn’t the sort of simplicity that makes it suitable for everyone. Putting aside how the behaviour of the Orb is configured, the physical design had two drawbacks:</p> <ul> <li>the need for an always-on PC and</li> <li>having the Orb physically tethered to the PC</li> </ul> <p>These were key to the new design.</p> <p>The always-on PC can be replaced with an Arduino Ethernet, connecting to an on-line service for configuration. This puts the base unit within cable reach of the broadband router - but unlikely to be near the ideal spot for the Orb to sit. So the cable between the Orb and Arduino can be replaced with a wireless link. It will still need a power supply, but that’s more likely to be available wherever the Orb ends up.</p> <p>This has the knock-on effect of requiring more than just a BlinkM, or equivalent, in the Orb; it needs an Arduino in the base. Looking at the available boards, the only potential one that might fit is the LilyPad - but even that is a bit large and doesn’t quite lend itself to the form factor I have in mind. This means it needs a custom board.</p> <p>With regard the BlinkM, I have already <a href="/2009/12/14/multi-channel-ambient-orb/">prototyped a custom LED board</a> to achieve multiple channels. That needs shrinking down further to a single board.</p> <p><img src="/blog/content/2012/03/orb-02.png" alt="" title="orb-02" /></p> <p>This brings me to the basic design of the new Orb. It consists of three boards that stack together - much like an Arduino and its shields. The top board holds three RGB LEDs and the resistors they need. The middle board includes the basic Arduino circuit, consisting of a micro-controller and the bits needed to make it tick, as well as a programming header. Finally, the bottom board has the main power connector and associated components along with a radio transceiver.</p> <p>All within the base of an Orb.</p> <p>Over the next couple of posts, I’ll write more about the boards themselves. They were the first PCBs I’ve designed in Eagle and taken all the way through to getting fabricated.</p> Calling Time 2012-03-21T00:00:00+00:00 <p>One of my plans for the year was to take photo everyday. I knew it would be a challenge as I spend most days at my desk in work. Trying to find a different, interesting photo everyday when I’m not visiting new places all the time meant opening my eyes to the everyday.</p> <p>But it hasn’t really worked out like that. Looking through the photos to date, too many have been taken after 10pm as I’m heading to bed and needed to take a photo of <em>something</em>.</p> <p>How else can I explain the <a href="">photo of an orange</a>?</p> <p>Then last week, we were ill and although I dutifully took a photo each day, I didn’t get around to uploading them. Last night, sat in bed, I realised I hadn’t done that day’s photo.</p> <p>All of these factors have resulted in my decision to stop the Flickr 366 project. It had become a chore and not fun.</p> <p>I’m glad I tried and I’m sure I’ll try again in the future, but the <a href="">366:2012 set on Flickr</a> is done.</p> Updated Arduino MQTT Client 2012-03-08T00:00:00+00:00 <p>A new version of the Arduino MQTT client is available - from the <a href="">usual place</a>.</p> <p>This release brings a handful of changes, but there are a couple I wanted to draw attention to; particularly as <em>one will require a minor change to sketches using this library</em>.</p> <p>When I wrote the first prototype of the client, I made the decision to limit packets to 128 bytes. This was a pragmatic decision to balance memory usage and usefulness; who would possible want to publish more than 128 bytes from an Arduino?</p> <p>I knew the limit was just a <code class="language-plaintext highlighter-rouge">#define</code> in the code, so anyone could change the limit if they wanted. But I didn’t think that all the way through and it turns out there were a couple of issues with this:</p> <ul> <li><code class="language-plaintext highlighter-rouge">uint8_t</code> was used in lots of places to store lengths, which put an internal limit of 256 bytes,</li> <li>the packet length was always encoded as a single byte - which is only correct for packets shorter than 128 bytes due to the multi-byte integer encoding MQTT uses.</li> </ul> <p>So, with this release, you can now modify the <code class="language-plaintext highlighter-rouge">#define</code> in <code class="language-plaintext highlighter-rouge">PubSubClient.h</code> to a larger value and it’ll work.</p> <p>The second change is a wide ranging tidy up of the types used in the library, both internally and on the API. The external part of this is primarily that all of the functions that were defined to return an <code class="language-plaintext highlighter-rouge">int</code> type to indicate success or failure, now return a <code class="language-plaintext highlighter-rouge">boolean</code> type. This change should be compatible with existing scripts.</p> <p>A knock-on effect of both of these changes will require a change to any sketch that provides a message callback. The <code class="language-plaintext highlighter-rouge">length</code> argument of the callback function signature has changed from <code class="language-plaintext highlighter-rouge">int</code> to <code class="language-plaintext highlighter-rouge">unsigned int</code>. When you try to compile a sketch without this change, you’ll get an error along the lines of:</p> <pre>error: invalid conversion from 'void (*)(char*, byte*, int)' to 'void (*)(char*, uint8_t*, unsigned int)' </pre> <p>Hopefully that will help the google-juice for anyone who hits this error and hasn’t read the change log.</p> <p>As ever, check out the <a href="">project page</a> for more details and a download link.</p> More About MQTT 2012-02-25T00:00:00+00:00 <p><img src="/blog/content/2012/02/mqttorg-glow.png" alt="" title="mqttorg-glow" /></p> <p>I’ve been working with MQTT for 5 years now. In that time I’ve learnt a thing or two about how it can be used in all manner of scenarios. With the opening up of the protocol and the <a href="htp://">growing community</a> around it, I find questions coming up on the <a href="">mailing list</a> that remind me how I’ve taken some of what I’ve learnt for granted.</p> <p>Some of that ends up on the <a href="">wiki</a>, but there are a number of things I want to write about in a longer format that I’ve decided to post here.</p> <p>If there’s something in particular you’re interested in, leave a comment or post it to the <a href="">mailing list</a>.</p> <p>Yes, this is another thread of content to bolster my output here, but there’s nothing wrong with that.</p> <p>I’ll update this post with links to the articles as they appear. You can find all of my MQTT related posts, new or old, <a href="">here</a>.</p> Making this site mobile friendly 2012-02-23T00:00:00+00:00 <p>Another post in the series about things I’ve done with this site rather than write actual content.</p> <p>I’ve been thinking about how to make things more friendly to mobile browsers around here. The easy option was to just install one of the established plugins, such as <a href="">WPTouch</a>, to do all the heavy lifting. Which is exactly what I did a while ago, but I was never completely happy with the result. I didn’t like the idea of having to create a mobile-specific theme, but without one the site would become yet another blog using WPTouch’s default theme.</p> <p>Going back to basics, there were essentially three things that needed looking at; layout, navigation and bandwidth.</p> <p>Here’s what the site looks like today:</p> <p><img src="/blog/content/2012/02/knolleary-ss.png" alt="" title="knolleary-ss" /></p> <p>A post’s meta-information floats over on the left, outside the column of the page. That’s fine if you’ve got the screen width to accommodate it, but on a mobile screen, it pushes the real content off the screen to the right.</p> <p>To fix that, CSS Media Selectors come to the rescue. They allow you to specify CSS rules that should only be applied if certain conditions are met. In this instance, the conditions relate to the width of the screen displaying the page.</p> <p>Here’s an excerpt from the <a href="">stylesheet</a> used to move the meta-information in-line with the main content:</p> <pre> @media screen and (max-width: 1000px) { .meta { background: #efefef; float: none; width: 780px; margin-top: 0px; margin-left: 10px; border-bottom: 1px solid #999; padding: 3px; } } </pre> <p>Which results in something like this:</p> <p><img src="/blog/content/2012/02/knolleary-ss2.png" alt="" title="knolleary-ss2" /></p> <p>There are a number of other tweaks in the CSS to adapt the main column width depending on the screen size. For example, the footer content that is normally presented in three columns drops to two columns below a certain width and down to a single column at the smallest (ie mobile) size.</p> <p>The nice thing with this technique is that it isn’t specific to mobile; try resizing your desktop browser window to see it in action.</p> <p>The next thing to tackle was navigation - in particular, keeping the links at the top accessible. A little bit of CSS and theme tweaking later, the site now has a static mini-header at the top that reveals itself as you scroll down the page. It is a simple effect, but one I find quite pleasing.</p> <p>All of the things described so far are handled in the browser. But some work was needed on the server side to reduce the size of the page in terms of bandwidth. The first step of which was to properly detect if the browser accessing the page is a mobile or not - easily done by checking the User-Agent header for signs of mobile. The only slight difficulty is determining what a mobile user-agent looks like. Of course there are a number of open-source plugins for doing this sort of thing, but for now I’ve <a href="">rolled my own</a>. It won’t detect every mobile browser under the sun, but it’ll hit the vast majority of them.</p> <p>With that done, there are two things this site will do if it finds it is being served to a mobile. First, the ‘recent things’ column in the footer is omitted. This column comes from FriendFeed and likely contains a lot of images from Flickr, YouTube and Vimeo - depending on what I’ve been up to. This has the added benefit of reducing the length of the page, which gets quite excessive once the footer’s three columns are stacked up.</p> <p>The second thing done is a filter is applied to the main content of the page to replace any Flickr-sourced image with a smaller version. This can be done thanks to the predictable nature of <a href="">Flickr URLs</a>.</p> <p>I normally use the 500px wide images, which, as per the flickr docs, have urls of the form:</p> <pre>http://farm{farm-id}{server-id}/{id}_{secret}.jpg</pre> <p>which just needs an <code class="language-plaintext highlighter-rouge">_m</code> adding after the <code class="language-plaintext highlighter-rouge">{secret}</code> to get the 240px version.</p> <p>Here’s the function that does just that:</p> <pre> function wp_knolleary_shrink_flickr_images($content) { if (wp_knolleary_is_mobile()) { return preg_replace('/(static\.?\/.*?\/[^_]*?_[^_]*?)(\.jpg)/',"$1_m$2",$content); } else { return $content; } } add_filter( 'the_content', 'wp_knolleary_shrink_flickr_images'); </pre> <p>Admitted I wrote that regex late last night so could probably be simplified. But it works.</p> <p>And that’s where I’ve got to so far. I’m sure there is a lot more I could do on this topic - any feedback would be welcome. I find it much more satisfying to figure these things out, rather than just activate a plugin that does it all for me.</p> <p>On a related note, I’ve put the <a href="">theme</a> and <a href="">plugin</a> that this site uses onto github. More as a backup than in expectation of anyone forking it.</p> Reintroducing the Orb 2012-02-06T00:00:00+00:00 <p><em>[ part of the <a href="/orb/">orb series</a> ]</em></p> <p>Can it really be <a href="/2009/12/14/multi-channel-ambient-orb">two years</a> since I last wrote about my ambient orb? Given my output rate here, I’m not that surprised.</p> <p>As a quick recap, I wrote about the idea of a multi-channel orb; using multiple RGB LEDs to let the orb show more than one colour at a time. I ended the post saying the plan was to put a small controller in the base to drive all the leds and that is how it has stayed for most of the last two years.</p> <p>More recently, I’ve been thinking about how to build on this idea, to actually make something with it. And rather than simply cobble together a one-off, I’ve been thinking about how to make something that others could build for themselves.</p> <p>I’ve actually been working on this on and off for a few months now; I wanted to get to a point where I had something real to write about, rather than let it be yet another idea that doesn’t go anywhere. It’s been at that point for a while now, but every time I’ve sat down to start writing, I’ve drawn a blank.</p> <p>Rather than let another month go by without doing something about it, I’m taking the pressure off myself by writing this post to introduce the subject, but without any real content.</p> <p>For now, here’s a preview of where it’s heading.</p> <p><a href="" title="Prototype base by knolleary, on Flickr"><img src="" alt="Prototype base" /></a></p> My plan for 2012 2012-01-01T00:00:00+00:00 <p>I had a <a href="/2010/12/31/my-plan-for-2011/">plan</a> last year. The intention was there, but the execution was lacking. This year will be different.</p> <p>Rather than some nebulous, but well intentioned, aphorism, here are some of things I intend to do this year in no particular order.</p> <ul> <li>Flickr 366 - a photo a day. I’ve tried and failed this before; I think I set the bar too high wanting a year of high quality photos. So, here are my rules: (at least) one photo taken everyday. Uploaded to Flickr as and when I can - no point stressing over not being near my laptop or an Internet connection at 11:59pm. I’d rather favour photos from my ‘proper’ camera, but I also want to get used to spotting the opportunities that come from having my phone in my pocket. Instagram may not be on Android yet, but the Flickr app just about works reliably. Why 366? It’s a leap year.</li> <li>Write more - I managed a paltry nine posts last year; not even one a month. I tend to write about the things I’ve made or done. I’m not sure I want to change that approach, which leads me to…</li> <li>Make more - more codey things, more physical things, more thing things. Bits, Atoms and the pieces in between.</li> </ul> <p>Hmmm, bit of a nebulous list there. Never mind.</p> <p>Onwards.</p> Arduino 1.0 and the MQTT Client 2011-12-01T00:00:00+00:00 <p>With the long-awaited <a href="">release of Arduino 1.0</a>, it is time to do a new release of the MQTT client.</p> <p>Aside from some internal updates needed due to API changes in 1.0, this client has the improved keep-alive handling that I mentioned back in <a href="/2011/05/25/staying-connected/">May</a>.</p> <p>As ever, links to the download on the <a href="">project page</a>.</p> Big Red (usb) Button 2011-10-04T00:00:00+00:00 <p>A few weeks ago, <a href="">Ben</a> was putting together a proof-of-concept for what turned into the <a href="">MQTT powered video wall</a>. He needed a button to trigger the publishing of an MQTT message. We happened to have a <a href=";tbm=isch&amp;biw=1920&amp;bih=968">Staples Easy Button</a> lying around that was ripe for the hacking.</p> <p><a href="" title="Big Red (usb) Button by knolleary, on Flickr"><img src="" alt="Big Red (usb) Button" /></a></p> <p>A short time spent with scalpel and soldering iron and I had a pair of wires sticking out the bottom of the button that could be plugged into an arduino; which was good enough for Ben’s demo.</p> <p><a href="" title="Big Red (usb) Button by knolleary, on Flickr"><img src="" alt="Big Red (usb) Button" /></a></p> <p>But then this week <a href="">Dale</a> was wanting a button to use as a part of a <a href="">Watson</a> talk next week. Rather than just hand him the button, an arduino and a guide to wiring them together, I decided to be more helpful.</p> <p>I knew the base of the button had some empty space once the batteries and speaker were removed and I had a spare <a href=";cPath=1&amp;products_id=15">Freeduino Nano</a> awaiting a project, so it was back to the scalpel and, this time, hot glue gun.</p> <p><a href="" title="Big Red (usb) Button by knolleary, on Flickr"><img src="" alt="Big Red (usb) Button" /></a></p> <p>So we now have a button with a usb connection that can plug straight into a laptop to do stuff. Unlike RIG’s <a href="">Big Red Button</a> it doesn’t emulate a keypress as it doesn’t appear as a HID device to the laptop - I’ll save that for another day.</p> <p>At the moment, the sketch on the arduino simply writes ‘hit’ to the serial port when the button is pressed, so it needs something to run on the laptop to get the message and act accordingly. Which is trivial enough with python:</p> <pre> import serial PORT="/dev/ttyUSB0" def buttonPressed(): print "Do your magic here" s = serial.Serial(port=PORT, baudrate=9600) while True: l = s.readline() if l[:3] == 'hit': buttonPressed() </pre> <p><a href="" title="Big Red (usb) Button by knolleary, on Flickr"><img src="" alt="Big Red (usb) Button" /></a></p> 10 Years an IBMer 2011-09-03T00:00:00+00:00 <p>Before going to university, when asked what I wanted to do, I would glibly say I’d probably go do a Computer Science degree and end up working for IBM. This wasn’t my life’s dream, it was just the easy reply to the question. I don’t remember why I picked IBM - I guess it was a big company that surely did interesting and worthwhile things that people would have heard of.</p> <p>In my final year at uni, getting to stage when I had to think about a job, companies were starting to run recruitment events. Having collected all the free pens, desk toys and post-it pads from the myriad of financial services and consulting companies eager for my soul, I decided it was time to dip my toe in the water and actually apply for something. Despite the freebies, I had little interest in what they had to offer by way of employment, so when IBM came along I thought it was worth a try. As did a dozen or so others from my course.</p> <p>After passing the aptitude and programming tests, I found myself headed to a two-day assessment centre in Hampshire with a group of my course-mates. It was a fun couple of days doing team exercises, a presentation and formal interview at the end of which we were told we’d hear within the next couple weeks. We headed for the train back to uni.</p> <p>About 10 minutes into the journey, one of the guys phone rang. It was the IBM recruiter offering him the job. The call lasted a couple minutes at the end of which he hung up and we all congratulated him. Then his phone rang again. Quickly answering it, we saw him look around to us all and say “Yes, they are travelling back with me”. He then passed his phone over to one of the other guys, who was then duly offered a job.</p> <p>Anyone who has travelled north of Winchester on a train and tried to use a mobile phone will know what happened next. The signal cut out. We had entered the mobile black hole of Hampshire - a zone that has barely improved since.</p> <p>Another nervous 10 minutes later and finally his phone rang again. One by one, it got passed around to each of us and we were all offered a job.</p> <p>And with that, I was employed by IBM.</p> <p>Fast-forward to September 3rd 2001 and I found myself turning up for my first day of work. 10 years ago today.</p> <p>In that time, I’ve tested our Storage Virtualisation product, worked with customers in its beta program, travelled to the US twice (San Jose both times, with an added weekend customer visit to rural Arkansas) and Germany once. I’ve lead the Pervasive messaging team, worked on MicroBroker and MQTT, spent a couple of hours pretty much every Thursday afternoon on interminable status calls with the US and helped mind-control taxi’s for TV.</p> <p>For what was always the easy reply to the question of what I wanted to do, I’ve done quite a lot.</p> <p>My current role in the Emerging Technologies team, which I started in July, holds the promise of doing even more varied and interesting things.</p> <p>But I do have the nagging feeling that I’m still here because it’s the easy option. I came close to taking a year-off last year to do interesting things with the now sadly gone Tinker. But then my son arrived and it wasn’t the right time to be doing something like that.</p> <p>Will I still be here in another 10 years? I don’t know. If I am, I hope it isn’t because it was the easy option.</p> Interesting 2011 2011-06-19T00:00:00+00:00 <iframe src=";byline=0&amp;portrait=0" width="500" height="281" frameborder="0"></iframe> <p>Sandy Noble’s <a href="">PolarGraph</a> plotter in action at Interesting2011.</p> <p><a href="" title="Whiteboard Pixel Russell by knolleary, on Flickr"><img src="" alt="Whiteboard Pixel Russell " /></a></p> Staying Connected 2011-05-25T00:00:00+00:00 <p>I’ve been running my <a href="/2011/04/04/currentcost-mqtt-bridge/">CurrentCost MQTT Bridge</a> for a couple of months now and it has been working well, but for one thing; on occasion it would lose its connection and not reconnect.</p> <p>It has taken some time to iron out the various issues that were causing this, but I think I’ve finally got there so I’ve updated the <a href="">gist</a> and herewith a run down of what it took to get things working properly.</p> <p>Whenever the code realised it had lost its connection, it would simply try to re-establish the connection. Now, as soon as the connection is lost, it performs a reset of the ethernet hardware, re-obtains a DHCP address and runs the DNS query. If either of these latter stages fails to complete in a reasonable time, it loops back to the hardware reset.</p> <p>In an earlier version of the reset code I wasn’t timing out the DHCP look-up. On more than one occasion I spotted, thanks to Wireshark, that the DHCP request being sent by the Bridge had a source MAC value of 00:00:00:00:00 - which was correctly being ignored. Without a time-out, the bridge was stuck sending out invalid requests and waiting for a reply that wasn’t coming. I haven’t been able to identify exactly why the bridge appeared to have forgotten its MAC address, but to rule out the obvious, I added code to restore the MAC from EEPROM as a part of the reset code.</p> <p>This combination of changes made things much more reliable, but there remained one final issue - it relied on the code noticing it had lost its connection. In most cases, this worked fine, but there was still a case where the bridge would be sat publishing its data with no-one listening.</p> <p>MQTT has a facility that is supposed to deal with this; the keep-alive timer. But my MQTT library only implemented half of this feature - it ensured a ping was sent within the keep-alive time if nothing else was sent, but it never checked that it received the expected response. As the bridge code publishes data regularly but doesn’t subscribe to any topics, then proper handling of the keep-alive pings is all the more vital to detecting loss of connection. I’ve made the necessary updates to the MQTT library on <a href="">github</a>, but I haven’t made a formal release of it yet.</p> <p>The lesson to learn from this is that it is easy to write code that works most of the time, but when you’re writing code to run on an unattended, embedded, device you need as many belts and braces as possible to ensure it keeps going.</p> Using MQTT with Pachube 2011-04-07T00:00:00+00:00 <p><img src="/blog/content/2011/04/pblogo.png" alt="Pachube" title="Pachube" /> I have been thinking about how MQTT could be integrated into <a href="">Pachube</a>, as a service that utilises their <a href="">public API</a>. With their <a href="">Hackathon</a> happening tomorrow, which I’m unable to attend, it felt like a good time to write up what I’ve done.</p> <h3 id="basic-pachube-concepts">Basic Pachube concepts</h3> <p>The way Pachube structures data lends itself well to a topic hierarchy:</p> <ul> <li>An Environment is a logical collection of datastreams. It can be used to represent a physical location, such as my house.</li> <li>A Datastream represents a single sensor/device within a particular environment, such as living room temperature.</li> <li>A Datapoint is a point-in-time value on a datastream</li> </ul> <p>Data is sent to Pachube using HTTP POST requests. It can retrieved by either polling a datastream’s value, or by configuring a trigger. A trigger is a push notification that happens when a defined condition occurs. This notification exists as an HTTP POST from Pachube to the URL configured as a part of the trigger. The POST contains a json payload that contains the details of the notification. There are some limitations on data-rates around triggers; the minimum interval between the same trigger firing is 5 seconds.</p> <p>All access to the API is controlled using API Keys. Each request to the API must be accompanied with either the <code class="language-plaintext highlighter-rouge">X-PachubeApiKey</code> HTTP header, or as a request parameter. Each user has a master API key which gives full access to their account. Additional API keys can be created with different levels of access for a particular user.</p> <p>The following scheme uses the API key as a part of the topic strings used. This relies on the broker having been configured to disallow any client to subscribe to the wild-carded topic “pb/#”. Beyond that, no more security is assumed (ie username/password, SSL etc), but of course could be added.</p> <p>There are two components to this; an MQTT subscribing application that can bridge into Pachube and an HTTP listening application that can bridge from Pachube triggers into MQTT.</p> <h3 id="publishing-to-pachube">Publishing to Pachube</h3> <p>A datapoint value is published to the topic:</p> <pre>pb/<api key="">/<environment>/<datastream>&lt;/pre&gt; which is received by the subscribing application, leading to it sending an HTTP PUT of the datapoint to the Pachube API: <pre><environment>/datastreams/<datastream>.csv&lt;/pre&gt; with header property: <pre>X-PachubeApiKey: <api key="">&lt;/pre&gt; No pre-configuration is needed here - it relies on the provided API key having the correct permission to update the datastream. There are some questions here on how to handle failures to post (permission denied, service unavailable etc), as there is no mechanism to report failures back to the originating publisher. ### Subscribing to Pachube Subscribing to a datastream is slightly more involved as it requires an HTTP intermediary for Pachube to push its notifications to which can then be forwarded to the broker. There is also no way to automatically create the trigger when a client subscribes to a topic on the broker, so an additional configuration step is required of the client. It is also possible to configure multiple triggers for a single datastream, so we cannot use a simple one-to-one mapping of datastream to topic for the outbound flow. This causes an asymmetry between publishing and subscribing, but I think it is necessary. A trigger is configured by publishing to the topic: <pre>pb/<api key="">/<environment>/<datastream>/trigger&lt;/pre&gt; with a payload that defines the trigger in json, for example: <pre>{ "topic":"lt15", // As defined in the pachube API: "url":"", "threshold_value":"15.0", "trigger_type":"lt" }</pre> This causes the appropriate HTTP PUT to the Pachube API to create the trigger. The trigger is configured with the url of the HTTP intermediary that is able to react to the HTTP POSTs sent by Pachube by publishing the value to the corresponding MQTT topic. This means clients can subscribe to: <pre>pb/<api key="">/<topic>&lt;/pre&gt; to receive the updates as they are received. As an aside, there could be an option to make the trigger feed 'public' on the broker, so a user's API key doesn't need to be shared. This would mean the trigger would get published to something like: <pre>pb/public/<topic>&lt;/pre&gt; To unsubscribe, the trigger needs to be deleted on Pachube. This is done by publishing to the same topic as was used to create the trigger, but with a payload that includes a 'delete' command. ### The bit at the end I have implemented pretty much all of what I describe above and it works. Although there are some drawbacks with the approach: * The rate-limiting that Pachube applies to triggers (at least 5 seconds between firings) defines the minimum granularity for getting data out to MQTT - so potentially values could be missed. * Subscribing to a datastream is not as simple as subscribing to a topic. * No consideration is made of the additional meta-data that is provided with the HTTP requests - this is very much just about the raw data points In an ideal world, the MQTT interface would sit much closer to the centre of Pachube and be more of a 1st class API citizen; but then I want to MQTT enable everything. Unfortunately I don't have the wherewithal to make it public facing for other's to play with at the moment, which is a shame. If you're really interested in seeing it in action, get in touch and I'll see what can be done. </topic></pre></topic></api></pre></datastream></environment></api></pre></api></pre></datastream></environment></pre></datastream></environment></api></pre> Processing the Theme 2011-04-06T00:00:00+00:00 <p>I often spend time tweaking the theme of this site rather than write content for it, but I thought the latest tweak was worth a few words of mention.</p> <p>I have tried to keep the theme quite clean - with all of the traditional sidebar content pushed right down at the bottom of the page. Previously, there was a simple division between the bottom content and the rest of the page; nothing more than a <code class="language-plaintext highlighter-rouge">border-top</code> CSS declaration:</p> <p><img src="/blog/content/2011/04/kss01.png" alt="" title="knolleary screenshot" /></p> <p>Inspired by the new look <a href="">Planet GNOME</a>, I wanted to do something a bit more interesting down there - some sort of ragged edge, but I didn’t want to spend an age trying to draw it by hand.</p> <p><img src="/blog/content/2011/04/gnomess.png" alt="" title="Planet GNOME screenshot" /></p> <p>Coincidentally, earlier today I was looking for an excuse to play with <a href="">Processing</a> and this seemed like just the thing. I very quickly got something producing a very regular pattern:</p> <p><img src="/blog/content/2011/04/edge000.png" alt="" title="edge001" /></p> <p>But that felt too boring - it needed some randomness.</p> <p><img src="/blog/content/2011/04/edge001.png" alt="" title="edge001" /></p> <p><img src="/blog/content/2011/04/edge002.png" alt="" title="edge002" /></p> <p>These were way too severe. To emulate the grass-like effect, I moved to using curves, rather than straight lines.</p> <p><img src="/blog/content/2011/04/edge004.png" alt="" title="edge004" /></p> <p><img src="/blog/content/2011/04/edge005.png" alt="" title="edge005" /></p> <p>After a couple attemps where I didn’t have my Bezier control points under control, I finally hit upon something I liked:</p> <p><img src="/blog/content/2011/04/edge003.png" alt="" title="edge003" /></p> <p>And that’s what you’ll find at the bottom of page. The nice thing with this approach is that I can easily tweak the script to produce variations and I don’t have to start from scratch each time.</p> <p>For the record - and so I can find it when I want to tweak it - here’s the processing sketch that produced the final result:</p> <p><em>update</em>: stuck the code in a <a href="">gist on github</a></p> <pre>void setup() { size(2000,50); background(240); smooth(); strokeJoin(ROUND); stroke(255); fill(255); rect(0,0,2000,40); stroke(153); fill(240); beginShape(); vertex(0,42); int x = 0; for (int i=0;i&lt;200;i++) { int y = (int)random(15,30); bezierVertex(x,37, x,30, x+random(-4,4),y); bezierVertex(x+2,30, x+2,37, x+random(8,9),40); x+=10; int w = (int)random(5,15); bezierVertex(x+(w/3),42, x+(2*w/3),42, x+w,40); x += w; } vertex(2000,42); endShape(); save("footerGrass.png"); }</pre> CurrentCost MQTT Bridge 2011-04-04T00:00:00+00:00 <p>Getting the data from a CurrentCost meter on-line has required either an always-on PC or something like the <a href="/2008/11/19/currentcostuino/">Arduino/Ethernet Shield</a> set-up I’ve been using. That was the case until CurrentCost released their <a href="">Bridge</a> device.</p> <p><img src="/blog/content/2011/04/314UBdcphML._SL500_AA300_.jpg" alt="CurrentCost Bridge" title="CurrentCost Bridge" /></p> <p>This little box plugs into the data port of the meter on one side and into your network on the other. It then sends the energy readings over HTTP, so you can then view it on your own <a href="">dashboard</a>.</p> <p><a href=""><img src="/blog/content/2011/04/mycc.jpg" alt="My CurrentCost Dashboard" title="My CurrentCost Dashboard" /></a></p> <p>Under the covers, the Bridge is essentially an Arduino and Ethernet shield relaid onto a single PCB by the good folk at the sadly-no-more <a href="">TinkerLondon</a>. This makes it relatively easy to reprogram for your own nefarious needs - or in my case, getting it to publish the data over MQTT.</p> <p>John Crouchley has <a href="">written</a> about some of the specifics of the board and how it can be reprogrammed with the standard Arduino development environment using a custom USB cable. Not wanting to bother with making a custom cable, I stuck to using my <a href="">USBtinyISP</a> to flash the board directly. Herewith an outline for how to do that (on Linux at least):</p> <ol> <li>Plug the ISP into the 6-pin header on the bridge board. If you have the ISP set to power the board make sure you haven’t got the bridge’s own power supply plugged in.</li> <li>Make sure you’ve followed the instructions at the bottom of the page <a href="">here</a> about udev to ensure you have the appropriate permissions to access the programmer.</li> <li>From the Arduino IDE, burn the bootloader. To do this, first ensure <code class="language-plaintext highlighter-rouge">Tools-&gt;Board-&gt;Arduino Pro or Pro Mini (3.3V, 8 MHz) w/ ATmega328</code> is selected. Then do <code class="language-plaintext highlighter-rouge">Tools-&gt;Burn Bootloader-&gt;w/ USBtinyISP</code>. Of course, if you’re using a different ISP, select the one for you. This takes a little while to complete, but it is worth the wait.</li> <li>Once you have a sketch you want to upload, select <code class="language-plaintext highlighter-rouge">Sketch-&gt;Verify/Compile</code>. This generates the hex file we need in a temporary directory with a name like <code class="language-plaintext highlighter-rouge">/tmp/build3473620529065332403.tmp</code>. The hex file itself will have the same name as your sketch, but with a <code class="language-plaintext highlighter-rouge">.cpp.hex</code> file extension.</li> <li>Use <code class="language-plaintext highlighter-rouge">avrdude</code> to upload the file and you are done.</li> </ol> <pre>$ cd /tmp/build3473620529065332403.tmp/ $ ls \*.cpp.hex MyBridgeCode.cpp.hex $ avrdude -pm328p -cusbtiny -D \ -Uflash:w:MyBridgeCode.cpp.hex ... </pre> <p>As for a sketch to use, the <a href="/2008/11/19/currentcostuino/">basic one</a> I wrote over two years ago needs some considerable updating. It uses absolute positioning in the XML feed to find the data it wants. This makes it fragile to any timing issues in the serial feed; something the CurrentCost can occasionally suffer from. Nor does the sketch handle the multiple sensors and channels that the newer hardware supports.</p> <p>On top of that, there are some differences specific to the Bridge that need to be accounted for. The serial connection from the CurrentCost is connected to digital pins 0 and 1 which means we can use the built-in <code class="language-plaintext highlighter-rouge">Serial</code> object to access it rather than use the <code class="language-plaintext highlighter-rouge">SoftwareSerial</code> library.</p> <p>As John mentions in his post, the Ethernet chip’s reset pin is connected to a digital pin on the Arduino - so something like the following can be used to to cleanly enable it:</p> <pre>pinMode(7,OUTPUT); digitalWrite(7,LOW); delay(250); digitalWrite(7,HIGH); pinMode(7,INPUT); delay(500);</pre> <p>With all that in mind, I’ve put a <a href="">new sketch on Github</a> that does all of this and more. It uses the UUID of the bridge (stored in EEPROM) to generate both the client ID to connect with and the topic to publish to. This lets the same sketch run on multiple bridges without having to recompile with instance-specific settings. Some of the other features of the sketch are:</p> <ul> <li>generates a new 16-byte UUID if it doesn’t find one in EEPROM</li> <li>uses the last 6-bytes of the UUID as the MAC address</li> <li>uses DHCP to obtain a local IP address</li> <li>uses DNS to look-up the server address if <code class="language-plaintext highlighter-rouge">SERVER_HOST_NAME</code> is defined - otherwise expects <code class="language-plaintext highlighter-rouge">SERVER_IP_ADDRESS</code> to define the address to use.</li> <li>reconnects to the MQTT server if the connection is lost</li> <li>publishes readings to the topic structure “<code class="language-plaintext highlighter-rouge">cc/[uuid]/[sensor]/[channel]</code>”</li> <li>if <code class="language-plaintext highlighter-rouge">PUBLISH_CONNECTION_STATE</code> is defined, connection status is maintained on the topic “<code class="language-plaintext highlighter-rouge">cc/[uuid]/s</code>”. A retained message is published with the value ‘1’ when the bridge connects. A Will message is set so that if the bridge loses its connection, a retained message will be published to this topic with the value ‘0’.</li> <li>if <code class="language-plaintext highlighter-rouge">PUBLISH_TEMPERATURE</code> is defined, temperature is published to the topic “<code class="language-plaintext highlighter-rouge">cc/[uuid]/t</code>”. It is published as a retained message and will only republish a temperature when it is 0.5 different to the last one sent</li> </ul> <p>The sketch was developed using arduino-0022, the latest version of my PubSubClient library and the DHCP/DNS libraries from <a href="">Adrian</a> - links to these are in the sketch. Once the next version of Arduino is released, I will refresh the sketch to use that - as the DHCP/DNS libraries ought to be part of the standard distribution by then.</p> <p><img src="/blog/content/2011/04/ccgraph.png" alt="Power Graph" title="Power Graph" /></p> My personal Flickr 2011-01-03T00:00:00+00:00 <p>The recent <a href="">Purple Screen of Death</a> unveiled an uncertain future for two of Yahoo!’s web services that I use; Delicious and Flickr.</p> <p>I wouldn’t call myself a heavy user of Delicious - I tend to go through periods of bookmarking everything one week, to not touching it the next. Despite their story changing from ‘sun-setting’ to ‘selling’, I’m done with Delicious and have moved over to <a href="">Pinboard</a>. But this post isn’t about that.</p> <p><a href="">Flickr</a> is a site I use a lot more. As the PSOD didn’t even mention it, there has been a lot of speculation about its future. My first reaction was that they couldn’t possibly shut the site down; it clearly has a large user base and must generate some amount of money with its Pro accounts. But I think <a href="">Jason Scott</a> sums up the naivety of that view very nicely:</p> <blockquote> <p>I am, frankly, a mixture of disappointed and sad that after Yahoo! shut down Geocities, Briefcase, Content Match, Mash, RSS Advertising, Yahoo! Live, Yahoo! 360, Yahoo! Pets, Yahoo Publisher, Yahoo! Podcasts, Yahoo! Music Store, Yahoo Photos, Yahoo! Design, Yahoo Auctions, Farechase, Yahoo Kickstart, MyWeb, WebJay, Yahoo! Directory France, Yahoo! Directory Spain, Yahoo! Directory Germany, Yahoo! Directory Italy, the enterprise business division, Inktomi, SpotM, Maven Networks, Direct Media Exchange, The All Seeing Eye, Yahoo! Tech, Paid Inclusion, Brickhouse, PayDirect, SearchMonkey, and Yahoo! Go!… there are still people out there going “Well, Yahoo certainly will never shut down Flickr, because <strong>__</strong><strong>__</strong><strong><em>” where __</em></strong>_ is the sound of donkeys.</p> </blockquote> <p>Unlike Delicious, I don’t plan to leave Flickr until there’s a more obvious alternative. Nor am I rushing to retrieve all of my photos from the site as some people are. This is more a result of the fact I have never used Flickr to archive my photos; I have only ever uploaded the small percentage that I have wanted to share. (Although, come to think of it, there are some screenshots and non-photo images on there that I should make sure I have copies of.) But this has meant I’ve needed a separate strategy for backing-up my photos.</p> <p>At the moment, my strategy is to copy them to as many hard-drives as I can. My laptop is the primary store of photos, but this is my work machine, so it clearly isn’t appropriate as a long term option. The Viglen box running my home automation has a meaty disk in it, so that is the first tier of backup. On top of that, I recently bought an external drive which lets me take a backup ‘off-site’. So that makes three copies in my physical reach where I am sat now.</p> <p>Finally, I also keep a copy in a bucket on Amazon’s S3 service. This one is the most out of date because I haven’t automated the uploading to it, but that’s something I’ll be looking at soon. At last count I have just over 9000 photos up there, dating back to 2003, with a backlog of about 4000 waiting to go up.</p> <p>In the early days, I would take lots of photos and wouldn’t delete anything - just in case I ever needed a blurry photo of Venice in 2005.</p> <p><img src="/blog/content/2011/01/blurryVenice.jpg" alt="" title="Blurry Venice" /></p> <p>How could I have possibly illustrated this blog post without that?</p> <p>Needless to say, I’ve come to realise in the last couple of years that I need to be more ruthless and delete the chaff. Particularly as the size of each individual photo has gone from around 1Mb in the early days, up to 6Mb with my shiny new Lumix GF1 - and that’s without having got to grips with RAW yet.</p> <p>With all of those photos on S3, I’ve found myself wanting an easy way to browse through them - partly to help weed out the ones not worth keeping - as well as just making the last eight years of photos available to actually look at. Basically, I want my own personal Flickr.</p> <p>A key thing here is that this wouldn’t be a replacement for how I use Flickr itself - particularly the social side of it. This would be purely for my own consumption - with the ability to share photos with specific people. For example, I have taken plenty of photos of Toby that won’t go onto Flickr, but I want some way of sharing them with his Grandparents and close friends.</p> <p>I know things like <a href="">Gallery</a> exist, which I could probably have up and running in no time. Coincidently, having just gone to the Gallery site, I see a recent <a href="">news item</a> on the ability to host Gallery photos on S3. But regardless, I’m going to have a play with a roll-my-own solution.</p> <p>I don’t know how far I’ll get, but I’ll certainly post updates along the way, all in the spirit of <a href="/2010/12/31/my-plan-for-2011/">my plan for 2011</a>.</p> My plan for 2011 2010-12-31T00:00:00+00:00 <p><a href="" title="My plan for 2011 by knolleary, on Flickr"><img src="" alt="My plan for 2011" /></a></p> Updated Arduino MQTT Client 2010-12-27T00:00:00+00:00 <p>I’ve been a bit remiss in keeping up with new releases of the <a href="">Arduino</a> software. The current Ethernet library has introduced some changes that broke my MQTT library - a fact I only discovered earlier today when playing with the newly released 0022 version. Coincidently, so did <a href="">Andy</a> - who reports it has been broken since 0021.</p> <p>Luckily, it’s a minor fix, which I’ve now made <a href="">available</a>. I’ve also updated the sample sketch to reflect the fact you must manually include <code class="language-plaintext highlighter-rouge">SPI.h</code> if you use the Ethernet library.</p> <p>On a related note, Adrian has released updated versions of his <a href="">DHCP and DNS libraries</a> which will hopefully make their way into the official distribution in the near future.</p> <p><em>Update:</em> <a href="!/andypiper/status/19496687420579840">Andy</a> has also reported that the library works with the <a href="">SparkFun WiFly Shield</a>. Thanks Andy!</p> Power Graphs (again) 2010-12-26T00:00:00+00:00 <p>For the last couple years, I’ve been logging my energy usage and producing daily graphs of the data. Back then, I <a href="/2008/05/05/power-graphing/">wrote</a> about how the Google Charts API could be used to generate nice looking graphs; the result looking like this:</p> <p><img src="/blog/content/2010/12/gc-power.png" alt="" title="Google Charts Power Graph" /></p> <p>But in practise this isn’t what I have been using. Instead, I have been using rrdtool to log the data and produce much less attractive graphs like this:</p> <p><img src="/blog/content/2010/12/rrdtool-power.png" alt="" title="rrdtool Power Graph" /></p> <p>Although rrdtool is very powerful, it suffers from a steep learning curve to figure out how to do just about anything with it; my whole set-up was cribbed from <a href="">Paul’s blog</a>. One advantage of using rrdtool for this type of long-term data collection is its ability to consolidate data - the older the data, the less fine-grained detail is kept.</p> <p><a href="/2009/06/19/166-days-of-power/">Last year</a> I stitched a bunch of the rrdtool-generated images together and produced a video of the graphs. That was mostly a learning exercise in how to do video on Linux, but it did highlight a drawback with rrdtool. As I mentioned in that previous post, the effect of the video is somewhat spoilt by the y-axis jumping around. Ideally, I would regenerate the graphs with a fixed axis to stop this. But rrdtool’s data consolidation makes this impossible - I no longer have the fine-grained data.</p> <p>Whilst I don’t necessarily plan to redo the video, last month I decided to start keeping the data intact for future use.</p> <p>The raw data is now going into a sqlite database so I can do interesting things with it - and worry about how to consolidate the data later on. As I am using an mqtt-powered pub/sub system behind all of this, it was trivial to add a new subscriber to the appropriate topic to grab the data without touching the rrdtool subscriber.</p> <p>After a few days of collecting the data, I started to think about what to do with it. Rather than produce just the current day’s power graph, I wanted to be able to browse the full data. This called for something more interactive than just the static graphs - which gave me enough of an excuse to play with the JavaScript library <a href="">Protovis</a>, which is capable of some very impressive <a href="">results</a>. While it also suffers from a steep-ish learning curve, it benefits from fairly comprehensive documentation and a lot of useful examples to learn from.</p> <p>Here’s a screenshot of the site I’ve produced:</p> <p><img src="/blog/content/2010/12/powergraphs.png" alt="" title="Power Graphs" /></p> <p>The site allows you to browse by year, month and day and also provides an indication of the actual cost of the energy usage. The data is served from a set of static files for each day, rather than driving it straight off the sqlite database. This makes it quite scalable - particularly as only the current day’s data will ever change.</p> <p>I have made a copy of the site available to play with <a href="">here</a>. This isn’t being updated with any new data, for privacy reasons, but does demonstrate the interactivity of the graphs very nicely.</p> <p>There are a few features I’m still thinking about adding, such as being able to compare two days usage and making it easier to identify patterns. I’ll also probably have a couple bugs to fix in 6 days time when I’ll have data spanning two years.</p> SlurpURL 2010-12-23T00:00:00+00:00 <p>I bought my iPad knowing that without owning a machine capable of running iTunes there would be certain things I wouldn’t be able to do. Thanks to friends with such machines, I managed to do the initial activation and recent upgrade to 4.2. But there are plenty of day-to-day tasks that I have foregone.</p> <p>It was never going to be a music-listening device for me, so the current lack of support for music syncing in <a href="">libimobiledevice</a> is not a problem. The library’s support for mounting the iPad as an external drive lets me copy off photos, which means it has become a part of my photo work flow.</p> <p>Putting photos back on the iPad isn’t currently possible - more specifically being able to push them back into the Photos app. Generally, pushing documents into whatever app might be registered to handle them isn’t possible.</p> <p>The <a href="">Dropbox app</a> solves a lot of these getting-files-on-to-the-iPad issues, with the added benefit of ensuring the documents are also available on my laptop and wherever else I have Dropbox running.</p> <p>What the Dropbox app is missing is the ability to add files from the iPad itself; for example, saving a copy of a pdf from Safari. As the app isn’t registered to handle any file types, it doesn’t appear as an option to save files to.</p> <p>To work around this, I’ve built a site that lets me submit a url which is then downloaded, or ‘slurped’, by the server and pushed to my Dropbox account, at which point it is available on the iPad. For those who care, the site was built using django and was a good lesson in doing proper oauth against the Dropbox API. As it currently stands, it looks like this:</p> <p><img src="/blog/content/2010/12/slurpurl.png" alt="slurpurl" title="slurpurl" /></p> <p>I even had the domain name picked - although this time around I didn’t rush out and buy it.</p> <p>You’ll notice I’m not linking to the site; for which there is a good, but unfortunate, reason. In order to connect to the Dropbox API, I got myself an API key. This key is tied to my user name and can only be used when connecting as me. To allow other people to use it, I have to apply for ‘production status’. Currently, Dropbox are only looking at apps that are run natively on mobile devices - they are not approving web apps.</p> <p>It is a pain, but I’m not complaining; if this is how they have choosen to focus their resources, I’d rather they did that then overreach what they can support.</p> <p>Even with my non-production status, I can use it myself when I need to. Hopefully at some point I’ll be able to get proper production status and make it more widely available - although whether this will still be an issue then, I don’t know.</p> Abandoned ideas 2010-12-22T00:00:00+00:00 <p>Having just marked a domain name to expire next month, I realise it has been two years since I had enough of an idea to register it in the first place, but never quite enough to make a site to fit it.</p> <p>It’s all a bit vague, but at the first PaperCamp, back in January 2009, I had a chat with <a href="">Alex</a> about one of the talks we’d just seen. I remember thinking there was something worth building as a result of our chat, but I’ve been trying to remember what triggered our conversation. I went back to <a href="">this</a> write-up from the day and found exactly what it was:</p> <blockquote> <p>… Beeker Northam … speaks about photographing paper. She doesn’t like throwing away books. She photographs her books. There’s something about photographing them that’s different to scanning them. She’d like to have some kind of web-based way for people to share those bits of books that have had an emotional impact on them but she hasn’t found it yet. There are book sharing sites out there but they all take a library-based approach.</p> </blockquote> <p>Rather than simply list what books I’d read, I wanted to be able to record the actual, physical objects - a thought that surfaced in <a href="/2009/03/12/on-books/">this post from a couple months later</a>. A few days after PaperCamp, I registered as the site never to be. I got as far as putting up a holding page and a blurb of text that feels awfully contrived looking back at it now:</p> <p><img src="/blog/content/2010/12/twr_ss1.png" alt="Things We Read" title="Things We Read" /></p> <p><img src="/blog/content/2010/12/twr_ss2.png" alt="Things We Read" title="Things We Read" /></p> <p>The site was basically going to let you list books, or other things, you’d read and have some nifty way to automatically include tagged photos from flickr to help emphasise the physical link. With all the social network trimmings of course.</p> <p>As evidenced by this post, the idea didn’t get any further and I suspect there were two main reasons why.</p> <p>Firstly, it was around this time that I found <a href="">bkkeepr</a>, which has more or less done all I needed in the book-tracking department where others failed.</p> <p>Secondly, as much as it sounds like a trendy post-digital thing to do, in reality I have a bookshelf full of books published in their thousand that hold no special place in the world as individuals. For every <a href="/2009/11/12/the-content-assignment/">Content Assignment</a>, which is an amazing physical object, there are far too many Dan Brown’s or Lee Child’s (yes, I know).</p> <p>So having sat on the digital shelf for two years, I’ve decided there isn’t enough in the idea to renew the domain.</p> Scamming calls 2010-10-27T00:00:00+00:00 <p>A few weeks ago I received a phone call at home that came up as an International number, so I was fully prepared for some sort of advertising. The typical silence after answering such calls was eventually answered by an Indian women confirming she was speaking to Mr O’Leary. Of course, there was the usual stumble as she tried to decide how to pronounce my surname. Eventually she said she was calling to help with the issues I’ve been having with my Windows computer. I pointed out to her that I don’t own a Windows machine, so that was the end of that. It didn’t occur to me until I’d hung up that this was probably a scam and some fun could have been had.</p> <p>Well, tonight I got the chance to play. I’d known for a while that an International number had again been ringing us through the day and Jo had taken to ignoring them. When I answered I tried to sound suitably naive and welcoming of their offer to fix the apparent problems on my (still non-existent) PC.</p> <p>After confirming which the ‘windows’ key on the keyboard is, he asked me to press Win-R and confirm what I could see. Unfortunately it’s been too long since I’ve used Windows so I couldn’t remember what this did. I suspected it was the run dialog (which it is) but I didn’t quite have the confidence to say so. So I launched into saying how, unfortunately for them I’m an engineer, know about computers and that I knew they were trying to scam me. I kept this up for a while, ignoring their denials until the line went dead - they hung up on me.</p> <p>An hour later, we got another International call. I had since confirmed that Win-R brings up the run dialog, so I was looking forward to taking it further. This time however it was quite different. After confirming they were speaking to the right person, I was told I should go kill myself, set fire to my house, dig a grave and lie in it and various other colourful suggestions. Clearly I had upset them. The guy on the phone, also with Indian accent, told me as an engineer I was lowering the intelligence of computer users. Oh the irony. The insults continued for a while, with occasional laughter in the background. To be fair, I was laughing back. Eventually I had had enough so hung up on them.</p> <p>The phone rang seconds later; he clearly hadn’t liked that one bit. He told me off quite clearly for hanging up and the insults and threats continued. I told him I was recording the call so it could be reported and have them shut down… not at all true, but there you go. Eventually I told him I’d had enough and suggested we just leave it there and get on with our lives. He agreed and we hung up together.</p> <p>It was quite a bizarre thing to experience. The saddest part of all is their scam clearly must work often enough for it to be worth doing. I’ll give BT a ring tomorrow to see if anything can be done to block the calls. Their website implies they can’t block international numbers as they are not properly recognised by the system, but that it is also very rare to get nuisance calls from overseas…</p> More Paper Graphs 2010-10-22T00:00:00+00:00 <p>I had to go through old posts here to remind myself when I last did anything on my Paper Graphs project. I was surprised to find it was as long ago as January last year that I wrote about the <a href="/2009/01/17/paper-graphs/">paper pie charts</a> after the first PaperCamp.</p> <p>A couple weeks ago, we reconvened at the Mecca for all good conferences, Conway Hall, for <a href="">PaperCamp2: Fold Harder</a>. Roo’s done a nice write-up over <a href="">here</a> if you want to read more about the event itself.</p> <p>Of the small number of talks during the day, it was the opening one from Matt Brown that chimed the most with me. His <a href="">write-up</a> is worth a read. The Origamic Architecture by <a href="">Gerry Stormer</a> reminded me of the books my parents had by <a href=";search-alias=books&amp;field-author=Masahiro+Chatani&amp;sort=relevancerank">Masahiro Chatani</a> from which I learnt a lot of the principles used in the paper pie charts.</p> <p>At the first PaperCamp, I said I wanted to produce a pop-up book of statistics. This was mostly an attempt to put the hastily crafted pie chart into some sort of context and I didn’t put much more thought into it. It’s still something I think would be fun to do - if I can find the right angle to approach it with. In the meantime, I felt inspired to start playing with other paper graph styles. Specifically, the humble bar chart.</p> <p>Having sketched a few ideas along the lines of Stormer’s work, I went back to the style of the pie graph. A bit more prototyping later and here’s where I got to.</p> <p><a href="" title="Paper Bar Chart by knolleary, on Flickr"><img src="" alt="Paper Bar Chart" /></a> <a href="" title="Paper Bar Chart - collapsed by knolleary, on Flickr"><img src="" alt="Paper Bar Chart - collapsed" /></a></p> <p>As with the pie charts, the pieces for the model are automatically generated and spat out as a pdf. I’ve updated the project page so you can generate your own <a href="">here</a>. It isn’t perfect - particularly if the values have a large range - but I think it works well enough as-is.</p> Updated Arduino MQTT Client 2010-07-20T00:00:00+00:00 <p>There’s an updated version of the Arduino MQTT client <a href="">available</a>. This fixes a bug where if the client lost its connection, you had to explicitly call disconnect before you could reconnect.</p> <p>Not a major issue, but one that had caught a few people out.</p> Joggler Music App 2010-07-08T00:00:00+00:00 <p>As a follow-up to my last post on creating Joggler applications under Linux, I thought I would share what I’ve been doing with it.</p> <p>At some point, I plan to get all of our CD’s onto the Viglen PC that’s sat behind the TV. When plugged into the hi-fi, it’ll make an ideal media box. It does lack one thing however; a nice UI for controlling it. This is where the Joggler can come in - and I’m not the only one to have considered it. I know others have got things like XMBC and Squeezebox running, but I wanted something that integrated better with the existing Joggler apps.</p> <p>The basic idea will be to use <a href="">mpd</a> on the Viglen as a simple music server for which the Joggler will act as an client that sends commands to control the music playback.</p> <p>The app will let us browse the music collection on there, pick an album/track, set up a playlist - all the usual things you’d expect.</p> <p>Here’s a video of the app as it current exists running in the emulator. You can see me browse through the music, add a Coldplay album to the playlist, start playing the list, then removing tracks from the playlist. You’ll note there is no sound on this video… mostly because I’ve not yet done the connection to the server part; all the album/artist data is mocked up so I can develop locally.</p> <object width="500" height="331"><param name="allowfullscreen" value="true" /><param name="allowscriptaccess" value="always" /><param name="movie" value=";;show_title=1&amp;show_byline=0&amp;show_portrait=0&amp;color=00ADEF&amp;fullscreen=1" /><embed src=";;show_title=1&amp;show_byline=0&amp;show_portrait=0&amp;color=00ADEF&amp;fullscreen=1" type="application/x-shockwave-flash" allowfullscreen="true" allowscriptaccess="always" width="500" height="331" />&lt;/embed&gt;</object> <p><a href="">Joggler Music App</a> from <a href="">Nicholas O'Leary</a> on <a href="">Vimeo</a>.</p> <p>Clearly there’s lots still to do to make it properly usable - but it has definitely been a useful way to learn more about ActionScript.</p> Creating Joggler applications under Linux 2010-05-31T00:00:00+00:00 <p>I recently got an O2 Joggler as a cheap touch-screen device that is easily hacked to become a generally useful device. To run Ubuntu on it is as simple as plugging in a suitably installed USB drive. I haven’t decided what I want to do with the device yet, but I wanted to see what could be done with the original software.</p> <p><img src="/blog/content/2010/05/o2-joggler-tablet-300x186.jpg" alt="" title="o2-joggler-tablet" /></p> <p>Under the covers, it already runs Linux with the entire interface built in Flash. This makes it a challenge to develop for - Linux certainly isn’t overrun with Flash development tools. After some trial and error, I have figured out the basics. So herewith a beginners guide to developing Joggler applications under Linux.</p> <p>Note, this is not intended to be a Flash or ActionScript tutorial - I am not an expert. Everything you see here I have worked out from scratch this week.</p> <h4 id="installing-dependencies">Installing dependencies</h4> <ol> <li>Install <a href="">swftools</a>.</li> </ol> <pre>$ sudo apt-get install swftools</pre> <p>This provides a number of useful tools for working with flash swf files. The key one for our purpose is <code class="language-plaintext highlighter-rouge">as3compile</code> - an ActionScript 3.0 compiler.</p> <ol> <li> <p>Install a standalone flash player. I use <a href="">Adobe’s own</a> - you can try one of the open-source alternatives if you want.</p> </li> <li> <p>Register for, and download the SDK from <a href="">OpenPeak</a>. This includes a complete version of the framework that lets you debug and run on your laptop, rather than have to copy over to the Joggler each time. The docs pdf included with the SDK explains how to do this in the ‘Testing Applications’ section.</p> </li> </ol> <pre>$ cd SDK/published/ SDK/published$ flashplayer openframe.swf</pre> <p><img src="/blog/content/2010/05/Screenshot-Adobe-Flash-Player-101-300x210.png" alt="" title="Screenshot-Adobe Flash Player 10" /></p> <h4 id="creating-a-new-application">Creating a new application</h4> <ol> <li>Create a new application directory under the SDK directory.</li> </ol> <pre>$ mkdir SDK/published/apps/DemoApp</pre> <ol> <li>Add an entry for the application to <code class="language-plaintext highlighter-rouge">applications.xml</code> file - this tells the framework where to find the various pieces.</li> </ol> <pre><app static="1" id="test" loc="./apps/DemoApp/" icon="icon.swf" app="main.swf" /></pre> <ol> <li>In the application directory, create a <code class="language-plaintext highlighter-rouge">language.xml</code> file. This contains all of the translatable text of your application. Even if you don’t plan to translate your application, you must still create this file with an entry for the name to display beneath the icon on the main menu.</li> </ol> <pre><copy> <mm en="Demo App" /> </copy></pre> <h4 id="creating-iconswf">Creating icon.swf</h4> <p>The documentation specifies what the icon movie must consist of - a 100x100 movie with a button. After some experimentation, I’ve got a template ActionScript file that does the job. Copy the following into a file called <code class="language-plaintext highlighter-rouge"></code> within the application directory.</p> <pre> package { import flash.text.TextField ; import flash.text.TextFormat ; import flash.display.MovieClip; import flash.display.SimpleButton; import flash.display.Shape; import flash.filters.ColorMatrixFilter; public dynamic class DemoIcon extends MovieClip { public function DemoIcon() { // Create a button var button:SimpleButton = new SimpleButton(); // Setup the various states of the button button.upState = new DemoIconButton(); button.downState = new DemoIconButton(); button.overState = button.upState; button.hitTestState = button.downState; // Add a fade to the down state of the button button.downState.filters = [( new ColorMatrixFilter([1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0.7, 0]))]; // Add the button to the stage button.x = 10; button.y = 10; addChild(button); } } class DemoIconButton extends Shape { public function DemoIconButton() { // Draw the button graphics.beginFill(0x996633); graphics.lineStyle(2,0xffffff); graphics.drawRoundRect(0, 0, 80, 80,20,20); graphics.endFill(); } } } </pre> <p>Next, compile it to the required swf file:</p> <pre>SDK/published/apps/DemoApp$ as3compile</pre> <p>If you launch the openframe now, you should see the icon in place.</p> <p><img src="/blog/content/2010/05/icon-e1275258354260.png" alt="" title="icon" /></p> <p>As you can see, it isn’t the most exciting icon. But it does have a nice fade effect when pressed. To change what it looks like, you can play around with the <code class="language-plaintext highlighter-rouge">DemoIconButton()</code> method - with a little help from the reference on <a href=""></a>.</p> <h4 id="creating-mainswf">Creating main.swf</h4> <p>I’m still working out how best to approach writing the main application. There are some parts in the documentation that simply don’t work for me - this may well be a result of not using the official Flash development tools.</p> <p>The docs say the main app needs to include a class that extends <code class="language-plaintext highlighter-rouge">op.framework.OpenFrameApplication</code>. I found importing this class, even without using it, made the app fail to launch. Not a great start. However, it seems to work without using this class.</p> <p>Copy the following into a file called <code class="language-plaintext highlighter-rouge"></code> within the application directory.</p> <pre> package { import flash.display.MovieClip; import flash.display.*; import flash.text.*; public class DemoMain extends MovieClip { function DemoMain() { graphics.beginFill(0x000000); graphics.drawRect(0, 0, 800, 410); graphics.endFill(); var textField = new TextField(); var textFormat = new TextFormat(); textFormat.font = "Arial"; textFormat.size = 30; textFormat.color = 0xffffff; textFormat.bold = true; textField.defaultTextFormat = textFormat; textField.text = "Hello World!"; textField.autoSize = TextFieldAutoSize.LEFT; var tm = textField.getLineMetrics(0); textField.x = 400-tm.width/2; textField.y = 200-tm.height/2; addChild(textField); } } } </pre> <p>As before, compile it to the required swf file:</p> <pre>SDK/published/apps/DemoApp$ as3compile</pre> <p><img src="/blog/content/2010/05/mainappscreenshot-300x208.png" alt="" title="mainappscreenshot" /></p> <h4 id="using-the-op-libraries">Using the <code class="language-plaintext highlighter-rouge">op.*</code> libraries</h4> <p>The framework includes a number of actionscript libraries to provide integration with the platform. They are provided as uncompiled actionscript files under the <code class="language-plaintext highlighter-rouge">SDK/op</code> directory. To compile against them, <code class="language-plaintext highlighter-rouge">as3compile</code> needs to know where they are. Assuming you’re still developing in the application directory created earlier, then the following does the job:</p> <pre>SDK/published/apps/DemoApp$ as3compile -I../../../</pre> <p>I have had mixed success using these libraries - here are some of my findings that may help you.. or not.</p> <dl> <dt>`op.framework.OpenFrameApplication`</dt> <dd>As mentioned previously, simply importing this class is enough to stop things working.</dd> <dt>`op.framework.OPLang`</dt> <dd>This class provides access to the entries in `language.xml` file. When included, I get the following compile error for which I have yet to find a workaround: <pre>op/framework/ error: can't convert type String to XMLList</pre> This currently rules out using this class. </dd> <dt>`op.framework.OPLink`</dt> <dd>This class provides the main link between the application and the underlying framework. When included, I get the following compile error: <pre>op/framework/ error: syntax error, unexpected *=</pre> This is easily fixed by adding a space between the `*` and `=` in two lines of the library: <pre>private static var apploc:* = null; private static var so:* = null;</pre> </dd> <dt>_Anything_ that imports `fl.*`</dt> <dd>It appears that whilst `as3compile` can handle references to the core `flash.*` packages, it knows nothing of the `fl.*` ones. Haven't found a solution to this yet.</dd> </dl> <h4 id="running-on-the-joggler">Running on the Joggler</h4> <p>You’ll need a means to copy over your newly developed application to the Joggler. Start by installing <a href="" title="Installing_Telnet">telnet</a>, then <a href="" title="Ssh">ssh</a> and finally <a href="" title="Scp">scp</a>.</p> <p>The install process is the same as I described before; creating a directory for the application, followed by adding an entry to <code class="language-plaintext highlighter-rouge">applications.xml</code>. This all goes under <code class="language-plaintext highlighter-rouge">/media/appshop</code>. Once in place, you need to restart the software to pick up the new app:</p> <pre>$ killall tango</pre> <h4 id="conclusion">Conclusion</h4> <p>As you can see, it isn’t the smoothest experience. But with some perseverance, the end results ought to be worth it.</p> How I got onto prime-time BBC One 2010-04-22T00:00:00+00:00 <p>It all started, as things do, with a tweet.</p> <p>As part of the Emerging Technologies group at IBM Hursley, <a href="">Kevin</a> gets to play with new technologies to see how they might be useful to IBM’s customers. One such item is the <a href="">Emotiv</a> headset (an electroencephalograph if you must), which can read signals in the brain. You can train it to recognise particular thoughts which has some very interesting applications from gaming to rehabilitative care. You can find out more about the headset in this piece from <a href="">The Times</a>. But I digress.</p> <p>The BBC were interested in finding out more about the headset and what sort of thing IBM had been doing with it. Knowing they were interested to see if a car could be controlled by the headset, Kevin was looking for something to make the demo more relevant, which led to his <a href="">tweet</a>.</p> <p><a href=""><img src="/blog/content/2010/04/kxb_twitter.png" alt="" title="The twitter that started it" /></a></p> <p>With only a couple days to put something together, I suggested we go down the route of wiring up an existing radio controlled car to an Arduino. Kevin already had the headset hooked up to MQTT, so it would be trivial to use my arduino MQTT library to get them all talking.</p> <p>A quick trip to Asda and I was the proud owner of a £9 blue Mini Cooper car, which I attacked with my soldering iron. It didn’t take much to get it working - I’ll blog the finer details of that bit soon.</p> <p>The demo went well and we discussed more about what they wanted to do for the programme itself. Some of their ideas were ambitious to say the least. Someone mentioned the idea of driving a bus through Whitehall… not sure how serious that was. But ultimately, a straight race between two taxis ‘driven’ by the two presenters was decided on.</p> <p>A couple weeks later, they were back in Hursley to film with <a href="">Jem</a> and <a href="">Dallas</a>. Now, there are some things that are best not left to the last minute. Such as realising they needed <em>two</em> radio controlled cars for filming - when I only had one. Luckily this dawned on me the day before they came down so I returned to Asda and got a shiny red sports car that would look good alongside the mini. I then discovered one of the reasons they were so cheap is that both worked on the same frequency… one remote drove both cars. With time running out, I went back and got a gaudy yellow jeep that was a completely different make and thankfully worked on a different frequency.</p> <p><a href="" title="The cars by knolleary, on Flickr"><img src="" alt="The cars" /></a></p> <p>A couple of weeks later, Kevin and I headed up to a barn in middle-of-nowhere-Northamptonshire where Jem had been working on the taxis. Now, a few people have said to me “yeah, but he doesn’t really do the work does he?”, to which I have to reply that he very much does; Jem really knows what he is talking about when it comes to building things and the enthusiasm he portrays on screen is just what he’s like in real life.</p> <p><a href="" title="Jem Stansfield by knolleary, on Flickr"><img src="" alt="Jem Stansfield" /></a></p> <p>Over the course of two freezing days, we got the radio units hooked up to MQTT, again via an arduino. This was probably the piece I was most worried about - it was one thing to hack a toy remote control but it was going to be quite another to do the same to an industrial radio set that cost considerably more. Not to mention the fact that they were also on loan for the project, so breaking them would have not made me any friends.</p> <p><a href="" title="In the workshop by knolleary, on Flickr"><img src="" alt="In the workshop" /></a></p> <p>We filmed the first test run and the relief was palpable when the car lurched forward thanks to Jem’s brain - not to mention the reaction when he managed to brake within a few inches of an oil drum. Although none of that made it into the final programme.</p> <p><a href="" title="Mission Control by knolleary, on Flickr"><img src="" alt="Mission Control" /></a></p> <p>And then we had the main event - the race itself at the <a href="">Santa Pod Raceway</a>. 8am on a freezing December morning is not the best time to be trying to wire up the last few connections and try to debug why the damn thing wasn’t working. But somehow we got there and eventually the taxis did what they were thought to do - even if one did plow into the crash barrier at some considerable speed.</p> <p><a href="" title="Dallas &amp; Jem by knolleary, on Flickr"><img src="" alt="Dallas &amp; Jem" /></a></p> <p>The plan had been to do two races; a straight race and an obstacle course. Technical hiccups along the way meant it wasn’t until after lunch that we got the straight race filmed, at which point we were running out of light. It was decided to put Dallas in the back of a taxi and have Jem drive him around. This was the first proper test of steering by mind-control. Let’s just say I wouldn’t enter into a slalom race any time soon.</p> <p>With all the filming done we packed up and headed home. Almost 5 months later, we got to see the end result on TV. Having spent the best part of 4 days filming, I was fascinated to see how they would edit it down to the 10 minutes or so they had to fill. I have to say I’m really please with the result. They may have given Kevin the speaking part out of the two of us, but I think I got more close ups. Given the target audience, I’m also not that surprised that they didn’t dwell on the finer details of the technology.</p> <p>That said, I’m a proud geek that managed to get both my Ubuntu lanyard and an Arduino onto prime-time BBC One.</p> <p><a href="" title="Arduino on BBC1 by knolleary, on Flickr"><img src="" alt="Arduino on BBC1" /></a></p> <p><a href="" title="Me by knolleary, on Flickr"><img src="" alt="Me" /></a></p> <p><em>Update</em>: you can see the bits of the programme that featured the taxis <a href="">here</a>.</p> I'm gonna be on the telly Mum! 2010-04-16T00:00:00+00:00 <p><img src="/blog/content/2010/04/camera.jpg" alt="" title="camera" /></p> <p>After what was a <a href="">quite stressful</a>, <a href="">cold</a>, but ultimately highly enjoyable few days at the end of last year, I’m going to be on TV.</p> <blockquote> <p>[<img src="/blog/content/2010/04/banggoesthetheory.jpg" alt="" title="banggoesthetheory" /></p> <p><strong>Can You Train Your Brain?</strong>](</p> <p>Wednesday, 21st April, 21:00 on BBC One</p> <p>A Bang Goes the Theory special to help you improve your brain. The science team gives the results of the world’s biggest ever brain training experiment and reveals how you can make yourself smarter.</p> <p>…</p> <p>Resident extreme engineer, Jem, taps into his brain to drive a car with the power of thought alone.</p> <p>…</p> </blockquote> <p>I’ll save the full story until after it airs so I don’t spoil anything - but it should be good viewing.</p> How I almost got into Wired UK 2010-03-04T00:00:00+00:00 <p><img src="/blog/content/2010/03/IMG_9707-Modified.jpg" alt="" title="Photo Studio" /></p> <p>Back in <a href="/2008/11/30/homecamp-08/">November 2008</a>, I spoke at HomeCamp about the Current Cost stuff we were doing as well as about the ambient orb I had made.</p> <p>A few days later, on my birthday in fact, I got an email from a freelance writer who was putting together a short piece for the then soon to be relaunched Wired UK. He was writing about HomeCamp, Current Cost and all those sorts of things and wanted to feature my ambient orb. We had a chat on the phone and I gave him a brief run down of the orb, what it was, how it could be used for energy monitoring. We left it at that and I waited to see what happened.</p> <p>A couple weeks later, I got a phone call from someone at Wired. Given the visual appeal of the orb they wanted to send around a photographer to help illustrate the article. Slightly bemused by it all, we organised a date and time for the photographer to come to my house and do his thing.</p> <p>It was a weekday in late January 2009. I slipped out of work after lunch mentioning something about an appointment and headed home. I really wasn’t sure what to expect. I was picturing a guy would turn up with a camera, spend 10 minutes taking a couple photos and be off to his next job - the life of a freelance jobbing photographer.</p> <p>When he arrived we had a quick chat about the orb, talked about various ideas for how to shoot it, and I gave him a tour of the house looking at potential locations. At this point, he went back to his car and started to unload his equipment and I got a lesson in professional photography.</p> <p>There were at least two cameras and multiple lenses, two spotlights on tripods and a another pair of tripods between which a huge roll of paper was hung as a backdrop. This wasn’t going to be a brief visit and sure enough, my living room was turned into a makeshift studio for the next 2 hours.</p> <p>And it wasn’t limited to that - we moved up to the spare room with my desk as well as half an hour spent in the kitchen. The search for the perfect shot was relentless. Ultimately it would be up to the Wired UK guys, but he wanted to make sure they had lots of options.</p> <p>In an attempt at small talk, I asked if he did this sort of thing often. He said that being based in Brighton (yes, he had driven over from Brighton for this - I did wonder at that point why Wired hadn’t hired a more local photographer…) he tended to do music based shoots. I’m glad I didn’t follow that up with a ‘anyone I’d know?’ type question…</p> <p>After he left, I decided to google him, Alex Lake, and I found his site <a href="">Twoshortdays</a>. It was about this point I suddenly flashed through my head everything I had said to him to make sure I hadn’t been a tit. Go and have a look. You see, his other jobs have included some great portraits of people like Amy Winehouse, Guy Garvey, Boy George and Bob f’ing Geldof. There are dozens of portraits of very famous people on there - and he was in my living room, helping me move my sofa so we could take some photos of my little ambient orb.</p> <p>Did I mention Bob f’ing Geldof?</p> <p>On top of that, he’s also an illustrator. The odd little doodle? No. He designed a number of Keane’s albums - including <a href="">Hopes and Fears</a>.</p> <p>Did I mention I asked him “so, are you a freelance photographer then? Or is this just something you do on the side?”</p> <p>Anyway.</p> <p>It would appear the article got spiked in the end - it still hasn’t surfaced in the magazine and I can’t believe they would have held on to it for this long. I did email Alex late last year in case he had any photos from that day without any reply.</p> <p>Bob f’ing Geldof.</p> <p>And that is how I almost got into Wired UK.</p> Somerset birds 2009-12-27T00:00:00+00:00 <p>One walk on the <a href="">Somerset Levels</a>, two extremes of bird watching.</p> <p>A single, solitary Kestrel, stalking its prey.</p> <p><a data-flickr-embed="true" href="" title="Kestrel"><img src="" width="320" height="240" alt="Kestrel" /></a><script async="" src="//" charset="utf-8"></script></p> <p>A thousand Starlings, coming in to roost.</p> <p><a data-flickr-embed="true" href="" title="A Thousand Starlings"><img src="" width="320" height="240" alt="A Thousand Starlings" /></a><script async="" src="//" charset="utf-8"></script></p> <p><a data-flickr-embed="true" href="" title="A Thousand Starlings"><img src="" width="320" height="240" alt="A Thousand Starlings" /></a><script async="" src="//" charset="utf-8"></script></p> Multi-Channel Ambient Orb 2009-12-14T00:00:00+00:00 <p>I’ve had my <a href="/2008/11/25/diy-ambient-orb-redux/">orb</a> sat beside my TV for over a year now and it has served its purpose very well. I’ve never got beyond using it to display my energy usage - or more specifically, to display when my energy usage is above ‘normal’. This has always felt a bit of waste; only using 2 colours out of the entire spectrum.</p> <p>Over tea with Andy a few weeks ago, we managed to place our collective fingers on a basic problem with ambient orbs like this; whilst they may be capable of displaying any colour, the key thing is they can only display one colour at a time - they are a single channel of information.</p> <p>For example, if I used blue to signify new messages for me on twitter, what should the orb do when my energy goes over 500 watts and someone has @knolleary’d me? The orb could alternate between the two colours, but that would feel too distracting for what is supposed to be an ambient device. A third colour could be defined for this combined state, but that wouldn’t scale very well.</p> <p>This train of thought brought us to identify what it would take to have an ambient orb capable of displaying more than one piece of information at any time. When put like this, the answer is fairly obvious; have an orb that can glow more than one colour at any time. An evening of soldering later, here’s where I got to.</p> <div style="text-align: center;"><object type="application/x-shockwave-flash" width="400" height="300" data="" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"> <param name="flashvars" value="intl_lang=en-us&amp;photo_secret=669ba69039&amp;photo_id=4185280955" />&lt;/param&gt; <param name="movie" value="" />&lt;/param&gt; <param name="bgcolor" value="#000000" />&lt;/param&gt; <param name="allowFullScreen" value="true" />&lt;/param&gt;<embed type="application/x-shockwave-flash" src="" bgcolor="#000000" allowfullscreen="true" flashvars="intl_lang=en-us&amp;photo_secret=669ba69039&amp;photo_id=4185280955" height="300" width="400" />&lt;/embed&gt;</object> [![Multi-Channel Ambient Orb](]( "Multi-Channel Ambient Orb by knolleary, on Flickr")</div> <p>The orb has three RGB leds in it that are individually controllable. When they all show the same colour, the orb is a solid colour, but when they are different, the orb displays multiple colours at once.</p> <p>This one doesn’t use blinkm’s for the simple matter of cost - I found a supply of the leds on e-bay that got me 50 for £15 - although this does mean I need to implement fading between the colours.</p> <p>The plan is to put a small controller in the base to drive all the leds - for now it’s plugged straight into an arduino.</p> The Content Assignment 2009-11-12T00:00:00+00:00 <p>I wish I knew the full history of this book; I don’t mean the story the book contains, rather the actual physical entity that is this piece of awesome. The story in the book is a rather mundane spy ‘thriller’ (I use that word with caution), set in a cold-war era with plenty of ‘intrigue’ (again, with caution). Written in the first-person, it follows British reporter John Terrant as he tries to track down the mysterious Ellen Content with whom he had a brief affair whilst stationed in Berlin. Published in 1954, the book is definitely a product of its age - but more on that later.</p> <p>I think it’s only fair to put in a <strong>spoiler alert</strong> at this point; I can’t guarantee that the rest of this post won’t reveal details of the (paper-thin) plot.</p> <p><a href="" title="The Content Assignment by knolleary, on Flickr"><img src="" alt="The Content Assignment" /></a></p> <p>Back to this actual book I’m holding in my hands - well, I was before I started typing. All I know is that it once belonged to my Gran and recently reached me via my parents. Before that, it belonged to one “Gioladys O Williams”. How do I know this? Well she kindly wrote her name in it. Now, I consider myself an expert in reading English having had 28 years of practice, but that first name certainly has me puzzled - “Gladys” maybe, but “Gioladys”?</p> <p><em>Update</em>: <a href="/2009/11/12/the-content-assignment/#comment-12451">Adrian</a> kindly points out below that it’s “Gwladys”. Considering the fact “Gwladys” wins in a <a href=";word1=gioladys&amp;word2=gwladys">google fight</a> with “Gioladys”, I suspect he’s right. In fact, now he’s pointed that out, I can’t read it as anything other than “Gwladys”.</p> <p><a href="" title="Inner Cover by knolleary, on Flickr"><img src="" alt="Inner Cover" /></a></p> <p>The book starts out telling how John, our first-person-protagonist (FPP) spots an announcement in a paper that a Miss Ellen Content is returning by sea to New York. For reasons as yet unexplained, he becomes determined to get to New York ahead of her. He convinces his editor to fund it as a writing assignment - the Content Assignment. It has taken all of four pages for the title of the book to be fully explained. Magic.</p> <p>Straight into chapter 2, where the fun starts. Whilst flying, John thinks back to how he first met Ellen Content. Taking us back to Berlin 1948, the scene is set in a city whose “de-nazification was almost complete in the British and America zones, proceeding slowly in the French zone and almost not at all in the Russian zone.” This is the first indication of when this book was written; clearly the Russians are in for a rough ride.</p> <p>This brings me to why I’ve bothered to write about an otherwise forgettable book; its former owner, Ms Williams, didn’t stop at just writing her name inside the cover. Oh no - she did much more.</p> <p><a href="" title="pg 12 by knolleary, on Flickr"><img src="" alt="pg 12" /></a></p> <p>Clearly she has taken exception to the tone of the book, but what amazes me is that she kept going…</p> <p>At first I thought it was the negative portrayal of the Russians that had upset her, but on the very next page she shows her disdain isn’t limited to that.</p> <p><a href="" title="pg 13 by knolleary, on Flickr"><img src="" alt="pg 13" /></a></p> <p><a href="" title="pg 22 by knolleary, on Flickr"><img src="" alt="pg 22" /></a></p> <p>Bearing in mind this book is written in the first-person, it’s hard to tell who these comments were directed at; the author or John The FPP.</p> <p>Up to this point, the comments have all referred to specific bits of text. It takes until page 38 for the dam to break and Ms Williams to write what she really thinks.</p> <p><a href="" title="pg 38 by knolleary, on Flickr"><img src="" alt="pg 38" /></a></p> <p>I bet she felt better for that - although it isn’t clear from her continuing stream of comments. One noticeable change is that no-one is safe from her pen. First up, are Britain’s security services and police.</p> <p><a href="" title="pg 44 by knolleary, on Flickr"><img src="" alt="pg 44" /></a></p> <p><a href="" title="pg 51 by knolleary, on Flickr"><img src="" alt="pg 51" /></a></p> <p>You know she means business; she’s signed her name.</p> <p>In a shocking turn of events, she goes on to agree with something in the book.</p> <p><a href="" title="pg 93 by knolleary, on Flickr"><img src="" alt="pg 93" /></a></p> <p>But she doesn’t let that deter her. New York building regulations are a surprise entry in her hit list.</p> <p><a href="" title="pg 95 by knolleary, on Flickr"><img src="" alt="pg 95" /></a></p> <p>She must have tired at this point, as her next comment is a single word which isn’t entirely obviously what her point is. Perhaps the irony of John The FPP feeling the part of something strange; the book itself.</p> <p><a href="" title="pg 103 by knolleary, on Flickr"><img src="" alt="pg 103" /></a></p> <p>Again Ms Williams can’t quite decide who she is venting at - insulting the poorly written character or the author of said poorly written character.</p> <p><a href="" title="pg 107 by knolleary, on Flickr"><img src="" alt="pg 107" /></a></p> <p>Towards the end of the book, she returns to her favourite subject</p> <p><a href="" title="pg 138 by knolleary, on Flickr"><img src="" alt="pg 138" /></a></p> <p>and again makes a dig at the quality of the writing.</p> <p><a href="" title="pg 148 by knolleary, on Flickr"><img src="" alt="pg 148" /></a></p> <p>In the final pages, she decides to stop holding back and say what she really thinks.</p> <p><a href="" title="pg 149 by knolleary, on Flickr"><img src="" alt="pg 149" /></a></p> <p>I hope they get the message - Ms Williams <em>does not like it</em>. I almost wish she left it there, but she ends with this.</p> <p><a href="" title="pg 156 by knolleary, on Flickr"><img src="" alt="pg 156" /></a></p> <p>I can hear her manically laughing as she put down her pen, closed the book and felt incredible pleased with having set the world to rights.</p> <p>This is what people did before Have Your Say existed.</p> <p>Wow. Simply wow.</p> Playful '09 2009-11-04T00:00:00+00:00 <p><a href="" title="Playful by knolleary, on Flickr"><img src="" alt="Playful" /></a></p> <p>Roo has done a <a href="">write-up</a> of the day which is much better than this one.</p> <p>I took a bunch of photos and managed to find a couple that were worth <a href="">sharing</a>. If you want any blurry photos of the speakers, let me know - I’ve got plenty.</p> New Zealand 2009-09-05T00:00:00+00:00 <p>New Zealand was awesome. It was everything we hoped it would be and more. Our first few days on North Island were overcast and occasionally soggy - but nothing we weren’t ready for considering it was the tail-end of winter out there. The rest of our time on South Island was clear blue skies all the way.</p> <div style="text-align: center;"><object width="400" height="300"> <param name="flashvars" value="offsite=true&amp;lang=en-us&amp;page_show_url=%2Fphotos%2Fknolleary%2Fsets%2F72157622195918130%2Fshow%2F&amp;page_show_back_url=%2Fphotos%2Fknolleary%2Fsets%2F72157622195918130%2F&amp;set_id=72157622195918130&amp;jump_to=" />&lt;/param&gt; <param name="movie" value="" />&lt;/param&gt; <param name="allowFullScreen" value="true" />&lt;/param&gt;<embed type="application/x-shockwave-flash" src="" allowfullscreen="true" flashvars="offsite=true&amp;lang=en-us&amp;page_show_url=%2Fphotos%2Fknolleary%2Fsets%2F72157622195918130%2Fshow%2F&amp;page_show_back_url=%2Fphotos%2Fknolleary%2Fsets%2F72157622195918130%2F&amp;set_id=72157622195918130&amp;jump_to=" width="400" height="300" />&lt;/embed&gt;</object></div> <ul> <li>Our <a href="">camper-van</a>, Marsha</li> <li>Visiting <a href="">Hobbiton</a></li> <li>The <a href="">Lady Knox</a> geyser</li> <li>The <a href="">Pancake Rocks</a> at Punakaiki</li> <li>Flying onto <a href="">Fox Glacier</a> - probably the highlight for me</li> <li>The <a href="">waterfalls</a> of Milford Sound</li> <li>Getting up-close to a <a href="">Sea Lion</a></li> <li>The <a href="">Moeraki boulders</a></li> <li>Watching <a href="">whales</a></li> </ul> Twitterlogue 2009-08-07T00:00:00+00:00 <p>For the next few weeks we’re going to be driving across New Zealand in a camper van. With Vodafone having suspended roaming charges abroad for the summer, I realised last week I would be able to freely twitter as we go. Whilst that in itself could end up being an good record of the trip, I couldn’t help feel it would be missing something.</p> <p>The tweets would help record what we were doing, but they would lose the where. So, after a few hours hacking, I have created a real-time <a href="">twitterlogue</a> of the trip.</p> <p><a href=""><img src="/blog/content/2009/08/aukland.png" alt="Twitterlogue Screenshot" title="Twitterlogue Screenshot" /></a></p> <p>This is a simple mash-up of a twitter feed along with a google map of where we were when I sent the update. As we’re driving, it also records how far we’ve driven at that point.</p> <p>Under the covers it works like this… I send a direct message to my sekret twitter account using the format:</p> <pre>[lat,long] [distance] [message]</pre> <p>When the dm is received, the information is injected into a database which feeds the page. It also sends the update, sans GPS/distance, to my <a href="">twitter</a> account, including a link to the update on the map. As each update has its own permalink on the map they are individually linkable. Awesome.</p> <p><img src="/blog/content/2009/08/trip-screenshot.jpg" alt="trip-screenshot" title="trip-screenshot" /></p> <p>To make life easier on the road, and because I can, I’ve put together a little python app for my phone that gathers the GPS location and lets me enter the other information before sending it off to twitter.</p> <p>Had I more time, it would have been cool to pull in photos I’ve taken around the time of the update. But given I’m not sure how often I’ll be uploading photos whilst out there, I thought it better to leave it for now.</p> <p>I’m not aware of any existing service that does quite this sort of travelogue thing - I wonder if there’s something more generic that could be spawned from this.</p> <p>It’s had some simple testing but it could well break whilst we’re away and there’ll be little I can do about it.</p> <p>(sorry, no bonus points for working out where I was when I took the screenshot of the phone app)</p> Updated Arduino MQTT client 2009-07-24T00:00:00+00:00 <p>Just a quick one to say I’ve made an update to the Arduino MQTT PubSubClient to fix a compile error with the latest version of the arduino software (0016).</p> <p>As ever, get the updated MQTT Arduino client from <a href="">here</a>.</p> 166 days of power 2009-06-19T00:00:00+00:00 <iframe src="" width="640" height="480" frameborder="0" webkitallowfullscreen="" mozallowfullscreen="" allowfullscreen=""></iframe> <p><a href="">166 days of power</a> from <a href="">Nick O&#039;Leary</a> on <a href="">Vimeo</a>.</p> <p>I’m still finding my feet with video on linux - it involves too many command line tools and too much documentation reading at the moment. This relatively simple video has taken much more effort than I was expecting.</p> <p>For a first attempt, it’ll do. It’s a shame the scale of the graph jumps around - it does detract from the effect slightly.</p> Cornwall 2009-05-25T00:00:00+00:00 <p><a href="" title="Oakhill B&amp;B by nol, on Flickr"><img src="" alt="Oakhill B&amp;B" /></a> <a href="" title="Untitled by nol, on Flickr"><img src="" alt="" /></a> <a href="" title="Porthcurno by nol, on Flickr"><img src="" alt="Porthcurno" /></a> <a href="" title="Cheesewring by nol, on Flickr"><img src="" alt="Cheesewring" /></a> <a href="" title="Wind Swept by nol, on Flickr"><img src="" alt="Wind Swept" /></a></p> <p>May Bank Holiday in Cornwall</p> Recent Gardens 2009-05-11T00:00:00+00:00 <p><a href="" title="Untitled by nol, on Flickr"><img src="" alt="" /></a> <a href="" title="Butterfly by nol, on Flickr"><img src="" alt="Butterfly" /></a> <a href="" title="Untitled by nol, on Flickr"><img src="" alt="" /></a> <a href="" title="Untitled by nol, on Flickr"><img src="" alt="" /></a></p> <p><a href="">Exbury Gardens</a> in the spring. <em>now with added <a href="">infomatics</a>.</em></p> <p><a href="" title="Pagoda by nol, on Flickr"><img src="" alt="Pagoda" /></a> <a href="" title="Walking in the Trees by nol, on Flickr"><img src="" alt="Walking in the Trees" /></a> <a href="" title="Kew by nol, on Flickr"><img src="" alt="Kew" /></a> <a href="" title="Temperate House by nol, on Flickr"><img src="" alt="Temperate House" /></a></p> <p><a href="">Kew Gardens</a> - 250th anniversary</p> Transfering files via Bluetooth from Linux to Nokia 5800 2009-05-01T00:00:00+00:00 <p>Yes that’s right - time for another google friendly post.</p> <p>I’ve just got a <a href="">Nokia 5800</a> which I am loving. The fact it’s my first new phone for 4 years means I’ve got 4 years worth of technology to catch up on. The fact the first thing I did with it was to install <a href="">python</a> says a lot about me and what I plan to do with it.</p> <p>Running Linux means some compromises such as not being able to run the Noika PC Suite for managing files on the phone, nor using the emulator that comes with the SDK. Some people have had varying degrees of success in running these things under Wine or VirtualBox, but I haven’t found a solid set of instructions.</p> <p>Putting the emulator question aside for the time being, I wanted to find out how to most easily transfer python files from the laptop to the phone. There are two main options:</p> <ol> <li>USB - the phone shows up as a regular storage device and just works. This is best for large transfers such as syncing music collections etc. It also gives you most freedom as to where the files end up on the phone. It isn’t ideal for the edit-run-debug-edit-run-debug cycle that typifies code hacking as you have to mount/unmount each time and it’s going to put unnecessary wear on the cable/ports.</li> <li>Bluetooth - using Gnome’s bluetooth applet, as is installed with Ubuntu etc, just works… most of the time. Files sent to the phone show up as messages in your inbox and the phone will do the Right Thing depending on the file type; mp3’s are added to your music store and .sis install packages are installed. It doesn’t work for sending python files over however. The phone treats them as text files and adds them to the Notes application. This is annoying when they need to be copied to a particular location on the memory card for them to work.</li> </ol> <p>Some further googling around has led me to obexftp - part of the <a href="">OpenOBEX</a> tool set. This is a command-line tool (with gui available if you really want), that provides a quick and easy way to transfer files to and from phone. As you can specify the remote destination, it bypasses the phone’s auto-handling of the file. Here’s a quick run down of using it.</p> <ol> <li>To begin with, make sure you’ve got it installed.</li> </ol> <pre>$ sudo apt-get install obexftp</pre> <ol> <li>Find the bluetooth address of your phone. Make sure the phone’s bluetooth visibility is enabled and then run:</li> </ol> <pre>$ hcitool scan Scanning ... 00:11:22:33:44:55 MyPhone </pre> <ol> <li>Test the connection. When you run this command, you will get a prompt on the phone asking to accept the connection - in case it wasn’t obvious, say ‘yes’.</li> </ol> <pre>$ obexftp -b 00:11:22:33:44:55 -l Browsing 00:11:22:33:44:55 ... Channel: 6 Connecting...done Receiving "(null)"... <?xml version="1.0"?> &lt;!DOCTYPE folder-listing SYSTEM "obex-folder-listing.dtd" [ &lt;!ATTLIST folder mem-type CDATA #IMPLIED&gt; &lt;!ATTLIST folder label CDATA #IMPLIED&gt; ]&gt; <folder-listing version="1.0"> <folder name="C:" user-perm="R" mem-type="DEV" label="NOKIA" /> <folder name="E:" user-perm="RW" mem-type="MMC" label="Memory card" /> </folder-listing>done Disconnecting...done </pre> <p>All being well, at this point you’re all set to put and get files to your heart’s content.</p> <p>To put a file:</p> <pre>$ obexftp -b 00:11:22:33:44:55 -c "E:\\Python\\" -**p**</pre> <p>To get a file:</p> <pre>$ obexftp -b 00:11:22:33:44:55 -c "E:\\Python\\" -**g**</pre> <p>To delete a file:</p> <pre>$ obexftp -b 00:11:22:33:44:55 -c "E:\\Python\\" -**k**</pre> <p>There are a few other options available that I won’t bother going into because if you’ve read this far, you’re probably capable of reading:</p> <pre>$ man obexftp</pre> <p>A couple gotchas that caught me out at first; make sure to use double-backslashes for the path separators and make sure to put a path separator at the end of the path. Not doing either of these caused the file to transfer fine, but then the phone’s auto-handling kicked it.</p> Blinking Picaxe 2009-04-15T00:00:00+00:00 <p>Now <a href="">PICAXE</a> chips can be programmed from linux, I’ve got a hold of some to play with.</p> <p>The obligatory Hello World of hardware hacking:</p> <p><a data-flickr-embed="true" href="" title="Blinky"><img src="" width="320" height="240" alt="Blinky" /></a><script async="" src="//" charset="utf-8"></script></p> <p><a href="" title="Picaxe Dev Board by nol, on Flickr"><img src="" alt="Picaxe Dev Board" /></a><a href="" title="LED by nol, on Flickr"><img src="" alt="LED" /></a><a href="" title="Picaxe by nol, on Flickr"><img src="" alt="Picaxe" /></a></p> Logic Gate Thing 2009-03-24T00:00:00+00:00 <p>Whilst playing around with some ideas for a project I have in mind, I have created something else along the way.</p> <p>I needed the ability to draw lines between elements on a webpage, with some built-in intelligence on how to route the lines so they look sensible. I couldn’t find anything in jQuery, Dojo or elsewhere that did this type of thing out of the box, although I’ll admit I didn’t look too hard.</p> <p>Having got something basically working, I decided to throw together a demo to help see if they did what I wanted. As is often the case, my little demo has taken a life of its own and has distracted me from the original project I was doing this for.</p> <p>And so came to life the <a href="/code/logic/">Logic Gate Thing</a>.</p> <p><a href="/code/logic/"><img src="/blog/content/2009/03/ss4.png" alt="Logic Gate Thing" title="Logic Gate Thing" /></a></p> <p>A couple disclaimers to make: it has only been tested with Firefox on Linux and is reliant on Javascript. That said, I’d be happy to hear how other browsers get on with it.</p> <p><em>Update</em>: It should now work on WebKit based browsers; Safari, Epiphany, etc and Chrome. As Jamie points out in the comments, it does work in IE8 albeit with some slightly weird lines.</p> On books 2009-03-12T00:00:00+00:00 <p>The progression of the digital age is changing what will become the cultural objects that mark our time. In 50 years, will passing on a 1st edition Kindle to ones children hold the same significance as passing on a 1st edition Alice’s Adventures in Wonderland? What about a vinyl copy of The Wall versus an iPod loaded with the entire Pink Floyd back-catalog?</p> <p>As the digital age makes it easier to access and share information, does the value in physical embodiments of these things reduce? Who would buy a 26-volume encyclopedia today when they could search wikipedia in an instant?</p> <p>A common sentiment at BookCamp and beyond is that you can’t replace the experience of holding a book. Whilst 1000’s of people may read a book, few will read the same book, the same physical embodiment of the text with all of its history and experience worn into its patina.</p> <p>There are many sites that help socialise the experience of reading a book - I have recently started to use <a href="">bkkeepr</a> but have also tried out <a href="">LibraryThing</a>. They are good for keeping a record of what you’ve read and what your friends have read but I can’t help feel they are missing something when it comes to the physicality of books. More on this, maybe, another day.</p> <p>For now, here’s what triggered me to write this today. Whilst books are important, sometimes ‘progress’ is inevitable.</p> <p><a href=""><img src="/blog/content/2009/03/sftwitter.png" alt="Stephen Fry on books" /></a></p> Holy Trinity Tesco 2009-03-11T00:00:00+00:00 <p><em>The whys and wherefores are <a href="">here</a>.</em></p> <p><a href="" title="Holy Trinity Tesco by nol, on Flickr"><img src="" alt="Holy Trinity Tesco" /></a></p> <blockquote> <p>…even the quiet village of Lyddle End was not immune. When the Church was forced to sell the property due to its own financial collapse, it was bought by one of the few international retailers remaining. However, it did not last.</p> <p>People had been badly affected by the collapse. It led to a huge backlash against faceless corporations and many people turned to the more traditional, local, retailers. When the Holy Trinity store proved to be no longer financially viable it was shutdown.</p> <p>The village council decided to leave the site untouched as a reminder of the hubris of an earlier age.</p> </blockquote> <p><a href="" title="Holy Trinity Tesco by nol, on Flickr"><img src="" alt="Holy Trinity Tesco" /></a> <a href="" title="Holy Trinity Tesco by nol, on Flickr"><img src="" alt="Holy Trinity Tesco" /></a><br /> <a href="" title="Holy Trinity Tesco by nol, on Flickr"><img src="" alt="Holy Trinity Tesco" /></a> <a href="" title="Holy Trinity Tesco by nol, on Flickr"><img src="" alt="Holy Trinity Tesco" /></a></p> We Made, We Hacked 2009-03-02T00:00:00+00:00 <p>As I had hoped, Saturday’s <a href="">Makers &amp; Hackers</a> event was a lot of fun. After a kick off by Alex from <a href=""></a>, we split up into four teams vying for the two £50 prizes up for grabs.</p> <p>I was in a team with jewellery maker <a href="">Kali</a> and advertising designer <a href="">Andrew</a>. After some time spent staring at our pile of arduinos, leds, beads and felt we set out to do something around interactive jewellery; something that could react to events around you and notify you of things. Cue five hours of prototyping, coding, soldering, beading, making and hacking.</p> <p><a href="" title="Interactive Jewellery by nol, on Flickr"><img src="" alt="Interactive Jewellery " /></a> <a href="" title="Team Fragile by nol, on Flickr"><img src="" alt="Team Fragile" /></a></p> <p>Our finished piece was a bracelet with 5 leds surrounding a central one, hidden by Kali’s great bead work, with a button to allow the wearer to interact with it. The concept was that the central led would alert the wearer to some event, such as they have new messages. The outer ring of leds could then be used to indicate the extent of the event, such as the number of new messages.</p> <p>Now, clearly you have to ignore the tail of wires running off the piece connecting it to an arduino. They don’t make it very practical to wear - but there is no reason why the arduino couldn’t have become a feature of a second piece that could be worn as well - particularly if we had a <a href="">Lilypad</a> to hand. We started talking about having a jewellery ‘hub’ where each piece you wear connects together into their own network. Slightly more than we could have practically achieved on the day.</p> <p><a href="" title="Team Creatorz of Crazyness by nol, on Flickr"><img src="" alt="Team Creatorz of Crazyness" /></a> <a href="" title="Team Nigel &amp; The Craft Girls by nol, on Flickr"><img src="" alt="Team Nigel &amp; The Craft Girls" /></a><br /> <a href="" title="Team Two Guys and 5 Legs by nol, on Flickr"><img src="" alt="Team Two Guys and 5 Legs" /></a> <a href="" title="Team Fragile by nol, on Flickr"><img src="" alt="Team Fragile" /></a></p> <p>The teams were:</p> <ul> <li>Creatorz of Crazyness, who made a tactile clock that allows you to feel if it is time to get up in the morning,</li> <li>Nigel and the Craft Girls, who made a set of RFID’d robot plushies that each triggered a unique MIDI effect generated by the arduino,</li> <li>Two Guys and Five Legs, who made a rotary encoder out of an office chair, some magnets and reed switches. This could be used to count how many times some spins around in their chair.</li> <li>Fragile, which was our team.</li> </ul> <p>After a quick show’n’tell to everyone we got to vote for our favourite. Nigel and the Craft Girls came out on top of the public vote and my team got the experts choice, as voted for by the guys there on the day.</p> <p> will be taking both of our pieces with them to the <a href="">UK Maker Faire</a> in a couple weeks, so we’ve got a bit of time to tidy and polish before they get shown off.</p> <p>You can see the rest of my photos from the day <a href="">here</a> and there is also the wider pool with photos from everyone else <a href="">here</a>.</p> <p><a href="" title="The Makers and The Hackers by nol, on Flickr"><img src="" alt="The Makers and The Hackers" /></a></p> Makers & Hackers 2009-02-24T00:00:00+00:00 <p><a href=""><img src="/blog/content/2009/02/makers_and_hackers-300x205.jpg" alt="makers_and_hackers" title="makers_and_hackers" /></a></p> <blockquote> <p><a href="">Makers and Hackers</a> is a one day competition open to makers of all kinds run by <a href="">Folksy</a> and <a href="">!</a> Show up in Sheffield or London on Feb 28th, and make a “Household items of the future” in 8 hours with the people around you!</p> </blockquote> <p>Tickets are free, but you do have to book - check out the <a href="">site</a> for details; it should be a lot of fun.</p> Links for 2008-12-22 to 2009-02-03 2009-02-03T00:00:00+00:00 <ul> <li><a href="">Infosthetics Competition: Build a Paper-Based Visualization - information aesthetics</a> - Inspired by my Paper Graphs, Infosthetics is hosting a competition to create a paper based visualisation. “Criteria: beauty and originality versus readability and usability”. I'm a judge.</li> <li><a href="">PMD - The Pac-Man Dungeons</a> - Pac-Man as a text adventure.</li> <li><a href="">News Blog - Times Online - WBLG: Greatest ever letter of complaint</a> - “imagine it’s Christmas morning and you’re sat their with your final present to open. It’s a big one, and you know what it is. It’s that Goodmans stereo you picked out the catalogue and wrote to Santa about. Only you open the present and it’s not in there. It’s your hamster Richard. It’s your hamster in the box and it’s not breathing.” Very funny letter of complaint sent to Richard Branson.</li> <li><a href=""> » graphtecprint</a> - Linux support for the Graphtec CC2200-20.</li> <li><a href="">Graphtec Craft ROBO - Mulberry design + print</a> - An A4 plotter.</li> <li><a href="">Adactio: Journal—PaperCamp</a> - Jeremy Keith's live blogging of PaperCamp.</li> <li><a href="">Open Manufacturing - (How to Build 30 (SERB) Kits)</a> - Interesting instructable on how they setup a tabletop factory for mass producing an arduino-based robot.</li> <li><a href="">Technobots Shop</a> - A source of toys.</li> <li><a href="">Motorway map of England, Scotland and Wales</a> - Motorway map in the style of the London Underground map.</li> </ul> Updated Client for MQTT library 2009-01-28T00:00:00+00:00 <p>I’ve made some updates to the MQTT library for the arduino ethernet shield and released version 1.1. You can find the library <a href="">here</a>. <strike>I'm thinking of moving it to github or google code once I can decide which to use - any suggestions?</strike> The code is now hosted on <a href="">github</a>.</p> <p>This update includes:</p> <ul> <li>smaller code footprint,</li> <li>support for Will messages - particularly useful for embedded devices,</li> <li>clarified licensing - this is released under the <a href="">MIT License</a>.</li> </ul> <p>Enjoy.</p> Paper Graphs 2009-01-16T00:00:00+00:00 <p>With <a href="">PaperCamp</a> happening tomorrow, alongside <a href="">BookCamp</a>, I have spent some time this week thinking about paper.</p> <p>The thought that caught my interest was how graphs can be represented with paper - not <em>on</em> paper, but <em>with</em> paper.</p> <p>To cut a rambling story short, I’ve written a tool that takes a set of pie-chart data and produces a series of outlines that can be cut out and assembled to create a 3D model of the chart. <img src="/blog/content/2009/01/pie_outline-150x150.png" alt="pie_outline" title="pie_outline" /> <img src="/blog/content/2009/01/pie_grid-150x150.png" alt="pie_grid" title="pie_grid" /> <img src="/blog/content/2009/01/pie_slices-150x150.png" alt="pie_slices" title="pie_slices" /></p> <p><img src="/blog/content/2009/01/piex4-300x93.png" alt="piex4" title="piex4" /></p> <p><a href="" title="Prototype Paper Pie Chart by nol, on Flickr"><img src="" alt="Prototype Paper Pie Chart" /></a></p> <p>Having a physical embodiment of the raw data means you can touch it, feel it, grope it; whatever helps you understand it.</p> <p>I think I’ll leave it there for now; it’s late, I have an early train to catch and I’ve used the word grope.</p> <p><em>Update:</em> as I mention in the comments, you can get your own custom paper pie chart from <a href="">here</a>.</p> Nokia LCD Shield for Arduino 2009-01-10T00:00:00+00:00 <p><a href="" title="Nokia 6610 + Joystick Shield by nol, on Flickr"><img src="" alt="Nokia 6610 + Joystick Shield" /></a></p> <p>I got an <a href=";cPath=1&amp;products_id=10">LCD shield</a> along with a second Arduino a couple weeks ago but what with Christmas and finishing off the <a href="/2009/01/05/monitoring-energy-use-with-an-orb/">Orb</a> I hadn’t had much of a chance to play with it. Until tonight.</p> <div style="text-align: center;"><object type="application/x-shockwave-flash" width="400" height="300" data="" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000"> <param name="flashvars" value="intl_lang=en-us&amp;photo_secret=ff86f2d899&amp;photo_id=3182729461" />&lt;/param&gt; <param name="movie" value="" />&lt;/param&gt; <param name="bgcolor" value="#000000" />&lt;/param&gt; <param name="allowFullScreen" value="true" />&lt;/param&gt;<embed type="application/x-shockwave-flash" src="" bgcolor="#000000" allowfullscreen="true" flashvars="intl_lang=en-us&amp;photo_secret=ff86f2d899&amp;photo_id=3182729461" height="300" width="400" />&lt;/embed&gt;</object></div> <pre>/************************************************ * Draws a rotating 3D cube on the LCD Shield * from Nuelectronics. * * Nicholas O'Leary * ************************************************/ #include "Nokia_lcd.h" #include &lt;avr/pgmspace.h&gt; float sin_d[] = { 0,0.17,0.34,0.5,0.64,0.77,0.87,0.94,0.98,1,0.98,0.94, 0.87,0.77,0.64,0.5,0.34,0.17,0,-0.17,-0.34,-0.5,-0.64, -0.77,-0.87,-0.94,-0.98,-1,-0.98,-0.94,-0.87,-0.77, -0.64,-0.5,-0.34,-0.17 }; float cos_d[] = { 1,0.98,0.94,0.87,0.77,0.64,0.5,0.34,0.17,0,-0.17,-0.34, -0.5,-0.64,-0.77,-0.87,-0.94,-0.98,-1,-0.98,-0.94,-0.87, -0.77,-0.64,-0.5,-0.34,-0.17,0,0.17,0.34,0.5,0.64,0.77, 0.87,0.94,0.98}; float d = 10; float px[] = { -d, d, d, -d, -d, d, d, -d }; float py[] = { -d, -d, d, d, -d, -d, d, d }; float pz[] = { -d, -d, -d, -d, d, d, d, d }; float p2x[] = {0,0,0,0,0,0,0,0}; float p2y[] = {0,0,0,0,0,0,0,0}; int r[] = {0,0,0}; Nokia_lcd lcd=Nokia_lcd(); void setup(void){ DDRB=0x2F; LCD_BACKLIGHT(1); lcd.cLCD_Init(); lcd.cLCD_Box(0,0,131,131,FILL,BLACK); } void loop(void) { r[0]=r[0]+1; r[1]=r[1]+1; if (r[0] == 36) r[0] = 0; if (r[1] == 36) r[1] = 0; if (r[2] == 36) r[2] = 0; for (int i=0;i&lt;8;i++) { float px2 = px[i]; float py2 = cos_d[r[0]]*py[i] - sin_d[r[0]]*pz[i]; float pz2 = sin_d[r[0]]*py[i] + cos_d[r[0]]*pz[i]; float px3 = cos_d[r[1]]*px2 + sin_d[r[1]]*pz2; float py3 = py2; float pz3 = -sin_d[r[1]]*px2 + cos_d[r[1]]*pz2; float ax = cos_d[r[2]]*px3 - sin_d[r[2]]*py3; float ay = sin_d[r[2]]*px3 + cos_d[r[2]]*py3; float az = pz3-150; p2x[i] = 65+ax*500/az; p2y[i] = 65+ay*500/az; } lcd.cLCD_Box(0,0,131,131,FILL,BLACK); for (int i=0;i&lt;3;i++) { lcd.cLCD_Line(p2x[i],p2y[i],p2x[i+1],p2y[i+1],RED); lcd.cLCD_Line(p2x[i+4],p2y[i+4],p2x[i+5],p2y[i+5],RED); lcd.cLCD_Line(p2x[i],p2y[i],p2x[i+4],p2y[i+4],RED); } lcd.cLCD_Line(p2x[3],p2y[3],p2x[0],p2y[0],RED); lcd.cLCD_Line(p2x[7],p2y[7],p2x[4],p2y[4],RED); lcd.cLCD_Line(p2x[3],p2y[3],p2x[7],p2y[7],RED); delay(5); } </pre> <p><a href="" title="LCD Shield by nol, on Flickr"><img src="" alt="LCD Shield" /></a></p> Monitoring energy use with an Orb 2009-01-05T00:00:00+00:00 <p>I spoke at <a href="/2008/11/30/homecamp-08/">Homecamp</a> recently about how an ambient orb could be used to monitor home energy usage. I’ve finally gotten around to putting some of it into practice so thought I would share some of the details of the setup as well as some more of my thoughts on the subject.</p> <p>There are three key pieces of hardware in use. The Viglen MPC-L is the heart of the system. As I’ve mentioned previously, this is a low powered linux box running Ubuntu. The CurrentCost meter is connected to the MPC over USB-serial and my trusty arduino acts as an integration point for homebrew toys - including my <a href="/2008/11/25/diy-ambient-orb-redux/">ambient orb</a>.</p> <p>The MPC is running a <a href="">Really Small Message Broker</a> (RSMB). This is a small-footprint pub-sub message broker that talks MQTT. Each time the CurrentCost sends out a update, a piece of perl (“<code class="language-plaintext highlighter-rouge"></code>”) parses the data, sticks it into an RRDTool database for graphing and also publishes it to the <code class="language-plaintext highlighter-rouge">home/cc/power</code> and <code class="language-plaintext highlighter-rouge">home/cc/temp</code> topics.</p> <p>Another piece of perl (“<code class="language-plaintext highlighter-rouge"></code>”) is subscribed to the <code class="language-plaintext highlighter-rouge">house/orb</code> topic. When messages arrive on that topic, they are passed over serial to the arduino.</p> <p>The sketch on the arduino currently listens on its serial port for commands that are then passed to the BlinkM in the orb. The format of the commands is identical to those in the BlinkMTester sketch that comes with the BlinkM. In the future this will do more as more things are attached to the arduino.</p> <p>So far I have described how the orb is controlled and how the power data gets into the system. The next piece is how the two are plumbed together. Unsurprisingly, yet another perl script is running the on the MPC that provides the glue for this mixed metaphor.</p> <p><code class="language-plaintext highlighter-rouge"></code> is subscribed to the <code class="language-plaintext highlighter-rouge">home/cc/power</code> topic so it receives all of the updates from the CurrentCost. It then makes a decision as to what colour the orb should be and then publishes the appropriate command to the <code class="language-plaintext highlighter-rouge">house/orb</code> topic.</p> <p><img src="/blog/content/2009/01/orbsetup.jpg" alt="Orb setup sketch" title="Orb setup sketch" /></p> <p>That is pretty much it - simple eh? Well, I did skip over the most interesting part - how to decide what colour the orb should be.</p> <p>There are two key philosophies that come into play here:</p> <ul> <li>Alert the unusual - have the ‘default’ state be the least obtrusive, such as ‘off’</li> <li>Band the data - don’t react to single-point changes in the value. For example, when monitoring temperature, nothing is worse than constantly saying “its 19°C, its 18°C, its 19°C…” These points should be applied whenever thinking about how to turn a raw stream of data into a useful stream of information.</li> </ul> <p>I currently have a very simple piece of logic controlling the orb:</p> <ul> <li>If <code class="language-plaintext highlighter-rouge">power</code> &lt; 400, turn the orb off</li> <li>If <code class="language-plaintext highlighter-rouge">power</code> between 400 and 1000, turn the orb orange</li> <li>If <code class="language-plaintext highlighter-rouge">power</code> &gt; 1000, turn the orb red When we’re sat in the living room watching TV, we typically use under 400 watts, so the orb will only show anything if we’re over this value. It is very unusual to go over 1 kw unless we’re cooking, which is why that gets a stronger alert.</li> </ul> <p><img src="/blog/content/2009/01/powergraph.jpg" alt="powergraph sketch" title="powergraph sketch" /></p> <p>Despite its simplicity, this has already had the effect of making us more aware of when lights have been left on or when the kettle has finished boiling.</p> <p>But as with all things, it could get much smarter. Having said we typically use under 400 watts watching TV, I also made the caveat ‘unless we’re cooking’. Given the system has a history of our power usage, it could feasibly determine suitable bands dynamically - so going over 1 kw between 6pm and 7pm isn’t necessarily a bad thing. Also, it chould only go red if the value goes over 1 kw for more than the time it takes to boil the kettle.</p> <p>It all comes back to determining what is ‘unusual’.</p> Light Tent 2009-01-04T00:00:00+00:00 <p>Inspired by <a href="">mattb</a>, <a href="">andyp</a> and <a href="">make</a>, I have made my own light tent. Made from a cardboard box, some tracing paper and masking tape, it took about 10 minutes to put together.</p> <p>My first subject for the tent was my growing collection of arduino boards and shields.</p> <div style="text-align:center;"><object width="400" height="300"> <param name="flashvars" value="&amp;offsite=true&amp;lang=en-us&amp;page_show_url=%2Fphotos%2Fknolleary%2Fsets%2F72157612145293846%2Fshow%2F&amp;page_show_back_url=%2Fphotos%2Fknolleary%2Fsets%2F72157612145293846%2F&amp;set_id=72157612145293846&amp;jump_to=" />&lt;/param&gt; <param name="movie" value="" />&lt;/param&gt; <param name="allowFullScreen" value="true" />&lt;/param&gt;<embed type="application/x-shockwave-flash" src="" allowfullscreen="true" flashvars="&amp;offsite=true&amp;lang=en-us&amp;page_show_url=%2Fphotos%2Fknolleary%2Fsets%2F72157612145293846%2Fshow%2F&amp;page_show_back_url=%2Fphotos%2Fknolleary%2Fsets%2F72157612145293846%2F&amp;set_id=72157612145293846&amp;jump_to=" width="400" height="300" />&lt;/embed&gt;</object></div> Links for 2008-11-20 to 2008-12-18 2008-12-18T00:00:00+00:00 <ul> <li><a href="">James Governor’s Monkchips » My Team Of The Year Award: IBM EightBar, Hursley Labs</a> - “My Team of 2008 award goes to IBM’s Eight Bar at Hursley Labs. RedMonk celebrates makers and doers - these guys exemplify getting on with it.”</li> <li><a href="">The Arduino AA Undershield</a> - A battery pack shield for Arduino.</li> <li><a href="">Making Electricity Usage Graphs with the Current Cost device</a> - Using rrdtool to log currentcost data. rrdtool is one of those things that I've never quite spent enough time understanding.</li> <li><a href="">Spectacular HomeCamp feedback!</a> - Some talk heads about HomeCamp. Interesting to hear what people got out of the event. I'm in there somewhere babbling on a bit wishing I had thought about what to say first.</li> <li><a href="">Tinker it now! » Stuff and things</a> - My arduino hacking gets a mention on</li> </ul> Upgrading the MPC-L from Feisty 2008-12-12T00:00:00+00:00 <p><em>In a break from the norm, I have gone for a search friendly blog title as this is useful information that I couldn’t find myself</em></p> <p>Currently, the Viglen MPC-L is shipped with Xubuntu 7.04 (Feisty). As of October this year, Feisty is no longer supported so any attempt to update the software results in plenty of 404 error messages like. A trawl around with google finally found <a href="">this Launchpad response</a>.</p> <p>Based on that, here are my notes for upgrading from Xubuntu 7.04 to 7.10.</p> <ol> <li>Check you really are on 7.04 by running:&lt;pre&gt;:~$ lsb_release -a&lt;/pre&gt;</li> <li> <p>Disable the screensaver. Once running, the upgrade takes an hour or so and you do not want the screensaver to activate. Both AndySC and I found we couldn’t unlock the screen after the upgrade had completed overnight and the only thing to do was reboot. That said, the reboot didn’t harm the upgrade, but it is better to be safe.</p> </li> <li>Download the <a href="">Xubuntu 7.10 alternate CD ISO</a> (<a href="">torrent</a>). It may seem obvious, but make sure you get the Xubuntu CD, not the Ubuntu one. Using the Ubuntu one will fail and you’ll have wasted half-an-hour downloading the ISO - I speak from personal experience.</li> <li> <p>Mount the ISO locally:&lt;pre&gt;:~$ sudo mount -o loop \ ~/Desktop/xubuntu-7.10-alternate-i386.iso \ /media/cdrom0&lt;/pre&gt;</p> </li> <li>A dialog will appear prompting you to upgrade from the CD. If it doesn’t appear, run:&lt;pre&gt;:~$ gksu “sh /cdrom/cdromupgrade”&lt;/pre&gt;</li> <li>When it asks if you would like to include the latest updates from the internets, you say “No”. If you say “Yes” you hit the same 404s you spent an hour trying solve before going down this path. Again, personal experience.</li> <li>After about a few minutes of sorting out what it needs to do it will start installing the new packages. I found it prompted me for input about 3 times early on in the ~1 hour install time and then ran to completion without further interaction - YMMV.</li> <li>Reboot as prompted and enjoy.</li> </ol> <p>This should hopefully get you onto Gutsy, from where you can use the regular Update Manager to upgrade to Hardy and beyond.</p> <p>Given the pain it is to upgrade from Feisty now, I hope Viglen will update their pre-install image soon. I’m also surprised that Ubuntu consider this an acceptable path even if Feisty is no longer supported.</p> <p><em>Update:</em> as per <a href="">Andy’s notes</a>, when you upgrade from Gutsy to Hardy, make sure the Universe repository is enabled, otherwise it will fail as the <code class="language-plaintext highlighter-rouge">xubuntu-desktop</code> package moved repositories between these releases.</p> 1000th Tweet 2008-12-03T00:00:00+00:00 <p><a href="" title="1000th Tweet Haiku"><img src="" alt="1000th Tweet Haiku" /></a></p> HomeCamp '08 2008-11-30T00:00:00+00:00 <p><img src="/blog/content/2008/11/hc-284x300.png" alt="" title="HomeCamp" /></p> <p>Today was the first <a href="">homecamp</a> - “an unconference about using technology to monitor and automate the home for greener resource use and to save costs”.</p> <p>Whilst I hadn’t entirely planned to, I found myself stood up front leading a couple sessions; one on what I’ve been doing with <a href="">arduino</a> and one a brief overview of what <a href="">MQTT</a> is.</p> <!--more--> <p>As the wireless was only staying up for 30 minutes at a time, my arduino talk is split over a couple UStream videos. I come in at about 22:40 of this first one…</p> <div style="text-align: center"><embed flashvars="autoplay=false" width="400" height="320" allowfullscreen="true" allowscriptaccess="always" src="" type="application/x-shockwave-flash" /></div> <p>…and the rest is here with only a small gap missing between them.</p> <div style="text-align: center"><embed flashvars="autoplay=false" width="400" height="320" allowfullscreen="true" allowscriptaccess="always" src="" type="application/x-shockwave-flash" /></div> <p>Here’s the brief piece I did on all things MQTT, RSMB and MicroBroker. You should listen to Chris talk about his <a href="">SketchUp plugin for Pachube</a> at the start of the video, or you can skip to me at 14:20.</p> <div style="text-align: center"><embed flashvars="autoplay=false" width="400" height="320" allowfullscreen="true" allowscriptaccess="always" src="" type="application/x-shockwave-flash" /></div> <p>The rest of the videos are worth watching - they can be found on the <a href="">homecamp ustream page</a>.</p> <p>Here are the slides I was using for the arduino talk. I will add some notes to them over the next couple days, so be sure to check back on <a href="">slideshare</a>.</p> <iframe src="//" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//" title="Arduino Home Automation Hacks" target="_blank">Arduino Home Automation Hacks</a> </strong> from <strong><a href="" target="_blank">Nicholas O&#x27;Leary</a></strong> </div> <p>One other contribution I made to the day was winning the race to respond to Andy’s <a href="">request</a> for a homecamp logo. You can see my effort at the top of this post as well as on the official <a href="">homecamp friendfeed room</a>. Since then, <a href="">@ribot</a> has produced this decidedly more <a href="">shiny one</a>.</p> <p>Today was a lot of fun with plenty of interesting and inspiring conversations. A big thank you and well done to Chris and Dale for putting this together so quickly. I look forward to HomeCamp ‘09.</p> DIY Ambient Orb: Redux 2008-11-25T00:00:00+00:00 <p><a href="" title="Orb by nol, on Flickr"><img src="" alt="Orb" /></a></p> <p><a href="" title="Orb Base by nol, on Flickr"><img src="" alt="Orb Base" /></a></p> <p><a href="" title="RJ11 by nol, on Flickr"><img src="" alt="RJ11" /></a></p> <pre> /************************************************ * Enables a BlinkM to be controlled over MQTT. * - subscribes to the topic 'blinkm' * - expects messages of the format 'RRGGBB' * - doesn't do any error handling * * Nicholas O'Leary * ************************************************/ #include <Wire.h> #include <Ethernet.h> #include <PubSubClient.h> #include <BlinkM_funcs.h> byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; byte ip[] = { 172, 16, 0, 50 }; byte server[] = { 172, 16, 0, 100 }; void callback(char* topic, byte* payload,int length) { byte a = toHex( payload[0],payload[1] ); byte b = toHex( payload[2],payload[3] ); byte c = toHex( payload[4],payload[5] ); BlinkM_fadeToRGB( 0x09, a,b,c); } PubSubClient client(server, 1883, callback); void setup() { BlinkM_beginWithPower(); BlinkM_setAddress( 0x09 ); BlinkM_stopScript( 0x09 ); Ethernet.begin(mac, ip); if (mqttClient.connect("blinkr")) { mqttClient.subscribe((uint8_t*)"blinkm"); } } void loop() { mqttClient.loop(); delay(500); } // a really cheap strtol(s,NULL,16) - taken from BlinkMTester #include <ctype.h> uint8_t toHex(char hi, char lo) { uint8_t b; hi = toupper(hi); if( isxdigit(hi) ) { if( hi &gt; '9' ) hi -= 7; hi -= 0x30; b = hi&lt;&lt;4; lo = toupper(lo); if( isxdigit(lo) ) { if( lo &gt; '9' ) lo -= 7; lo -= 0x30; b = b + lo; return b; } } return 0; }&lt;/pre&gt; </ctype.h></BlinkM_funcs.h></PubSubClient.h></Ethernet.h></Wire.h></pre> CurrentCostuino 2008-11-19T00:00:00+00:00 <p>With <a href="">homecamp</a> fast approaching I spent some time last weekend reviving some CurrentCost hacking I started in the summer.</p> <p>One of the downsides of the data connection on the CurrentCost is the need for an always-on computer in order to capture the data. In practice, the existence of low powered machines like the MPC-L, or slug, make this a pretty inconsequential downside. That said, it was enough motivation to see what I could do with an Arduino and Ethernet Shield.</p> <p>I have <a href="/2008/05/05/power-graphing/">previously</a> mentioned that part of the setup used to graph the power data online was to publish the data to a broker-in-the-sky using MQTT. To achieve this with an arduino it was clear that I needed one thing in particular - an arduino that is capable of connecting and publishing to an MQTT server.</p> <p>With my ethernet shield in hand, I set about implementing enough of the MQTT spec to support basic publishing and subscribing. With that working, I looked into getting the arduino and CurrentCost talking. Thanks to <a href="">Alexis</a>, this was very straightforward, although I did rework the parsing of the data to take advantage of its somewhat-fixed length format.</p> <p>Here’s the sketch I ended up with:</p> <pre> #include <Ethernet.h> #include <PubSubClient.h> #include <SoftwareSerial.h> byte mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED }; byte ip[] = { 172, 16, 0, 50 }; byte server[] = { 172, 16, 0, 103 }; PubSubClient client(server, 1883,0); #define rx 6 #define tx 7 SoftwareSerial softSerial = SoftwareSerial(rx, tx); void setup() { pinMode(rx, INPUT); pinMode(tx, OUTPUT); softSerial.begin(9650); Ethernet.begin(mac, ip); delay(500); } void connect() { while(!client.connect("nolcc")) { delay(10000); } delay(200); } int offset = -1; void loop() { if (!client.connected()) { connect(); offset = -1; } client.loop(); // scan the data for <msg> while(offset == -1) { char c = ' '; while (c!='&lt;') { c =; } if ('m' &amp;&amp;'s' &amp;&amp;'g' &amp;&amp;'&gt;') { offset = 5; } } // skip to the current value while(offset &lt; 156) { offset++;; } char current[5]; int value = 0; for (int i=0;i&lt;5;i++) { current[i] =; } offset += 5; // skip to the temperature value while(offset &lt; 243) { offset++;; } char temp[5]; for (int i=0;i&lt;4;i++) { temp[i] =; } offset += 4; client.publish("current",(uint8_t*)current,5); client.publish("temp",(uint8_t*)temp,4); // scan for </msg> while(offset != -1) { char c = ' '; while (c!='&lt;') { c =; } if ('/' &amp;&amp;'m' &amp;&amp;'s' &amp;&amp;'g' &amp;&amp;'&gt;') { offset = -1; } } } &lt;/pre&gt; It publishes each power reading to a topic called 'current' and each temperature reading to a topic called 'temp'. This isn't quite the right configuration for the existing broker setup, but it wouldn't take much to modify it appropriately. You'll note this sketch isn't much use without the library that does the MQTT bit. Luckily, I have been given permission to release the library to the world. I should stress at this point, in case it wasn't obvious from the fact I am releasing this code on my _personal_ site, that my employer has nothing to do with it. You'll find the Client for MQTT library [here]( If you want an MQTT server to play with, go buy [Lotus Expeditor]( to get your hands on a microbroker <gratuitous-plug />, or check out the links on []( for the Really Small Message Broker. </SoftwareSerial.h></PubSubClient.h></Ethernet.h></pre> Arduino Client for MQTT 2008-11-18T00:00:00+00:00 <p>Moved to <a href=""></a></p> Photo Calendar 2009 2008-11-10T00:00:00+00:00 <p>I like to take the odd <a href="">photo</a> now and then. Last year, with the help of <a href="">Lulu</a> I put together a calendar that featured some of my favourite photos as a Christmas present for my parents and in-laws.</p> <p>This year I have done the same, and as the surprise factor is no longer there, I have decided to make it available to anyone who wants to buy one. They cost £8.25 excluding postage - none of which is profit or will go to charity.</p> <p>[<img src="/blog/content/2008/11/preview.jpg" alt="" title="Calendar 2009 Preview" /></p> <p><img src="" alt="Support independent publishing: buy this calendar on Lulu." /> ](</p> Arduino Ethernet Shield 2008-11-07T00:00:00+00:00 <p>Earlier this year, I was fortunate enough to <a href="">get</a> a pre-release Arduino Ethernet shield from the good folk at <a href=""></a> to play with. At that time, I had spent some time trying to improve the TCP-stack for the <a href=";cPath=1&amp;products_id=4">nuelectonics ethernet shield</a>, but that was hard work and I wasn’t making much progress.</p> <p>The advantage of this new shield is that all of the hard work is done for you - the API provided is very simple to use and doesn’t take up too much program space. It also includes an SD card slot and although that isn’t supported in the library yet, it’s potentially a very cool addition.</p> <p>Massimo has now <a href="">announced</a> the shield to the world. Hopefully they will be available from the <a href=""> store</a> soon.</p> <p><a href="" title="Arduino Ethernet Shield by nol, on Flickr"><img src="" alt="Arduino Ethernet Shield" /></a></p> <p>I have a cool project on the go using the shield that I’m going to hold off writing about for the moment. It touches close to what I get paid to do at work, so I have to go through a few hoops before I can make the code available publicly. Anyone following my <a href="">twitter stream</a> will have a good idea what it is already.</p> <p>My plan is to write it up and release the code in time for <a href="">HomeCamp</a>, as it will tie in very nicely to some of the topics being discussed there.</p> Links for 2008-10-20 to 2008-11-06 2008-11-06T00:00:00+00:00 <ul> <li><a href="">Arduino Blog » Blog Archive » The Internet of Arduino</a> - The arduino ethernet shield is announced. I have been playing with this for a few months now - lots of potential uses.</li> <li><a href="">Star Wars ABC: A is for Ackbar</a> - “Illustrator Michael Fleming has drawn a wonderful kid-friendly alphabet using characters from the Star Wars universe.”</li> <li><a href="">Multicolr Search Lab - Idée Inc.</a> - Search for CC photos on flickr based on colours.</li> <li> <p><a href="">Charlie’s Diary: On finishing</a> - “Creativity is a weird thing. You can plod along a steep uphill road for months, or it can hit you like an express train. I've learned to go with the flow when this happens: if a story wants to escape it's best to let it out, to go with the flow and worry about cleaning up the schedule afterwards.”</p> <p>Charlie gives an insight into what its like to finish writing a novel. Even if it is 18months early.</p> </li> <li><a href="">BBC Common Platform: From my notebook</a> - “Rights” is often used as the reason the BBC struggles to be more open with its content. As a part of his Common platform work, Steve points out that “there are other obstacles to sharing—many in fact. This chart is a first attempt at capturing all the reasons the BBC might come up with for not opening up…”</li> </ul> Playful: The Morning After 2008-11-01T00:00:00+00:00 <p>Yesterday’s Playful conference was a lot of fun. The day was packed with a lot of varied talks with no low spots. Here’s a very brief summary of what happened. It’s brief as I don’t take notes at conferences. Not because I don’t believe in it, but because I forget.</p> <p><a href="" title="Play by nol, on Flickr"><img src="" alt="Play" /></a></p> <ul> <li><a href="">James Wallis</a> spoke about <a href="">OuLiPo</a></li> <li>Roo <a href="">rocked out</a></li> <li>Matt Irvine Brown talked about drawing more dots to enable others to join them. His <a href="">TrumpetHero</a> and <a href="">Singing Sock Puppets</a> were inspired.</li> <li><a href="">Adrian Hon</a> gave a very honest account of how gaming can be addictive as well as cause addiction.</li> <li>Eric Clough took us through the incredible <a href="">New York apartment</a> he designed as one big puzzle.</li> <li>Chris Delay of <a href="">Introversion Software</a> demonstrated how procedural content generation can be used to generate entire city-scapes.</li> <li><a href="">Kars Alfrink</a> did something interesting.<del datetime="2008-11-17T13:03:08+00:00">but my brain has ejected it. Sorry Kars, I look forward to rediscovering your talk.</del> He has put a full transcript of his talk <a href="">online</a>. Thanks Kars!</li> <li><a href="">Alex Fleetwood</a> spoke about his project to combine virtual and real world dramatic experiences</li> <li><a href="">Tom Armitage</a> went from saying “all games today are multiplayer” to “all games have always been multiplayer”.</li> <li>Sandy Spangler from Sony stepped in at the last minute to talk about the history and future of the EyeToy.</li> <li><a href="">Eric Zimmerman</a> gave a very energetic performance. Again, my brain fails me this morning.</li> <li><a href="">Matt Biddulph</a> explained how game controllers can easily be tinkered with to provide new and interesting uses</li> <li><a href="">Iain Tate</a> talked about why the Highscore is a key feature of social gaming.</li> <li><a href="">Jolyon Webb</a> talked about teeth with some very life-like CG heads</li> <li><a href="">Kieron Gillen</a> ended the day encouraging us all to plagarise ideas. Why has no-one cloned The Sims yet?</li> </ul> <p>Well done to the guys at <a href="">Pixel-Lab</a> for putting this together. I look forward to next year.</p> Playful:Game Design London 2008-10-20T00:00:00+00:00 <p>I saw <a href="">Roo</a> twitter about <a href="">Playful</a> a while ago and thought it looked like fun. Then I realised that as it was on a Friday there was next to no chance that I would be able to go. Then I realised I’m on holiday that week - not a going-away holiday, but a not-at-work holiday. Then I realised my Sister-in-law is coming down from Norfolk on that Friday to stay with us over the weekend. Then I realised we’re going up to London to meet her on the Friday. Then I realised that would involve a long day shopping in London with my wife and Sister-in-law and possible a trip to see Mamma Mia!<super>\*</super></p> <p>Then I realised I have a wonderful wife.</p> <p><a href=""><img src="/blog/content/2008/10/playful_attending.jpg" alt="" title="Playful:Game Design London" /></a></p> <p>Then I realised this all means I probably get to act as a roadie for Roo’s <a href="">one-man (rock) band</a>.</p> <p>Then I realised I had over-used a certain linguistic technique.</p> <p>(In case you don’t get it… I’m going to <a href="">Playful</a>)</p> <super>\*</super> <p>The ‘!’ is part of the name; it in no way signifies any excitement on my part.</p> The End is Nigh 2008-10-10T00:00:00+00:00 <p>Do not mistake the title of this post as one by <a href="">Robert Peston</a>.</p> <p>Today marks the end of a fairly intense period at work. As I write this, the final build is being tested ready for its release later in the year. It has been a particularly tough few months getting to this point, so much so that I have rarely touched my laptop outside of work.</p> <p>The other big drain on my time has been the redecorating of our living room and dining room. Having started the work in February, and going at a somewhat leisurely pace, the new carpets finally arrive next Wednesday. With just this weekend of finishing touches to go, soon I won’t have to face any sanding, glossing, plastering, wiring, painting or building again. Until we start redoing the bathroom of course.</p> <p>This means all of those projects that I’ve been badly neglecting will soon get dusted off and start moving forward.</p> <p>So, although the End is Nigh, the Beginning is right around the corner.</p> Links for 2008-08-26 to 2008-10-08 2008-10-08T00:00:00+00:00 <ul> <li><a href="">Daves Logs</a> - An environmentally friendly source of logs for the wood burner.</li> <li><a href="">MAKE: Blog: Understanding I2C</a> - “a helpful primer on interfacing electronic components via the handy I2C protocol and Arduino”</li> <li><a href="">Young Bond: The Shadow War</a> - Six To Start's latest offering - an ARG tied to the Young Bond series. A new mission will be put each week for the next seven weeks, culminating with a live event on October 8th featuring Charlie Higson.</li> <li><a href="">Google on Google Chrome - comic book</a> - Google announce the Google Chrome web browser in the form of a comic book drawn by Scott McCloud. Some interesting things here.</li> <li><a href="">MAKE: Blog: Build Your Own Mid-Century Modern Birdhouse</a> -</li> </ul> Recent happenings 2008-09-04T00:00:00+00:00 <p>If my last post was not intended to be a post about blogging, what did I intend for it?</p> <p>Well, it was going to be a summary on the recent happenings in these here parts. Which set me off onto a train of thought as to why I hadn’t blogged about any of them already, etc etc etc. But I’m not going to make that mistake again.</p> <p>So, what has been happening?</p> <p><a href="" title="Fire Clubs by nol, on Flickr"><img src="" alt="Fire Clubs" /></a></p> <p>A couple weeks ago Jo and I spent a week up in Scotland. We were staying next to Loch Lomond at the <a href="">Auchenheglish lodges</a>. We drove up with no real plans for the week - which worked out very nicely. We spent a most days watching the Olympics in the mornings and then heading out to explore in the afternoons. I realise some might find it sacrilege to be in that part of the world and spend half your time sat on a sofa. But this was our holiday, and it was exactly what we needed.</p> <p>At the end of the week we spent the day in Edinburgh, where we experienced a flavour of the Fringe festival, walked the Royal Mile from <a href="">Edinburgh Castle</a> down to the <a href="">Scottish Parliament</a> and learnt that city center parking is very expensive (£17 for 7 hours). All of this was despite the fairly damp weather that had followed us all week. We barely scratched the surface of what Edinburgh has to offer - especially with the Fringe on. We would happily return in the future to spend a good 3 or 4 days doing it all properly.</p> <p>Afterwards, we spent the weekend with friends in Falkirk, where we took in the wonders of <a href="">The Falkirk Wheel</a> and heard all about the new stand being built by the owner of Schuh at Falkirk FC’s stadium. See, there’s my trademark wry sense of humour kicking in. To be fair, the Falkirk Wheel is an impressive feat of engineering and worth a visit if you’re in the area. We were, however, saved from experiencing the Falkirk stadium first hand.</p> <p>What I found most noticeable when we got back to Southampton was how little I had missed the laptop and internet. In fact, with the prospect of returning to work, I would have happily left it all behind for another week.</p> <p>Another highlight of recent times was the installation of our wood-burning stove. Although that was not entirely stress-free. Before going to Scotland, I laid some tiles in the fireplace for the stove to sit on. To my dismay, when we got back, the ready-mixed tile adhesive I had got from Wickes had not gone off and the tiles were not set. This led to a slightly frantic Bank Holiday Monday as I relaid the tiles, this time with some adhesive bought from a tiling shop. Thankfully the adhesive took and a quick bit of grouting later, the fireplace was ready for the stove guys the following Thursday.</p> <p><a href="" title="Fire! by nol, on Flickr"><img src="" alt="Fire!" /></a></p> <p>Working from home that day, I hid away in our bedroom whilst the two guys did their thing. With the chimney breast running through that room, it was clear who was the apprentice - he spent a lot of the day on the roof trying to get the lining down the chimney, whilst having instructions shouted back up at him from the other guy.</p> <p>Eventually they were done, and we were left with our Clearview Pioneer 400 ready for action. The next step is to find a suitable supply of fuel. For now, I have been saving all of the off-cuts from the renovations - there’s plenty of old skirting board to get through.</p> <p>Most recently, we went to Oxford last Saturday for <a href="">Megan and Ben’s wedding</a>. It was a great day for everyone, the weather was absolutely superb - which was very lucky considering August has been one of the least sunny on record. It was certainly one of the grandest weddings I have been to - set in the surroundings of Mansfield college.</p> <p>And that’s it.</p> Unblocking my brain 2008-09-03T00:00:00+00:00 <p>I’m going to say it up front - I did not intend this to be a blog post about blogging, but I think that is the way it is going.</p> <p>I don’t write as often as I would like. I think I have set the bar too high in my mind in terms of what I should and shouldn’t write here. I have all manner of ideas for posts, but they never seem to make their way onto the proverbial paper.</p> <p>I get very self-critical about anything I write. In fact, its taken me about 10minutes to get this far - just over 100 words. I often ask myself why I’m like this. I think it is because I want every post to be a good one. I want anyone reading this blog to read one entry and feel like it is worth reading more.</p> <p>I need to get over this. I need to find my own style and stop trying to produce the perfect post every time. I need to recognise that finding my own style does not, and will not, happen overnight. I need to not be afraid of getting it wrong.</p> <p>Now, I could just delete what I have written here and carry on with my life. I could save it as a draft to keep it around as a reminder of these things. Or, I could publish it to the world and really mean it.</p> <p>Publishing it means I’ve got to think of a suitable title - do I go for my favoured wry sense of humour or a more obvious statement of the content. Give me 10 minutes and I’m sure something will come to mind.</p> Links for 2008-06-27 to 2008-08-13 2008-08-13T00:00:00+00:00 <ul> <li><a href="">Video: How Sheldon is created » Drawn! The Illustration and Cartooning Blog</a> -</li> <li><a href="">BBC NEWS ‘ Entertainment ‘ New role for BBC Two controller</a> - “BBC Two controller Roly Keating is to become the corporation's first director of archive content, responsible for improving public access to the BBC's constantly expanding archive of television, radio and multimedia content.”</li> <li><a href="">Tilted Twister - a Lego Mindstorms robot that solves Rubik’s cube</a> -</li> <li><a href=""> - Roo Reynolds and Nick O’Leary ‘ O’ReillyGMT</a> - Craig Smith recognised us at Open Tech 2008. Here is the video interview we did about</li> <li><a href="">Interesting 2008 ‘ From the Guardian ‘</a> - Videos from the day are being hosted by the Guardian.</li> </ul> DIY Ambient Orb 2008-07-19T00:00:00+00:00 <p>Take one garden light bought at M&amp;S for £3.50 and dissect it:</p> <p><a href="" title="Dissected Ambient Light by nol, on Flickr"><img src="" alt="Dissected Ambient Light" /></a></p> <p>Connect a BlinkM up to an arduino and set it running:</p> <p><a href="" title="BlinkM by nol, on Flickr"><img src="" alt="BlinkM" /></a></p> <p>Combine the two and you have a home-brew ambient orb:</p> <div style="text-align: center;"><object classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" width="437" height="370" id="viddler"><param name="movie" value="" /><param name="allowScriptAccess" value="always" /><param name="allowFullScreen" value="true" /><embed src="" width="437" height="370" type="application/x-shockwave-flash" allowscriptaccess="always" allowfullscreen="true" name="viddler" />&lt;/embed&gt;</object></div> <p>Next step, do something interesting with it.</p> <p>I’ve been on the lookout for suitable materials to make an ambient orb for a while - particularly something to diffuse the light. My original plan, which I may still do, was to take an ordinary lightbulb and put an RGB led inside it. However, modern lightbulbs prove quite tricky to take apart without shattering something. When I spotted these lights in M&amp;S last week I knew they were exactly what I wanted. So I bought three.</p> <p>They were pleasingly easy to dissect - just some gentle persuasion with a craft knife. The led’s they come with, which you can see <a href="">here</a>, are going to be handy to reuse in the future.</p> <p>I still need to work out how best to mount the BlinkM beneath it. Given their I2C interface, it is going to be very easy to chain lots of them together, working as a group.</p> <p>Ambient orbs are fascinating interfaces - they provide an abstraction that can convert an data source into a simplified, yet powerful, source of information.</p> <p>Converting data into information is something I have been meaning to write about for a while. But given it’s my wife’s birthday and we’re heading out for the evening in 5 minutes, that post will have to wait for another day.</p> Open Tech 2008 2008-07-07T00:00:00+00:00 <p>Roo and I presented at Open Tech 2008 this weekend on the Current Cost hacking that has been going on in these parts. It was my first time presenting to quite such a large audience and all things considered, it was a lot of fun and went very well.</p> <iframe src="//" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen=""> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//" title="Current Cost presentation at Open Tech 2008" target="_blank">Current Cost presentation at Open Tech 2008</a> </strong> from <strong><a href="" target="_blank">Roo Reynolds</a></strong> </div> <p>Although I took some notes through the day of the other talks I went to, Roo has done a much better write-up than I would. So I’ll point you <a href="">his way</a>.</p> Another little project 2008-07-07T00:00:00+00:00 <p>Having done <a href="/2008/06/30/a-little-project/">tetris</a> last week, I got a number of suggestions for improvements in the comments. Whilst I don’t rule out tackling them at some point, I had an itch to do something different. So I started the next project. Admittedly, this one has taken quiet a bit more time to do and it isn’t a finished end-to-end game like tetris is. But still, the concept is there and I could spend some more time on it or I could start the next project.</p> <p>So, for still no reason in particular, I present to you <a href="/games/scroller/">JavaScript Mario</a>. Enjoy.</p> <iframe width="100%" height="800px" border="0" src="/games/scroller/"></iframe> A little project 2008-06-30T00:00:00+00:00 <p>Thanks to spending a few hours first watching <a href="">Jay-Z at Glastonbury</a> and then the 5-set epic that was the <a href="">Murray-Gasquet</a> match at Wimbledon today, I have found myself sat in front of the laptop hacking away at a little project that came to mind late last week.</p> <p>I’m sure it has been done many time before, but as I’ve said before, reinventing the wheel can be quite fun sometimes.</p> <p>So, for no reason in particular, I present to you <a href="/games/tetris/">JavaScript Tetris</a>. Enjoy.</p> <iframe width="100%" height="800px" border="0" src="/games/tetris/"></iframe> Links for 2008-05-14 to 2008-06-23 2008-06-23T00:00:00+00:00 <ul> <li><a href="">russell davies: interesting timelapse</a> - A great summary of the day from the stage's point of view.</li> <li><a href="">Wii nunchuck data logger ‘ Windmeadow Labs</a> - Arduino based data logger for the Wii Nunchuck. Can store 21,000 readings from the Nunchuck.</li> <li><a href="">Wordle - Beautiful Word Clouds</a> - “a toy for generating “word clouds” from text that you provide. The clouds give greater prominence to words that appear more frequently in the source text.” [via:rooreynolds]</li> <li><a href="">She Went Of Her Own Accord</a> - My wife went to the Caribbean. Jamaica? No, she went of her own accord. Dot com. A weeks worth of hacking by Roo and myself.</li> <li><a href="">cityofsound: The Personal Well-Tempered Environment</a> - “Essentially, the idea is for a system that makes previously invisible aspects of people's behaviour visible, in order to help change individual and collective behaviour.”</li> </ul> Interesting2008 2008-06-22T00:00:00+00:00 <p><a href="" title="Russell Davies by nol, on Flickr"><img src="" alt="Russell Davies" /></a></p> <p>Yesterday was the awesome Interesting2008 organised by <a href="">Russell Davies</a>. Roo had told me all about last year’s Interesting2007, so I was very much look forward to it - and I wasn’t disappointed.</p> <p>The talks covered a diverse range of subjects and they were all interesting - as you would hope for such an event. Here are some of my highlights of the day.</p> <p><a href="" title="Roo Reynolds by nol, on Flickr"><img src="" alt="Roo Reynolds" /></a></p> <p><a href="">Roo</a> opened with 3 minutes on Lego. With 30 slides playing through behind him, he even remembered to breathe. A great start to the day. Roo has now <a href="">posted</a> his slides with audio on slideshare.</p> <div style="float: right; margin: 0px 20px;">[![Michael Johnson](]( "Michael Johnson by nol, on Flickr") [Michael Johnson]( gave us tour through the history of guitar music and graphic design. Was this just an excuse to play Jimi Hendrix on a stage in front of 300 people? [![Matt Dent](]( "Matt Dent by nol, on Flickr") [Matt Dent]( spoke about the new coin designs he did. Fascinating insight to the amount of detail that goes into this type of thing. [![Lloyd Davis](]( "Lloyd Davis by nol, on Flickr") [Lloyd Davis]( played the ukulele and then helped the whole room meditate and bring the mountain within. And relax. [![Anna Pickard](]( "Anna Pickard by nol, on Flickr") [Anna Pickard]( just about kept a straight face whilst talking about unintentionally funny words. I was most pleased to see Plinth on the list. [![Andrew Walkingshaw](]( "Andrew Walkingshaw by nol, on Flickr") [Andrew Walkingshaw]( explained why naming things is hard. [![Jim Le Fevre](]( "Jim Le Fevre by nol, on Flickr") [Jim Le Fevre]( amazed us all by showing off his turntable-based zoetrope. I could have watched this for hours. Definitely something to play with in the future. _Update:_ [Russell]( has linked to a YouTube video of this stuff - amazing. Russell did a great job of organising the day - I'm very much looking forward to Interesting2009. [![Russell Davies](]( "Russell Davies by nol, on Flickr") </div> Words with style 2008-06-14T00:00:00+00:00 <p><a href="/blog/content/2008/06/wordle.png"><img src="/blog/content/2008/06/wordle-300x172.png" alt="" title="wordle" /></a></p> <p>From Roo’s <a href=""> stream</a>, I came across <a href="">Wordle</a> by IBM Research’s Jonathan Feinberg.</p> <p>It is “a toy for generating ‘word clouds’ from text that you provide”. The results it produces are absolutely stunning. Tag clouds are quite a common thing these days - it would take a couple clicks to add one in the sidebar of this blog. But they are often quite dull. Wordle shows this doesn’t have to be the case.</p> <p>The site lets you specify a list of words to generate the cloud from, or it will do it from your tags. You can see my tags in the cloud above.</p> <p>At the moment it is a java applet - but it would be great to see it run as a web service so these clouds can be integrated more readily.</p> Three things: Linux, Vegetables and swohoa 2008-06-07T00:00:00+00:00 <p>It has been a bit quiet around here recently, which is not to say I haven’t been up to stuff.</p> <p>Last week I was involved with an event at work for ‘showing off’ interesting things around linux. I was signed up to demo the N800 internet tablet, but I stretched the definition a bit and had my arduinos out as well. Ian took some photos of the event. <a href="">Here</a> you can see talking to someone about the arduino’s whilst <a href="">James</a> plays with the Wiimote paired with my N800 and <a href="">here</a> you can see my rather hastily built custom Wii Sensor bar. I’m glad I had my box of arduino bits with that morning, otherwise I would have been stuck! For more on what else was going on, check out Graham’s <a href="">write-up</a> of the event.</p> <p><a href="" title="Veg Beds by nol, on Flickr"><img src="" alt="Veg Beds" /></a></p> <p>A couple weekends ago, we finally got the veg beds planted. Click through to flickr for a very well annotated list of what we planted. Sadly, last night the neighbour’s blasted cat dug up half our carrot seedlings. I’ve laid out some temporary netting and this weekend I’ll be building a better cat <strike>trap</strike> defense.</p> <p>The third thing I wanted to mention was the <a href="">Super Secret Side project</a> that Roo and I have been hacking on for the last week: <a href=""></a>.</p> <p>This is an idea that has been bouncing around between us for years - but only now have we done something about it. It is essentially a (growing) collection of the most pun-tastic jokes out there for all to share. After a fairly frantic week of hacking in the evenings, we managed to coax together enough of a site to reveal it today. We still have plenty of features to add, so keep an eye out for updates.</p> Google Treasure Hunt 2008-05-19T00:00:00+00:00 <p>Earlier today, I spotted a tweet from <a href="">@thomasj</a> about the <a href="">Google Treasure Hunt</a>. This was followed later in the day by a <a href="">series</a> <a href="">of</a> <a href="">tweets</a> from <a href="">@graham_alton</a> (warning: spoiler in tweets!) that picqued enough interest that I decided to have a go.</p> <p>In summary, the challenge is to state how many unqiue paths there are across an arbitrarily sized chessboard from the top-left corner to the bottom-right corner, but only moving down or right at each step. Graham has written up his solution, along with the original question on his <a href="">blog</a>.</p> <p>He has gone straight for the proper solution using a simple formula… and some perl to handle the fact the numbers involved are beyond most desk calculators.</p> <p>I decided to go for a bit more of a brute force approach… and some python to handle the fact I haven’t touched perl in a while.</p> <p>Here’s my solution:</p> <blockquote> <pre>#!/usr/bin/python # the dimensions of the board w=54 h=30 lr = [1]*w for y in range(1,h): lc = 1 for x in range(1,w): lc = lc + lr[x] lr[x] = lc print lc</pre> </blockquote> <p>The secret here is that for any given square on the board, the number of unique paths is equal to the sum of the number of unique paths from the squares immediately to the right and below.</p> <p>Here’s an attempt to express it slightly more formally:</p> <blockquote> <p>f(x,y) = f(x+1,y)+f(x,y+1)</p> <p>where f(w,y) = f(x,h) = 1</p> </blockquote> <p>The next question of the treasure hunt is due any minute now…</p> Links for 2008-05-06 to 2008-05-13 2008-05-13T00:00:00+00:00 <ul> <li><a href="">Official Google Reader Blog: Brand new Google Reader for iPhone</a> - Look forward to trying this on the N800 which does struggle with the standard reader interface.</li> <li><a href="">MAKE: Blog: Arduino wifi shield</a> - The first wifi shield I've seen. But at $295, not something I'm going to rush out to get.</li> <li><a href="">CrunchBang ~ GIMP Lomo Filter</a> - Nice lomo effect filter for GIMP</li> <li><a href="">Open Tech 2008 - 5th July in London.</a> - “Open Tech 2008 is an informal, low cost one-day conference on technology, society and low-carbon living, featuring Open Source ways of working and technologies that anyone can have a go at.”</li> <li><a href=";products_id=4&amp;zenid=f4a809dbb7bbdf8de855ded5dd7452e9">Ethernet Shield V1.0 for Arduino Arduino Etherenet Shield with ENC28J60 SPI [] - £12.99 :, new idea, new project, new electronics</a> - Another ethernet shield for an Arduino. Cheaper than the XPort based one, but not as powerful out of the box. Am currently working on an improved TCP stack for it.</li> </ul> Links for 2008-03-20 to 2008-04-23 2008-05-06T00:00:00+00:00 <ul> <li><a href="">MAKE: Blog: Open GPS tracker</a> - A low cost, simple, GPS tracker</li> <li><a href="">pachube :: connecting environments, patching the planet -</a> - “a service that enables people to tag and share real time environmental data from objects, devices and spaces around the world. The key aim is to facilitate interaction between remote environments, both physical and virtual.”</li> <li><a href="">xkcd - A webcomic of romance, sarcasm, math, and language - By Randall Munroe</a> - “I've taught it obstacle avoidance and blogging.”</li> <li><a href="">Milliways: Infocom's Unreleased Sequel to Hitchhiker's Guide to the Galaxy -</a> - Fascinating background on the game that wasn't to be. The comments are worth reading, as my of the parties involved have commented - not all happy about the article.</li> <li><a href="">Home Easy - SAVE your money, by reducing ‘Standby’ power and wasted electricity.</a> - The range of home automation products that B&amp;Q sells.</li> <li><a href="">Ethernet shield for Arduino</a> - Ethernet enabling an arduino - including running a webserver on the board. Interesting possibilities for tweetjects. <em>Update</em>: odd that I bookmarked this one, and not the <a href=";products_id=4">one</a> I actually bought.</li> <li><a href="">BBC NEWS ‘ Magazine ‘ World’s best-known protest symbol turns 50</a> - Interesting history to the CND logo. Never knew it was based on the semaphore for N and D.</li> <li><a href="">We Tell Stories - ‘The 21 Steps’, by Charles Cumming</a> -</li> </ul> Power Graphing 2008-05-05T00:00:00+00:00 <p>Since getting my [ meter online, it has been sending its readings to a server in the sky so charts like <a href="">these](/2008/04/10/going-power-crazy/) can be produced. To really play with the data, I needed to start logging it locally and producing my own graphs. Here is a rambling rundown of how I got from XML being spat out by the device every 6 seconds to something like this:</a></p> <p><img src=",20,1,7&amp;chxt=y,x&amp;chxl=0:'0'1'2'3'4'5'1:'2100'2300'0100'0300'0500'0700'0900'1100'1300'1500'1700'1900&amp;chxp=1,2,11,19,27,36,44,53,61,69,78,86,95&amp;chs=400x200&amp;cht=lc&amp;chds=0,5&amp;chm=B,f3f3f3,0,0,0&amp;chco=aaaaff&amp;chd=t:0.348,0.348,0.319,0.311,1.285,0.683,0.338,0.349,0.398,0.39,0.356,0.477,0.274,0.264,0.572,0.637,0.406,0.382,0.368,0.324,0.325,0.917,0.322,0.358,0.342,0.318,0.191,0.124,0.209,0.204,0.192,0.213,0.205,0.196,0.374,1.753,1.537,0.976,0.552,0.556,0.513,0.433,0.363,0.341,0.575,0.46,0.188,0.123,0.248,0.204,0.195,0.116,0.215,0.207,0.226,1.574,0.636,0.214,0.206,0.209,0.204,0.648,0.694,0.628,0.599,0.689,0.743,1.222,0.278,0.394,0.326,0.394,0.225,0.184,0.13,0.145,0.23,0.226,0.222,0.133,0.133,0.237,0.231,0.223,0.13,0.234,0.229,0.222,0.203,0.197,0.193,0.17,0.198,0.176,0.275,0.328,0.287,0.212,0.159,0.192,0.19,0.212,0.292,0.298,0.418,0.332,0.789,0.894,0.719" alt="" /></p> <h3 id="step-one---parsing-the-data">Step One - Parsing the data</h3> <p>This was the easy part; AndySC had already put together a perl script for reading the serial port and doing the necessary parsing.</p> <h3 id="step-two---publishing-the-data">Step Two - Publishing the data</h3> <p>Again, not much for me to do here as Andy’s script already publishes the data to a set of topics over <a href="">MQTT</a>.</p> <h3 id="step-three---logging-the-data">Step Three - Logging the data</h3> <p>Finally, something for me to do. A couple years ago, I would have joined Andy’s perl script with one of my own, but python is more my thing these days. I already had a piece of python that subscribes to the appropriate topics and posts the temperature values to <a href="">twitter</a>. It didn’t take much to get the same script to subscribe to the power data and dump it into a MySQL database on the local machine.</p> <blockquote> <pre>+----------------+---------+ ' when ' power ' +----------------+---------+ ' 20080502231747 ' 0.34 ' ' 20080502231753 ' 0.33 ' ' 20080502231804 ' 0.34 ' +----------------+---------+ </pre> </blockquote> <p>The table in the database is a very simple one at the moment; logging the power along with a timestamp. An entry is added to the table for every reading from the meter and having been running for 3 days there are just over 11,000 of them. I’ll have to keep an eye on this to make sure it doesn’t run away with my free disk space.</p> <h3 id="step-four---graphing-the-data">Step Four - Graphing the data</h3> <p><a href="">Roo</a> showed me some stuff he’s been playing with using the <a href="">Google Chart API</a>. Whilst I generally prefer to roll my own (aka, reinvent the wheel), I couldn’t ignore just how easy it is to produce pretty graphs this way.</p> <p>Before delving into the API, I needed to decide just what I wanted to produce. As there is such a range of chart types available, there are plenty of interesting things that could be done. Initially, however, I decided to stick with the traditional “power-usage-in-the-last-24-hours” chart.</p> <p>The API has a limit on the amount of data that can be passed to it. So I needed to find a meaningful way to reduce the 4200 data points generated in 24 hours to around 100 at most. I soon settled on using the average value for each 10 minute period. This loses some resolution in the data, but it still shows the trends.</p> <p>Generating the averages is a simple question of the right query on the database. With some trial and error, I eventually got to:</p> <blockquote> <p><code class="language-plaintext highlighter-rouge">select concat(substring(substring(</code>when<code class="language-plaintext highlighter-rouge">,1,11),9,4),'0'), truncate(avg(</code>power<code class="language-plaintext highlighter-rouge">),3) from currentcost where </code>when<code class="language-plaintext highlighter-rouge"> &gt; SUBDATE(NOW(), INTERVAL 1 DAY) group by substring(</code>when<code class="language-plaintext highlighter-rouge">,1,11) order by </code>when<code class="language-plaintext highlighter-rouge">;</code></p> </blockquote> <p>Simple huh?</p> <p>This produces results like this:</p> <blockquote> <pre>+------+-------+ ' 2240 ' 0.714 ' ' 2250 ' 0.637 ' ' 2300 ' 0.406 ' +------+-------+</pre> </blockquote> <p>With that in place, all it took was to throw it together into an appropriate URL for google to generate the image:</p> <blockquote> <p>,20,1,7&amp;chxt=y,x&amp;</p> <p>chxl=0:’0’1’2’3’4’5’ 1:’2100’2300’0100’0300’0500’0700’0900’1100’1300’1500’1700’1900&amp; chxp=1,2,11,19,27,36,44,53,61,69,78,86,95&amp;chs=400x200 &amp;cht=lc&amp;chds=0,5&amp;chm=B,f3f3f3,0,0,0&amp;chco=aaaaff&amp; chd=t:0.348,0.348,0.319,0.311,1.285,0.683,0.338,0.349, 0.398,0.39,0.356,0.477,0.274,0.264,0.572,0.637,0.406, 0.382,0.368,0.324,0.325,0.917,0.322,0.358,0.342,0.318, 0.191,0.124,0.209,0.204,0.192,0.213,0.205,0.196,0.374, 1.753,1.537,0.976,0.552,0.556,0.513,0.433,0.363,0.341, 0.575,0.46,0.188,0.123,0.248,0.204,0.195,0.116,0.215, 0.207,0.226,1.574,0.636,0.214,0.206,0.209,0.204,0.648, 0.694,0.628,0.599,0.689,0.743,1.222,0.278,0.394,0.326, 0.394,0.225,0.184,0.13,0.145,0.23,0.226,0.222,0.133, 0.133,0.237,0.231,0.223,0.13,0.234,0.229,0.222,0.203, 0.197,0.193,0.17,0.198,0.176,0.275,0.328,0.287,0.212, 0.159,0.192,0.19,0.212,0.292,0.298,0.418,0.332,0.789,0.894,0.719</p> </blockquote> <p>Again, simple huh? Well, maybe not so much. Here’s a break down of what that does (and just to confuse matters, this is in a different order to where things appear in the url above…)</p> <blockquote> <p> This is base url of the Google Charts api - all the magic comes from here.</p> </blockquote> <blockquote> <p>chs=400x200 Set the size of the image.</p> </blockquote> <blockquote> <p>&amp;cht=lc Set the type of chart - a line chart.</p> </blockquote> <blockquote> <p>&amp;chg=100,20,1,7 Gives the chart a grid in the background.</p> </blockquote> <blockquote> <p>&amp;chxt=y,x &amp;chxl=0:’0’1’2’3’4’5’ 1:’2100’2300’0100’0300’0500’0700’0900’1100’1300’1500’1700’1900 &amp;chxp=1,2,11,19,27,36,44,53,61,69,78,86,95 Describes the axis labels. 0 to 5 on the y-axis, and the relevant times along the x-axis. The script figures out approximate positions along the axis for the labels.</p> </blockquote> <blockquote> <p>&amp;chds=0,5 Specifies the minimum and maximum values for the data - although I occasionally go over 5Kw, I decided it wasn’t worth squeezing the data for 99% of the time when it is below that.</p> </blockquote> <blockquote> <p>&amp;chm=B,f3f3f3,0,0,0 Fills the area under the line with a light gray.</p> </blockquote> <blockquote> <p>&amp;chco=aaaaff Draws the line with a light blue.</p> </blockquote> <blockquote> <p>&amp;chd=t:0.348,0.348,0.319,0.311,1.285,0.683,0.338,0.349, 0.398,0.39,0.356,0.477,0.274,0.264,0.572,0.637,0.406, 0.382,0.368,0.324,0.325,0.917,0.322,0.358,0.342,0.318, 0.191,0.124,0.209,0.204,0.192,0.213,0.205,0.196,0.374, 1.753,1.537,0.976,0.552,0.556,0.513,0.433,0.363,0.341, 0.575,0.46,0.188,0.123,0.248,0.204,0.195,0.116,0.215, 0.207,0.226,1.574,0.636,0.214,0.206,0.209,0.204,0.648, 0.694,0.628,0.599,0.689,0.743,1.222,0.278,0.394,0.326, 0.394,0.225,0.184,0.13,0.145,0.23,0.226,0.222,0.133, 0.133,0.237,0.231,0.223,0.13,0.234,0.229,0.222,0.203, 0.197,0.193,0.17,0.198,0.176,0.275,0.328,0.287,0.212, 0.159,0.192,0.19,0.212,0.292,0.298,0.418,0.332,0.789,0.894,0.719 Specifies the data points.</p> </blockquote> <h3 id="what-next">What next?</h3> <ul> <li><a href="">Chris</a> has started doing some interesting data analysis to see if he can automatically spot ‘events’ on the graph. Will be interesting to see what can be achieved here.</li> <li>Andy’s <a href="">twittering house</a> got some linkage last week from both <a href="">Earth2Tech</a> and <a href="">Wired Science</a>. They mention the power orb that was written about <a href="">last year</a>. I really like the idea of an ambient device for displaying this information - another project for the arduino list.</li> </ul> Turning an Arduino into a Tweetject 2008-04-24T00:00:00+00:00 <p>A couple weeks ago I bought an <a href=";products_id=4">Ethernet shield</a> for my <a href="">arduino</a> with a plan to get it connecting to the world directly. The stripped down TCP implementation that comes with it is much more geared towards making the board act as a web-server rather than client. This means you have to poll for information rather than have the arduino send it out. Unfortunately, it is this second case that I am most interested in.</p> <p>Julian Bleecker’s paper <a href="">“Why Things Matter”</a> introduced the term “blogjects” - objects that blog. A natural extension of which are “tweetjects” - objects that twitter. A good example being <a href="">Tower Bridge</a> which twitters whenever the bridge opens or closes and announces which ship is passing by. (Roo has written more on this stuff <a href="">here</a>.)</p> <p>I see an ethernet-enabled arduino as a perfect platform for building such objects from. But to do this, it needs to establish the outgoing connection itself.</p> <p>There is <a href="">another</a> shield that provides a much more powerful option based on the Lantronix XPort ethernet hardware. It even has a <a href="">demo sketch</a> that will post to twitter. The advantage of this shield is that it provides a serial interface to run the connection - putting much less strain on the arduino. However, this power comes at a cost - the XPort itself is around $50, whereas the Nuelectronics shield is £12.99 already assembled.</p> <p>So for now, I am persevering with the cheaper board and have been learning far more about TCP packets than I ever thought necessary. So far, I have got my board doing an ARP request to map IP to MAC address, establishing a TCP connection and then sending a single blob of data (naturally, “HELLO WORLD”). This is all a couple layers lower down in the <a href="">OSI model</a> than I am used to working with in my day job.</p> <p>It is still a bit too hard-coded to be easily reused and attempts to re-factor what I’ve done has broken it all; so I still have lots to do.</p> <p>One thing I have discovered whilst playing with this all is <a href="">Wireshark</a> - a superb tool for sniffing packets on the network and examining their content. This has helped a lot in working out which bits and bytes are going awry in my packets.</p> <p>DIY and decorating allowing, I hope to make progress on this over the next couple weeks.</p> Getting the doorbell online 2008-04-13T00:00:00+00:00 <p>Following <a href="">Andy’s</a> example, I have setup a twitter account for my own <a href="">house</a> to share its thoughts with the world on.</p> <p>So far, it is busy twittering the changing temperature of my living room, as measured by the <a href="/2008/04/10/going-power-crazy/">CurrentCost meter</a>. At the moment it only tweets when the temperature changes at least 10 minutes since the previous change. This reduces some of the noise, but I think there is more to do. I don’t necessarily care if the living room is 16°C rather than 17°C but do care if it is too cold or hot. This is a subject I will come back to another time once I have done some more experiementing.</p> <p>More immediately, I set myself a challenge this weekend; to get the doorbell twittering.</p> <p>To begin with I bought a cheap wireless doorbell set from Wickes to play with. This came with two receivers; a portable battery-powered one and a mains-powered one. Wasting no time I took one of them apart to find this:</p> <p><img src="/blog/content/2008/04/doorbell1.jpg" alt="" title="doorbell circuit" /></p> <p>The circuit has a very conveniently packaged daughterboard containing the wireless receiver. The clue was the aerial connecting to it (the white wire to the right) and even better, in the top left corner are the four pins connecting to the rest of the circuit. With my trusty multimeter, and the labels printed on the board, it didn’t take long to work out the leftmost pin is ground, the rightmost pin is +3v and the other two are the magic data pins.</p> <p>With my newly acquired <a href="">arduino</a>, itself a topic for another day, I hooked-up the four pins and fairly instantly had the arduino writing over its serial link to my laptop whenever the doorbell button is pushed. Of the two data pins, I found the one labeled “DAT” is the best trigger to use; the other one, “IDEL”, seems to be more noisy and needs investigating.</p> <p>The doorbell lets you pick from one of 15 channels to ensure you don’t get interference from the neighbours. It also lets you pick one of three chimes to use, from the traditional ding-dong to the full Big Ben. As that setting is on the button itself, it must be sending it over the wireless signal. Currently my circuit triggers regardless of the channel setting of the button. I assume I need to do some more signal analysis on both the DAT and IDEL pins to figure this part out. For now, it works enough to prove the idea.</p> <p>Here is the sketch I used. It does some simple debouncing by not triggering twice within 3 seconds.</p> <pre> int potPin = 5; // Connected to 'DAT' int val = 0; long time = 0; long debounce = 3000; void setup() { Serial.begin(9600); } void loop() { val = analogRead(potPin); if (val &gt; 0) { if (millis()-time &gt; debounce) { Serial.print("ON:"); time = millis(); } } } </pre> <p>From this point, a bit of python and mqtt magic and the doorbell would be twittering. I say “would” as I haven’t done this final piece of plumbing yet. I only have one arduino at the moment and I am not yet ready to dedicate it to any one task. Clearly I need to order a second arduino - its always good to have separate development and production systems.</p> Going power crazy 2008-04-10T00:00:00+00:00 <p>As a part of the <a href="">“CurrentCost craze”</a> that is rife at Hursley these days, I’ve had mine up and running for a few days and am very excited by the possibilities.</p> <p>The <a href="">CurrentCost</a> meter gives you a real-time display of how much electricity your house is using. On its own, this would be interesting, but hard to relate to anything. One of its cool features is its awareness of how much electricity actually costs - being told your usage costs £17 per month is much more effective than saying you’re using 1.23kw.</p> <p>I’m not the only one who has got this meter setup and then obsessively gone around the house to reduce how much is being wasted.</p> <p>What is great about this particular device, and what appeals to the Hursley crowd, is the ability to plug it into a PC and capture the information it produces. It doesn’t take much plumbing to get the data being graphed online - something which does raise an interesting issue.</p> <p>I can now see when Jo has got home from work as I see the spike of the kettle being used, but equally, <em>anyone</em> can get a pretty good idea if we’re in the house. Is that a genuine concern? There are certainly easier ways of working out if we’re in or not.</p> <p>That aside, the graph it produces definitely does give food for thought. Here’s a snapshot of the graph from today (click through for a fully annotated version on flickr): <a href="" title="Power Usage by nol, on Flickr"><img src="" alt="Power Usage" /></a></p> <p>This graph has left me intrigued to find out what the 100watt-ish blip is that happens every 2 hours; there is a PC running all the time, which has the meter connected - I wonder what <a href="">cron</a> tasks are configured.</p> <p>The meter has definitely left me feeling slightly guilty for the 600watts of halogen spot-lights we put in the new kitchen.</p> A new home 2008-03-30T00:00:00+00:00 <p>As the server hosting this blog is soon to disappear, I have been migrating the site to a new host. With any luck, the fact you are reading this post means the DNS updates have happened and the new host is live.</p> <p>For those visiting the site rather than subscribed to the feed, you’ll notice a return to the default theme around here. The knolleary theme was hand-crafted for an old version of Wordpress and with the move of host I’ve taken the opportunity to upgrade to the newly released Wordpress 2.5 as well. I will update the theme sometime soon.</p> <p>[update] after a slighty bumpy day of email back-and-forth with my new hosting company, I think we’re up and running. Still a few broken links; my uploads directory seems to have moved so a handful of posts are missing images, and I haven’t got my planet installed and running on the new server.</p> <p>Also, just discover a problem with writing posts with links in them; Roo hit this with his blog and is something to do with a mod_security setting. Here we go with some more email tennis wih support…</p> <p>[update2] it <em>should</em> now let me post html.</p> Links for 2008-02-11 to 2008-02-20 2008-02-21T00:00:00+00:00 <ul> <li><a href="">The “Blog” of “Unnecessary” Quotation Marks</a> - “Great” stuff.</li> <li><a href="">Nifty Corners Cube - freedom to round</a> - Javascript library for adding rounded corners to elements on a page. Some very Nifty examples.</li> <li><a href="">BBC - Radio Labs - Now Playing in the Cloud</a> - Using XMPP to be notified of what is being played on BBC radio stations. It will be interesting to see what else is done with this. XMPP is being used to glue more and more parts of the internet together these days.</li> <li><a href="">FreakAngels</a> - “FREAKANGELS is a free, weekly, ongoing comic written by Warren Ellis and illustrated by Paul Duffield.”</li> <li><a href="">Five Chinese Crackers: Misquoting the Archbishop</a> - 5CC does a great job of pointing out the flagrant misreporting in the tabloid press. The incorrect reporting of the Archbishop of Canterbury’s speech from last week is a great example. For all those calling for him to resign, I wonder how many have actual</li> </ul> Playing (and winning) the MP3 game 2008-02-13T00:00:00+00:00 <p>For a long time there hasn’t been a viable option for linux users to buy main-stream DRM-free music in the UK. I have looked at the various independent stores out there, but they simply don’t have the music I want. I have been eagerly waiting for Amazon to launch its MP3 store - something they recently <a href="">confirmed</a> would happen at some point this year.</p> <p><img src="/blog/content/2008/02/adele-best-for-last-album.thumbnail.jpg" alt="Adele - 19" /></p> <p>But today, out of nowhere, have just launched their own <a href="">music store</a>. That’s right - I didn’t say ‘announced’ or ‘promised’; they have actually launched it today and I have bought my very first MP3 album (for the record, Adele’s 19).</p> <p>Their <a href=";r=HOME">FAQ</a> have all the gory details; music is availble at either 192Kbit/s or 320Kbit/s depending on the recording, can be redownloaded a (limited) number of times and are unencumbered of any DRM. They only have content from EMI and some independent labels at the moment, but this is certainly a good start.</p> <p>According to <a href="">The Register</a>, where I first heard the news, all tracks will be 320Kbit/s eventually, with the average price being 70p/track or £6.99/album.</p> <p>All considered, play have won the race as far as I am concerned. Amazon will have to do something pretty special to make me look their way.</p> The mind of a 7-year old 2008-02-03T00:00:00+00:00 <blockquote> <p><strong>Ben, a 7-year old:</strong> What do you call Darth Vader kissing Luke Skywalker?</p> <p><strong>Me:</strong> I have no idea.</p> <p><strong>Ben:</strong> Luke Darth Vader</p> <p><strong>Me:</strong> What?</p> <p><strong>Ben:</strong> <em>in complete hysterics</em></p> <p><strong>Me:</strong> What?!</p> <p><strong>Ben’s Dad:</strong> Don’t ask. It only gets more random from here.</p> <p><strong>Me:</strong> What?!!</p> </blockquote> Pure Evoke-1S 2008-02-02T00:00:00+00:00 <p><a href="" title="photo sharing"><img src="" alt="" /></a> <a href="">Pure Evoke-1S</a></p> <p>Jo and I didn’t do a good job of buying each other Christmas presents last year, so we agreed to by ourselves a joint something or other. We soon settled on a digital radio for the newly refurbished kitchen.</p> <p>Before rushing out to buy one, we borrowed one from my parents so we could see if the reception in our kitchen was good enough. They have no reception where they are in Devon so weren’t doing much with it.</p> <p>When I first tried it a couple weeks ago all I seemed to get was Hampshire Radio. That was good enough for some background music whilst cooking, but didn’t offer quite the range of stations we wanted. What it did prove was how often we would listen to the radio if it worked.</p> <p>This morning I moved the radio to the window sill in the kitchen and suddenly found it could receive all the stations we could possibly want. With that discovery, we headed into town to finally buy our present; only a month and a bit late.</p> <p>We had already picked out the one we wanted; the Pure range of radios have always caught my eye as objects with some style and the Evoke-1S is the perfect radio to have in the kitchen. The only short coming is the single speaker; the mono-sound was noticeable after the stereo of my parent’s borrowed radio. However, Pure do sell an optional added speaker that we held off from buying initially, but will be getting the next time we’re in town.</p> Links for 2008-01-10 to 2008-01-27 2008-01-28T00:00:00+00:00 <ul> <li><a href="">Amazon MP3 store to go global - Boing Boing</a> - Confirmation that Amazon’s MP3 store will be rolling out this side of the pond.</li> <li><a href=";hl=en">Randy Pausch’s Last Lecture</a> - I can’t think of enough words to describe this. Powerful and moving don’t begin to cover it.</li> <li><a href="">Tetrical - 3D Tetris Stacking Game Developed by Digital Machina</a> - 3D tetris - what more can I say?</li> <li><a href="">Evan Davis: How interest rates work</a> - Evan does a good job of explaining the economics of interest rates. It’s simple stuff, but I always struggle to make it stick in my head.</li> <li><a href="">Scripted Web Frameworks, and why Django Got it Right</a> - “The problem I started to run into fairly early on was I didn’t know the ‘Ruby on Rails’ way to do it, but knew perfectly well how I might have done it in a chitzy CGI.” Interesting piece on why Django was a better fit for one person. I am inclined to ag</li> </ul> Links for 2008-01-01 to 2008-01-08 2008-01-09T00:00:00+00:00 <ul> <li><a href="">Gaming the Creative Commons for Profit</a> - Fascinating explanation of how a CC license is the perfect tool for someone wanting to make money suing over copyright infringement. As a non-professional, not-out-to-scam, hobbyist photographer, I’m happy to stick with CC.</li> <li><a href="">Amazon S3 plugin for WordPress : tan tan noodles - msg free since 2005</a> - A plugin that seamlessly integrates S3 into Wordpress. WIll be interesting to see if this works over the xml-rpc upload mechanism as well.</li> <li><a href="">Black Bird ‘ Progetti » WordPress Library</a> - A python library for accessing Wordpress installations over XML-RPC.</li> <li><a href=";show=sinfest">;show=sinfest</a> - The true meaning of the New Year.</li> <li><a href="">Dave Gorman: Problem Solving.</a> - Dave proposes a solution to the “a month isn’t exactly 4 weeks” problem. I for one look forward to Gormanuary and Intermission ‘08.</li> </ul> Hello 2008 2007-12-31T00:00:00+00:00 <p>With 2007 fast coming to a close, I thought it would be a good time to come up with some plans for 2008. This is not a comprehensive list; I hope to achieve a bit more than just this, but it will be interesting to look back a year from now to see how I got on. It’s probably also obvious this list reflects some of the immediate things on my mind (my N800 should arrive later this week and Zelda on the DS arrived for Christmas).</p> <ul> <li>Hack around with the N800 - lots of thoughts on this, too many for this post.</li> <li>Get the MythTV box fully up to date - its running an ancient version on a hand-built Debian kernel and X server which makes any update near impossible to apply. Fortunately, a stock Ubuntu these days has all of the hardware support that was missing 4 (!) years ago.</li> <li>Finish Zelda on DS and Wii - progress on the Wii has been held up by kitchen renovations. Zelda on the DS is my favourite game for a very long time, although it will have stiff competition from Super Mario Galaxy when I finally get around to buying it.</li> <li>Renovate the bathroom - how hard can it be to move a doorway, build a walk-in shower and re-plumb the entire thing?</li> <li>Decorate the dining room and living room - say goodbye to the Artex ceilings and walls.</li> <li>Read more of the books I should have read</li> <li>Take more photos.</li> <li>Blog more - I’ve lost the momentum over the past few months. It feels as good a time as any to pick it up again.</li> <li>Visit our friends in Scotland and the Shetlands - that really is a long way North.</li> </ul> <p>As for New Year’s resolutions, well, I’ve got a couple more hours to think of them.</p> Links for 2007-12-27 2007-12-28T00:00:00+00:00 <ul> <li><a href="">tinker store</a> - UK source of arduino boards.</li> <li><a href="">3D LED Charlieplex Cube from Chrismas Tree Lights - The World’s Biggest Show &amp; Tell - tech, diy</a> - Another LED cube project; this uses charlieplexing to provide control over the LEDs.</li> <li><a href="">DVD playback now standard for Dell Ubuntu customers</a> - Mark shares the news that the pre-installed Ubuntu image comes with DVD playback as standard. No word on the app used, but it is a good start.</li> <li><a href="">Howto : Scratchbox + Maemo 4.0 + Ubuntu Gutsy = Nokia N800 Dev Environment</a> - Howto setup the Nokia N800 Dev environment using: Scratchbox + Maemo 4.0 + Ubuntu Gutsy Nokia N800 Dev Environment. Now the N800 is £129.99 on <a href=";searchsource=0"></a>, this may soon be something I’ll want to play with.</li> <li><a href="">GETSful</a> - Joe provides the term for sites that aim for RESTful, but miss.</li> </ul> <p /> <p><em>Note to self: now these are being auto-blogged, make more of an effort to add useful descriptions!</em></p> Hillier in the Autumn 2007-10-14T00:00:00+00:00 <p><a href=""><img src="" alt="Green" /></a><a href=""><img src="" alt="Orange" /></a><br /> <a href=""><img src="" alt="Yellow" /></a><a href=""><img src="" alt="White" /></a></p> <p>Hillier Gardens in the Autumn.</p> Adam and Helena 2007-08-14T00:00:00+00:00 <p><a href=""><img src="" alt="Staying still" /></a><a href=""><img src="" alt="Fire!" /></a><br /> <a href=""><img src="" alt="Lean on me" /></a><a href=""><img src="" alt="Adam and Helena" /></a><br /> <a href=""><img src="" alt="Three Man and a Wall" /></a><a href=""><img src="" alt="Tall Trees" /></a></p> <p>Adam and Helena’s Wedding.</p> <p>Lots more from our three days in Germany in the <a href="">set</a>.</p> <p><em>[update]</em> As Roo points out in the comments, there is now a <a href="">flickr group</a> to enjoy.</p> Catherine and Scott 2007-08-05T00:00:00+00:00 <p><a href=""><img src="" alt="Catherine and Scott" /></a><br /> <a href=""><img src="" alt="Scott" /></a><a href=""><img src="" alt="Catherine" /></a><br /> <a href=""><img src="" alt="Matrimonial Camper Van" /></a></p> <p>Catherine and Scott’s wedding.</p> <p>Playing with my new lens adapter for the S3, using a Hoya polarizing circular filter.</p> Paris 2007-07-19T00:00:00+00:00 <p><a href="" title="Photo Sharing"><img src="" alt="Louvre Pyramid" /></a><br /> <a href=""><img src="" alt="Eiffel" /></a><a href=""><img src="" alt="Cite Metro" /></a><a href=""><img src="" alt="Waterlilly Detail" /></a><a href=""><img src="" alt="Metropolitain" /></a></p> <p>A long weekend in Paris for Jo’s birthday.</p> <p>More photos in the <a href="">set</a>.</p> A couple days in the life 2007-06-09T00:00:00+00:00 <p><a href=""><img src="" alt="Misty Door" /></a><a href=""><img src="" alt="Pebble Dashed" /></a><a href=""><img src="" alt="Floating Cliffs" /></a><a href=""><img src="" alt="Durdle Door" /></a></p> <p>A foggy day at Durdle Door.</p> <p><a href=""><img src="" alt="Watching Bird" /></a><a href=""><img src="" alt="Lens Vignette" /></a><a href=""><img src="" alt="Looking out" /></a></p> <p>Playing with the Canon TC DC58B Tele Lens</p> Wedding Album 2007-06-08T00:00:00+00:00 <p>Only 7 months after the wedding, we have finally gotten around to putting the ‘official’ photos online. These are a combination of photos taken by my Dad and ones added by our friends to the <a href="">flickr pool</a>. The hard-copy of the album was done at Christmas, but it has taken a while to figure out what we wanted to do online; there didn’t seem to be an obvious choice for maintaining the look and feel of the actual thing. So, enjoy some hand-crafted html and javascript - would you expect anything less?</p> <p><a href="">Jo &amp; Nick’s Wedding - the album</a></p> S3, EC2, SQS - The AWS Triumvirate 2007-06-05T00:00:00+00:00 <p><img src="/blog/content/2007/06/aws.gif" alt="Amazon Web Services" /></p> <p>The Amazon Web Services have been on my radar for a while now, but a recent talk at Hursley by <a href="">Jeff Barr</a> renewed my resolve to have a play. Jeff works in Amazon’s Web Services Evangelism team and if there is one thing Jeff does well, it’s evangelism. I left his talk impressed by how they are delivering the on-demand vision for web services that IBM has talked about for so long.</p> <p>They offer a global storage architecture with S3, scalable computing capacity with EC2 and web-scale messaging with SQS. This full house of web services gives you most of what you want to build highly scalable systems without having to invest in hardware. Jeff cited an example of a podcasting site, whose name I forget, that managed to get their IT infrastructure in place for $100 in one week. They have no hardware costs, no maintenance, no 2am wake-ups when a hard-disk fails.</p> <p>Whilst I don’t currently have a need for a scalable-web-architecture of my own, I have been looking at the storage side of things to provide some remote backup to my photo collection. At the moment they are dotted across various machines with little backup or redundancy. I had considered getting a <a href="">NSLU2</a> with some disks attached, but whilst I don’t rule out this option for the future, with a house move imminent I decided the time was right for an off-site solution.</p> <p>The one sticky issue with S3 is that you can’t just use it; Amazon provide an API but you have to rely on third-party applications to make use of it. Here is where my fun began.</p> <p>The first tool I looked at was the ruby-based s3sync. Following some instructions google found for me on <a href="">John Eberly’s</a> blog, I set about creating a storage bucket and began uploading the 1.5Gb of photos from my main server. Unfortunately, this server runs an old version of debian which, as it’s also my TV, is not about to be updated. This left me with the wrong version of ruby, so I moved to the python based tools provided by <a href="">Hanzo</a>. Besides, I’m more a python guy these days.</p> <p>After a couple hours of uploading, I decided to play some more with the API; I wanted to see how to change the permissions of the files so that I could share them publicly. After reading more of the documentation, it appeared that the only way to do this is via the REST or SOAP APIs. I soon found another python library to try - this time one available on the Amazon site itself - that would do the job. However, every attempt I made failed as it couldn’t find the bucket I had created. Getting quite frustrated with it I eventually found the problem; to access a bucket via HTTP, the it’s name needs to be all lowercase - which was not the case with the bucket I had created. Given the inability to rename an existing bucket, I stopped uploading, deleted the bucket and started again.</p> <p>That mistake cost me about $0.10. Not exactly the end of the world.</p> <p>24 hours, 1 internet outage and $0.34 later, 1800 photos and a handful of videos from April 2003 to August 2005 are now safely stored and accessible from anywhere with an internet connection.</p> <p>Just another 4000 to go - once I get them copied off the various laptops.</p> Silver-Wrapped Crack 2007-04-29T00:00:00+00:00 <p>I blame <a href="">Roo</a>. Having had a look at some of his <a href="">Perplex City</a> cards last week, today I went and got some of my own from Season 2.</p> <p>Here are today’s stats:</p> <ul> <li>3 packs bought - initially I bought 2, but when Jo went back to the till to buy another book, I added another pack.</li> <li>18 puzzle cards and 3 ‘warm-up’ sudoku and hitori cards.</li> <li>15 cards solved - particularly pleased with #161.</li> <li>4320 points gained.</li> <li>Ranked 1124 at the time of writing.</li> <li>Close to solving the 16th card (#164) - I know the reference, just not sure how to get an answer to fit.</li> <li>Unsure where to begin with the 17th card (#235).</li> <li>18th card available for swaps (#004).</li> </ul> <p>This is an addiction. I am an <a href="">addict</a>. I need more cards.</p> Sponsored by Crayola 2007-04-18T00:00:00+00:00 <p>In the mildly alarming, if not surprising, news that <a href="">inflation has jumped to %3.1 last month</a>, I find a small glimmer of humour. A requirement of “the system” is that the Bank of England’s governor sends the government an open letter when this type of thing happens<sup><a href="#jump1">[1]</a><a id="jump1_back"></a></sup>. In this instance, because he was brought up well by his parents, Gordon Brown<sup><a href="#jump2">[2]</a><a id="jump2_back"></a></sup> wrote a nice letter back<sup><a href="#jump3">[3]</a><a id="jump3_back"></a></sup>.</p> <p>I can imagine the scene. Gordon has dictated the letter to a civil servant who has duly typed it up for him. It has then been presented back to Gordon to sign, who, being the busy man he is, signs it with whatever he has in his hand.</p> <p>Given the look of his signature, I really do wonder what he was doing at the time… perhaps trying very hard to keep the colouring in-between the lines?</p> <p><img src="/blog/content/2007/04/gb_crayon.gif" alt="Gordon Brown" /></p> <p><sup><a id="jump1">[1]</a></sup>Specifically, it was the fact that the inflation rate was 1% above or below the target rate of 2%.</p> <p><sup><a id="jump2">[2]</a></sup>Gordon got to respond in his role as Chancellor, not PM-in-waiting.</p> <p><sup><a id="jump3">[3]</a></sup>The letters are link to from the <a href="">BBC News article</a>.</p> Hillier in the Spring 2007-04-15T00:00:00+00:00 <p><a href=""><img src="" alt="Spring Blossum - 1" /></a><a href=""><img src="" alt="Spring Blossum - 2" /></a><br /> <a href=""><img src="" alt="Spring Blossum - 3" /></a><a href=""><img src="" alt="Spring Blossum - 4" /></a></p> <p>Visiting <a href="">Rachel’s</a> <a href="">exhibition</a> at <a href="">Sir Harold Hillier Gardens</a></p> Surreal Recipe 2007-04-09T00:00:00+00:00 <p><a href="" title="photo sharing"><img src="" alt="" /></a> <a href="">Surreal Recipe</a></p> <p>Seen in a bookshop in Norwich; I always like a cup of Surrealism with my Soup.</p> My (Google) Maps 2007-04-09T00:00:00+00:00 <p><a href=";hl=en&amp;om=1&amp;z=16&amp;msid=110093968748740361746.00000111d7c2f1bf7531f&amp;msa=0"><img src="/blog/content/2007/04/gmaps.gif" alt="My First Google Map" /> My First Google Map</a></p> <p>Google have added a new feature to their maps service; the ability to create your own annotated maps that can not only be shared with others but also that will show up in the search index.</p> <p>The maps can be annotated with markers, lines drawn to mark routes and areas highlighted to… well, whatever you want.</p> <p>This is a really useful service to have. In the past, I’ve used <a href="">Pin In The Map</a> to share places with others, but that is quite limited in comparison. I’ve also tried using Microsoft’s Live maps service which lets you share maps, but after my first map came up blank with all I shared it with, I soon gave up.</p> <p>Google also lets you download your maps in <a href="">kml format</a> so they can be used within Google Earth.</p> <p>For my first foray into My (Google) Maps, <a href=";hl=en&amp;om=1&amp;z=16&amp;msid=110093968748740361746.00000111d7c2f1bf7531f&amp;msa=0">here </a> is a short walk Jo and I often go on to escape the sofa and get some air in our lungs.</p> Easter in Cromer 2007-04-09T00:00:00+00:00 <p><a href=""><img src="" alt="Big Sky" /></a><a href=""><img src="" alt="Cromer Pier" /></a><a href=""><img src="" alt="IX" /></a></p> <p><a href=""><img src="" alt="Liftboat Station" /></a><a href=""><img src="" alt="Three Lights" /></a><a href=""><img src="" alt="Cromer Waterfront" /></a></p> <p>Cromer at Dusk</p> <p><a href=""><img src="" alt="Windmill at Cley" /></a><a href=""><img src="" alt="Tilting At Windmills" /></a></p> <p>Windmill at Cley-next-the-Sea</p> <p><a href=""><img src="" alt="IMG_5533.JPG" /></a><a href=""><img src="" alt="IMG_5524.JPG" /></a><a href=""><img src="" alt="Mandarin Duck" /></a><a href=""><img src="" alt="Lapwing" /></a><a href=""><img src="" alt="Lapwing" /></a><br /> <a href=""><img src="" alt="Friendly Goose" /></a><a href=""><img src="" alt="Flamingo &quot;AAA&quot;" /></a><a href=""><img src="" alt="Tufted Duck" /></a><a href=""><img src="" alt="Avocet" /></a><a href=""><img src="" alt="Thistle Detail" /></a></p> <p><a href="">Pensthorpe Nature Reserve</a></p> Jobs on Music: redux 2007-04-02T00:00:00+00:00 <p><small><a href="/2007/02/06/jobs-on-music">part 1</a></small></p> <p>It appears that Apple were prepared to <a href="">rock the boat</a> after all.</p> <p>With the launch next month of the ‘premium’ versions of EMI tracks, I can begin to see a world where I am prepared to buy music online. I’m happy to pay the slight extra cost for the DRM-less tracks - especially given their higher quality.</p> <p>However there remains for me one barrier to this utopia; iTunes remains a proprietary application that is only available on Mac and Windows. Given they don’t have DRM as an excuse anymore, will an open alternative become viable?</p> <p>I say “viable” as I know there are already unburdened music stores out there. But without the “mainstream” artists available through their sites, the appeal is unfortunately limited.</p> Signs things are improving 2007-03-27T00:00:00+00:00 <p>As I mentioned on <a href="">Andy’s blog</a> yesterday, I have used Ubuntu Linux on my work laptop for about two years now. From the early days of running Lotus Notes under <a href="">Wine</a> emulation to the successes of Notes 7 running natively within IBM Workplace Managed Client, I have seen many ‘challenges’ come and ago that are part and parcel of using any technology that is not considered mainstream.</p> <p>One of the issues that comes up from time-to-time is that of document format. For a long time, the standard office tools were <a href="">Lotus Smartsuite</a> - not surprising since IBM owns it - and then, a few years ago, IBM moved to the Microsoft Office suite. Given a company of a few hundred-thousand, it wasn’t surprising that not everything and everyone moved to Office overnight. Even now there are plenty of legacy documents in the various Lotus formats.</p> <p>I survived with <a href="">OpenOffice</a> through perseverance and accommodating colleagues who would convert the odd WordPro or Freelance file for me - although I haven’t had to resort to that for months now.</p> <p>OpenOffice isn’t 100% perfect - there are occasional formatting oddities which can make some documents interesting to read. For example, one document I am currently working on has a few bulleted lists. One of the lists persists in formatting with <strong>huge</strong> bullet points despite all of my efforts to convince it otherwise. (Before anyone asks, no I haven’t raised a bug for this; I’ll see if I can get a non-confidential example document that exhibits the same behaviour first.)</p> <p>This eventually brings me to today’s entry in “Signs things are improving”; my manager sent out a planning spreadsheet today that was in OpenOffice Calc format. It took a few moments for the shock to pass before I remembered that she is running a beta of <a href="">Notes 8</a> that includes its ‘Productivity Tools’ - aka OpenOffice.</p> <p>Open Standards for teh win!</p> Exbury Gardens 2007-03-26T00:00:00+00:00 <p><a href=""><img src="" alt="White Drops" /></a><a href=""><img src="" alt="Pink Geometry" /></a><a href=""><img src="" alt="Bark" /></a><br /> <a href=""><img src="" alt="Falling Water" /></a><br /> <a href=""><img src="" alt="Whites" /></a><a href=""><img src="" alt="White" /></a><a href=""><img src="" alt="Weathered Face" /></a></p> <p>Spring at <a href="">Exbury Gardens</a></p> Manor Farm Country Park 2007-03-18T00:00:00+00:00 <p><a href=""><img src="" alt="Stuck in the Mud" /></a><a href=""><img src="" alt="Mallard" /></a><a href=""><img src="" alt="Swan" /></a><br /> <a href=""><img src="" alt="Looking Southward" /></a><a href=""><img src="" alt="Looking Northward" /></a></p> <p>After the rain at Manor Farm Country Park</p> Highcliffe 2007-03-11T00:00:00+00:00 <p><a href=""><img src="" alt="Rocky" /></a><a href=""><img src="" alt="Spray" /></a><a href=""><img src="" alt="Mussel Rock" /></a><br /> <a href=""><img src="" alt="Needles in the Solent" /></a><a href=""><img src="" alt="Spring Yellow" /></a><a href=""><img src="" alt="Highcliffe Castle" /></a></p> 1.33 Miles of Pier 2007-03-09T00:00:00+00:00 <p><a href="" title="photo sharing"><img src="" alt="" /></a></p> <p><a href="">1.33 Miles of Pier</a></p> <p>Why do pier’s tend to burn down? True they are often made of wood, but they are also usually surrounded by sea.</p> <p>Southend’s pier maybe the longest <em>pleasure</em> pier in the world, at a leg aching 1.33 miles, but I wonder what’s a pier thats not for pleasure? A <em>business</em> pier?</p> <p>When arriving at the sea-side, do you get stopped and asked if you are there for business or pleasure and then get directed to the appropriate pier?</p> <p>This may seem a random post, but at least it proves the hook-up from flickr to this blog works.</p> Snow 2007-02-08T00:00:00+00:00 <p><a href=""><img src="" alt="The First Snowman" /></a><a href=""><img src="" alt="Snow Leaf" /></a><a href=""><img src="" alt="Half and Half" /></a><a href=""><img src="" alt="Snow Detail" /></a></p> Goodbye MSN 2007-02-08T00:00:00+00:00 <p>I’ve had an MSN id since University. It has had two uses:</p> <ul> <li>to act as a spam hole for sites that need an e-mail address,</li> <li>and to let me chat on the MSN network.</li> </ul> <p>I’ve <a href="/2006/11/23/windows-live-mail/">mentioned</a> before that I don’t feel particularly compelled to keep the ID active. I’m not sure what has led me to make this decision, but today I have decided to let the ID expire and say goodbye to the MSN network.</p> <p>Most of my contacts are on Google Talk and this evening I have prompted one to start using it more - I even got him to install the Windows version of <a href="">Gaim</a>.</p> <p>So, if you’ve got my MSN id and not my Google Talk one (unlikely)… let me know. If you don’t have either and want it… let me know.</p> <p>In other news, my Web2.0 rating has just increased a notch with the addition of my <a href=""></a> account.</p> Jobs on Music 2007-02-06T00:00:00+00:00 <p>Steve Jobs has <a href="">posted</a> some thoughts on DRM. He highlights many of the issues with DRM and considers some of the possible future directions Apple could take.</p> <p>He makes the very good point that over 90% of the music sold by the companies last year was DRM-free; by virtue of being sold on unencumbered CDs. Sure the labels have tried out DRM-laden CDs, but it’s not always worked out well - yes <a href="">Sony</a>, I’m looking at you.</p> <p>Whilst its good to read what Apple’s position is on a DRM-free future, I felt it lacked any mention of what Apple is doing about it.</p> <blockquote> <p>Perhaps those unhappy with the current situation should redirect their energies towards persuading the music companies to sell their music DRM-free.</p> </blockquote> <p>Its not clear whether Apple is one of those unhappy people with energies needing direction.</p> <p>It seems to me that they do what the “Big Four” record companies require in order to distribute music and they won’t do anything to rock that lucrative boat.</p> <p><em>Update</em>:</p> <p><a href="">Engadge</a> and <a href="">Wired</a> provide some thoughts on the subject. I had missed the link to the EU’s calls for Apple to open up iTunes.</p> <p>Also, note the irony that this has been published the day after <a href="">Apple and Apple have settled their legal dispute</a>.</p> <p><em>Update again</em>, nice post by <a href="">Jono</a> on the subject.</p> نيكهولاس Ùˆ'لياري 2007-02-06T00:00:00+00:00 <p>Has <a href="">Roo</a> started a <a href="">meme</a>? Only time and the internet Gods will tell.</p> <p>My name transliterated to Arabic is نيكهÙ^ٔاس Ù^’ٔيار.</p> <p>Thanks go to <a href="">Rob’s transliteration tool</a>.</p> <p><em>update:</em> weirdness abounds. The trackback to Rob’s blog has worked fine, however the trackback on Roo’s post has broken the URL. All three blogs are running Wordpress, and Roo’s is hosted on the same machine as this one. Odd.</p> Winchester 2007-02-04T00:00:00+00:00 <p><a href=""><img src="" alt="St Giles Hill" /></a><a href=""><img src="" alt="Center Tile" /></a><a href=""><img src="" alt="Blue and Yellow" /></a><br /> <a href=""><img src="" alt="Unaligned Reflections" /></a><a href=""><img src="" alt="Window Tree" /></a><a href=""><img src="" alt="Cathedral Mirror" /></a><a href=""><img src="" alt="Structure" /></a></p> <p>A sunny day in Winchester</p> Sam Ruby on Standards 2007-01-30T00:00:00+00:00 <p><a href="">Sam Ruby</a>, a colleague I have not met within or without work, has this to say on Standards:</p> <blockquote> <p>A <strong>standard</strong> is one that has multiple, inter-operable, independent implementations. An <strong>open</strong> standard, at least in the software world, is one where at least one of those implementations is open source.</p> </blockquote> <p>The full piece can be read <a href="">here</a>.</p> The BBC's Fifteen Web Principles 2007-01-29T00:00:00+00:00 <p>Roo has got an interesting <a href="">post</a> on his recent visit to the BBC; it certain sounds like they are heading in good directions.</p> <p>In the comments, <a href="">Andy Piper</a> linked to <a href="">The BBC’s Fifteen Web Principles</a>. They make a lot of sense and should be required reading for anyone interested in the subject.</p> <p>I particularly like the line:</p> <p><strong>13. Let people paste your content on the walls of their virtual homes</strong></p> <p>It encapsulates everything I enjoy about the internet these days.</p> Get a Mac - UK style 2007-01-29T00:00:00+00:00 <p>Whilst I may not worship at the alter of Apple beyond owning an elderly 3G iPod, I do like their series of <a href="">Mac/PC TV ads</a>. Their blatant stereotyping of the suit-wearing PC compared to the hip and cool, hoodie-wearing Mac may be a little unfair, but they do get their message across. What would it take to get Bill Gates and Steve Jobs together in front of a camera for a real-life debate of their respective platforms? Unlikely I reckon.</p> <p>Apple do invest a lot of time and effort on image and style. They certainly benefit from controlling the hardware that their software runs on - both in terms of controlling the look of the brand as well as being able to profit directly from it. I wonder how many people buy a Mac because of the look of the laptop they get to carry, or the operating system it runs?</p> <p><img src="/blog/content/2007/01/indextop20072601.jpg" alt="Get a Mac - Mitchell and Webb " /></p> <p>Either way, Apple just went up another notch in my cool scale; the Get a Mac TV adverts have been redone for the UK market, albeit it web-only, with the very funny <a href="">Mitchell and Webb</a> taking the Mac and PC roles. It’s a perfect fit for their characters - whichever advertising creative realised this, should be rewarded. There are only 6 ads up at the moment, but I’m sure more will follow.</p> <p>Its not enough to get me to switch though - I’m in a position where I pay nothing for my Operating System and don’t have to pay a premium for the shiny white box to run it on.</p> <p>Go check out the UK ads <a href="">here</a>.</p> <p><a href="">Numberwang</a>.</p> moving $HOME 2007-01-23T00:00:00+00:00 <p><a href=";ID=MRWESHA15J353"><img src="/blog/content/2007/01/forsale.jpg" alt="For Sale" /></a></p> <p>After two and a half years in my flat, Jo and I have decided its time to move on. Where exactly has yet to be determined, although we do plan to stay in or around <a href=",+Southampton,+Hampshire,+United+Kingdom&amp;ie=UTF8&amp;hl=en&amp;sll=53.098145,-2.443696&amp;sspn=12.421367,29.882813&amp;z=12&amp;ll=50.932902,-1.332779&amp;spn=0.101695,0.346069&amp;om=1&amp;iwloc=addr">West End</a>.</p> <p>Since Jo moved in after the wedding, we have struggled for space and found ourselves frequently taking trips to the local charity shop with bags of clothes and other assorted items.</p> <p>When I bought the flat, the plan had been for the 2nd bedroom to become an office. Unfortunately I have barely spent any time in there; it became the dumping ground for anything that didn’t fit elsewhere. It was quite a task last Sunday to de-clutter the surfaces so it would be ready for viewings. I just hope the estate agent doesn’t open any of the wardrobes…</p> <p>It seems a bit ironic that it’s the prospect of selling the flat that has lead us to tidy it this much.</p> <p>More details on the flat are available <a href=";ID=MRWESHA15J353">here</a>.</p> January 21st, 2007 2007-01-22T00:00:00+00:00 <p><a href=""><img src="" alt="Goose" /></a><a href=""><img src="" alt="Sunset Lake" /></a><a href=""><img src="" alt="Power Lines at Dusk" /></a><a href=""><img src="" alt="Postal Swan" /></a><br /> <a href=""><img src="" alt="Mullet Duck" /></a><a href=""><img src="" alt="Mirrored Reeds" /></a><a href=""><img src="" alt="Bud" /></a><a href=""><img src="" alt="Playground Foreground" /></a></p> <p>Visiting friends in Andover</p> Power Cut in West End 2007-01-16T00:00:00+00:00 <p><a href=""><img src="" alt="Candle-lit Meeting" /></a></p> Telegraph Woods 2007-01-14T00:00:00+00:00 <p><a href=""><img src="" alt="Arboreal Barcode" /></a><a href=""><img src="" alt="Hollow" /></a><br /> <a href=""><img src="" alt="Crispy Leaves" /></a><br /> <a href=""><img src="" alt="New Bud" /></a><a href=""><img src="" alt="Tenticle" /></a></p> <p>Telegraph Woods</p> Planetary Updates 2007-01-04T00:00:00+00:00 <p>I’ve been running my own local <a href="">planet</a> for a while now. Whilst I doubt there are many other people out there who read it, I have finally got around to updating its style. Now it feels more a part of the site as a whole.</p> January 1st, 2007 2007-01-01T00:00:00+00:00 <p><a href=""><img src="" alt="Pipeline" /></a><a href=""><img src="" alt="Blur-past" /></a><a href=""><img src="" alt="Ripples" /></a><br /> <a href=""><img src="" alt="Blue Sky" /></a><a href=""><img src="" alt="House by the reeds" /></a><a href=""><img src="" alt="Reeds in detail" /></a></p> <p>New Year’s Day walk at the beach by Netley.</p> Wedding Photos 2006-12-12T00:00:00+00:00 <p>Its over a month since the wedding and I have yet to put up the ‘official’ photos of the event to the <a href="">flickr pool</a>, although there are over 200 photos available to see. Actually, some have their privacy settings turned on, so there may be less than that visible to the general public. Regardless, there are plenty of great photos that really help to relive the event.</p> <p>The ‘official’ photographs were taken by my Dad who did a great job of sticking around to the very end. Jo and I have been poring over a couple CDs worth of images he has since sent us to try to put together the official Wedding Album. In particular, we wanted to find <em>the</em> image of us that we could send with christmas cards to all the aunties and uncles who didn’t come to the wedding.</p> <p>For quite a few of the photos, I have had to spend some time in the <a href="">GIMP</a> (GNU Image Manipulation Program; the linux equivalent of Adobe’s Photoshop) touching up the images to make them just right. This has including doing minor things like removing red-eye to quite major jobs like swapping around heads from different photos to get everyone looking the same way with ‘nice’ smiles and no grimaces. I am particular happy with the results of our main portrait.</p> <p>Below is a side-by-side comparison of the before and after images. Play spot the differences - there are 8 to be found. (With apologies to Meg and David for having removed them so callously - oops, thats give away two of the differences.)</p> <p><img src="/blog/content/2006/12/original_small.jpg" alt="Original Wedding Portrait" /> <img src="/blog/content/2006/12/after_small.JPG" alt="Wedding Portrait after editing" /></p> Selphy CP510 under Linux 2006-12-10T00:00:00+00:00 <p><em>Having googled around this subject, I was surprised to see how little information was out there that could help me. This post is intended to increase the Google Juice.</em></p> <p>I bought a Canon Selphy CP510 printer earlier in the year to replace my Canon Stylus Photo 950. The ‘950 was proving too expensive to run given how little photo printing I was doing and the large amount of text printing Jo was doing. (The Selphy only replaces the photo printing side of the 950; I also bought a HP Scanjet F320 which acts as a better general printer).</p> <p>The Selphy is a great little portable photo printer that does almost all I want. Being able to directly attach either of our cameras is really handy. However I came across an annoying issue with it; it refuses to print an image that has been edited on the computer and put back on the camera. It simply says ‘cannot print’ with no further explanation. Google doesn’t help much here, aside from a few reports that agree with my experience. Unfortunately I couldn’t get to the bottom of the issue so I had to get the Selphy directly attached to my laptop.</p> <p>The <a href=""></a> page for this printer details the state of support; currently described as ‘mostly’. The laptop is running Ubuntu Dapper which makes a lot of this easier.</p> <p>The last time I tried printing to the Selphy, I added a printer to the system, selecting the Canon CP100 driver; there wasn’t a CP510 driver listed as was suggested there should be. When I tried printing from EOG it failed, wasting a shot of the Selphy cartridge. Not wanting to waste more, I left it at that.</p> <p>This time, I read around a bit more, and got the impression that gimp-print/gutenprint was the way to go. That meant installing some extra packages that I didn’t have - an <code class="language-plaintext highlighter-rouge">apt-cache search</code> on <code class="language-plaintext highlighter-rouge">gutenprint</code> reveals a number of them.</p> <p>I then fired up the <a href="">GIMP</a> and loaded one of my recent photos. Under the print dialog I added a new printer called ‘Selphy’ and under the ‘Setup Printer’ dialog I selected the Print Queue of the original CP100 printer I added previously.</p> <p>For the driver, I expected there to be a CP510 driver in the list having installed the new packages, but there simply wasn’t. I selected the Canon CP220 driver this time, and was happy to discover that I was able to successfully print directly to the Selphy.</p> <p>One last note; before printing, make sure you click ‘Save Settings’ - otherwise it will print and then forget all the settings you’ve made.</p> Spitfire Flypast at Hursley, UK - Google Video 2006-12-08T00:00:00+00:00 <p>As a mark of respect for a member of the original Spitfire design team who passed away recently, the Spitfire Society arranged a flypast of Hursley House on Friday 8th December.</p> <iframe width="560" height="315" src="" frameborder="0" allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe> <p>I forgot my camera, so my phone stood in. The video quality isn’t great, but it does give some sense of the event. There were plenty of photographers out on the lawn, so I’m sure <a href="">Flickr</a> will soon be flooded with photos from the event.</p> <p>Hursley was where the Spitfire engines were developed - long before IBM got there.</p> Opera 2006-12-01T00:00:00+00:00 <p>Tonight my parents treated Jo and I to our first opera; the <a href="">Welsh Nation Opera</a>’s production of <a href="">La Bohème</a> at Southampton’s <a href="">Mayflower</a> theatre.</p> <p>It was a very impressive production, although the orchestra did drown the leads on occasion.</p> <p>The lesson I learned tonight, taken from the programme:</p> <blockquote> <p>There were girls who did and girls who didn’t and the rule of thumb was that those who succumbed rarely made it to the final curtain.</p> </blockquote> <p>Apparently, Mimi is a girl who did.</p> Prior Art 2006-11-25T00:00:00+00:00 <p>Apple clearly weren’t the first to think white headphones would look good.</p> <p><img src="/blog/content/2006/11/rembrandt.jpg" alt="Portrait of an Elderly Man - Rembrandt" /><br /> Rembrandt’s Portrait of an Elderly Man</p> Windows Live Mail 2006-11-23T00:00:00+00:00 <p>Neil’s <a href="">Hotmail Haiku</a> reminded me to log into Hotmail to keep my id alive. I’m close to abandoning Hotmail altogether - I don’t use the email address for anything other than as a spam hole and enough of my contacts are on gtalk these days.</p> <p>Once logged in, I was given the opportunity to use Windows Live Mail. Out of interest, I clicked through the registration process to see what its like. Having got to the end, it told me I would be able to access it within the next two weeks. Oh well, I thought. I then went back to hotmail, only to find I could access it already.</p> <p>Well, not quite - here’s the message I got when I logged in:</p> <p><img src="/blog/content/2006/11/hotmail.gif" alt="hotmail" /></p> <p>Unsurprisingly, you need IE6.0 at least to make full use of Windows Live Mail - with a nice link to the IE homepage. Oh, apparently the full version also works with Firefox 1.5 - no link to the firefox page though.</p> <p>I say apparently, because, you’ve guessed it, I <em>am</em> using Firefox 1.5.</p> <p>I’m that bit closer to abandoning Hotmail altogether.</p> Matrix Transform 2006-11-22T00:00:00+00:00 <p><a href=""><img src="" alt="" /></a></p> <p>Having recently relearned about rotation matrices for my <a href="/2006/10/04/ruby-cubes/">Ruby Cubes</a> post, this made me laugh.</p> <p>(via the excellent <a href="">xkcd</a>)</p> Y! Maps Update 2006-11-17T00:00:00+00:00 <p><a href="/blog/content/2006/11/yahoo_maps.gif" title="Yahoo! Maps"><img src="/blog/content/2006/11/yahoo_maps.thumbnail.gif" alt="Yahoo! Maps" /></a></p> <p>Since Flickr added the ability to geo-tag photos by dropping them onto a map provided by <a href="">Yahoo Maps</a> I have been keenly updating my photos as best I can. The main limitation has been the level of detail available outside the US on their maps. Note the past tense. Whilst working on my <a href="">honeymoon photos</a> I spotted that whilst I was away they have been busy updating their maps. I don’t know how wide-spread the update is, but it certainly looks good for my <a href=";q1=southampton%2Cuk&amp;trf=0&amp;lon=-1.410198&amp;lat=50.910069&amp;mag=6">local</a> area, as well as where we were went on holiday - <a href=";trf=0&amp;lon=-13.706818&amp;lat=28.918889&amp;mag=4">Lanzarote</a>.</p> <p>Although, Google still wins for level of detail in the satellite image for that particular area - <a href=";z=19&amp;ll=28.91702,-13.710441&amp;spn=0.001103,0.001829&amp;t=h&amp;om=1">here’s</a> our hotel.</p> Head Table 2006-11-03T00:00:00+00:00 <p><a href=""><img src="" alt="Head Table" /></a></p> <p>By this time tomorrow, I’m a married man.</p> <p>Today has been a long day of chores; suits picked up, cravats replaced with the correct ones, tables laid and nerves calmed.</p> <p>Keep an eye on flickr, we’ve setup a <a href="">group</a> for the day.</p> 5 Years, 2 Months 2006-11-02T00:00:00+00:00 <p>Today, after 5 (and a bit) years, I said goodbye to IBM System &amp; Technologies Group. I wasn’t at all sure what today would be like - I have experienced plenty of other people leave; the obligitory leaving presentation by the manager, the traditional drinks at the Dolphin and the card signed by everyone our secretary could badger. But it all went well in the end.</p> <p>As this is the first full-time job I’ve had since leaving university, the thought of coming into work after my honeymoon to a completely new job in a new part of Hursley is simply weird.</p> <p>I have made many great friends in the Storage group and have really enjoyed working on the SAN Volume Controller. I’m sure I will be back to visit there - a lunchtime without a crossword just wouldn’t be the same.</p> <p>So, as of November 27th, I am an employee of IBM Software Group in the Pervasive and Advanced Messaging team. I’ll work out what that actually means when I get there.</p> <p>As careful readers will have spotted, I’m off on my honeymoon on Monday, which gives away the fact that I am getting married on Saturday. More on this in the near future (if last minute organisation/panics allow).</p> <p>Just call me Nick “one-major-life-event-at-a-time-isn’t-enough” O’Leary.</p> Mr O'Lantern 2006-10-31T00:00:00+00:00 <p><a href=""><img src="" alt="Mr O" /></a></p> <p>First name Jack.</p> for(laziness in javascript) 2006-10-18T00:00:00+00:00 <p>This post is mostly a note-to-self.</p> <p>Compare the following pieces of javascript:</p> <pre style="background: #eee; margin: 10px; padding: 2px; float: left;"> var list = ['a','b','c']; for(x in list) { alert(x); }</pre> <pre style="background: #eee; margin: 10px; padding: 2px; float: left;"> var list = ['a','b','c']; for(**var** x in list) { alert(x); }</pre> <div style="clear:both;">&nbsp;</div> <p>Either works in Firefox. Only the one on the right works in IE. Hint: the difference is in <strong>bold</strong>.</p> <p>The IE behaviour encourages you to define your variables and not get lazy. Given that I develop under Firefox and don’t test under IE until late in the day, remembering this one will be a huge time-saver.</p> Overflowing Quirks 2006-10-16T00:00:00+00:00 <p>Much has been written about the inconsistencies of CSS within the various browsers. A great resource for learning about them is the <a href="">QuirksMode</a> website. Despite having read a lot of that site, I spent far too much time today trying to debug one.</p> <p>Its worth mentioning upfront that this is with Firefox on my laptop. Since writing up this post, I have found that Firefox 1.0.x on my desktop does not behave as described here. I’m not sure which is infact ‘correct’.</p> <p>For a website I’m developing in work, I have a drag-and-drop interface that allows the user to arrange objects represented as <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code> elements onscreen. If the user cancels the drag, the <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code> is animated returning to its original position. For reasons that are not important, the <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code> can return to one of two containing <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code> elements. This all works well, apart from the fact that for one of the containers, an object that is returning to it consistently ends up 1 pixel out of position.</p> <p>I initially assumed this was to do with the <a href="">Box model</a> issues, but the css for the two containers have identical <code class="language-plaintext highlighter-rouge">border</code> declarations. A bit of further digging and I found the only difference was the <code class="language-plaintext highlighter-rouge">overflow</code> style - and funnily enough, thats exactly where the problem lies.</p> <p>To demonstrate, below are two identical <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code> elements that contain a child <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code> which has <code class="language-plaintext highlighter-rouge">position: relative</code> and is offset by <code class="language-plaintext highlighter-rouge">5px</code> on both <code class="language-plaintext highlighter-rouge">top:</code> and <code class="language-plaintext highlighter-rouge">left:</code>. The sole difference between them is the first has <code class="language-plaintext highlighter-rouge">overflow: hidden</code>, and the second <code class="language-plaintext highlighter-rouge">overflow: visible</code>. Each of the inner <code class="language-plaintext highlighter-rouge">&lt;div&gt;</code>s are armed with the following piece of javascript:</p> <pre>onclick=" var a=this.parentNode; this.innerHTML = 'dx:'+(this.offsetLeft-a.offsetLeft)+ ' dy:'+(this.offsetTop-a.offsetTop); "</pre> <p>When you click on them, the offset between the child and its parent is shown. Remember that the css is identical for these examples in all but <code class="language-plaintext highlighter-rouge">overflow</code>.</p> <div id="outer_one" class="outer" style="overflow: hidden;"><div onclick="var a=this.parentNode; this.innerHTML = 'dx:'+(this.offsetLeft-a.offsetLeft)+' dy:'+(this.offsetTop-a.offsetTop);" id="inner_one" class="inner">click me</div></div> <div id="outer_two" class="outer"><div onclick="var a=this.parentNode; this.innerHTML = 'dx:'+(this.offsetLeft-a.offsetLeft)+' dy:'+(this.offsetTop-a.offsetTop);" id="inner_two" class="inner">click me</div></div> <p>As you can see, assuming you have clicked away and your are running the right level of the right browser, the offsets are different and its all thanks to the <code class="language-plaintext highlighter-rouge">overflow</code>. I have not fully investigated the various fixes for CSS issues such as <a href="">Strict mode</a>, but I was fortunately able to code a workaround using javascript that dynamically alters the <code class="language-plaintext highlighter-rouge">overflow</code> style depending on whether the user is dragging. I admit its not the cleanest solution.</p> <style>div.outer { margin-bottom: 30px; display: block; width: 150px; height: 50px; border: 1px solid #000; background: #eee; } div.inner { position: relative; top: 5px; left: 5px; background: #fee; width: 150px; height: 50px; border: 1px solid #000; cursor: pointer;}</style> Canon PowerShot S3 IS 2006-10-13T00:00:00+00:00 <p>I’ve been wanting a new camera for a while now. My old one, a <a href="">PowerShot S45</a>, has served me well over the past 4 years, but it had started to show its age.</p> <p>Having played with Roo’s <a href="">PowerShot S2 IS</a>, I’ve had its successor, the <a href="">S3</a> on my <a href="">Amazon wishlist</a> for a few months now.</p> <p>With my Honeymoon fast approaching, I decided last week to go ahead and buy it - it didn’t hurt that Jo said it would be my wedding/birthday/christmas present from her!</p> <p>It arrived today along with some rechargable batteries and a slightly paltry 16Mb SD card - there are 2x 512Mb SD cards in the post.</p> <p>I’m not going to do a full review of the camera here - read the one linked to above if you are interested. The only feature I feel the need to really show-off is the Super Macro mode. This enables the camera to focus at 0mm - ie objects touching the lense.</p> <p>By way of comparison, check out these two photos of a spider who has been living on my window for the past couple months. The left-hand image was taken with Jo’s new <a href="">IXUS 60</a>, the right-hand image with my new toy. The crispness of S3’s image is astounding.</p> <p><a href=""><img src="" alt="Spider - IXUS60" /></a><a href=""><img src="" alt="Spider - S3IS" /></a></p> <p>For some reason, macro photos of spiders seems to be the <a href=""> photo of choice</a> recently.</p> Surfing Weekend 2006-10-11T00:00:00+00:00 <p><a href=""><img src="" alt="The Surfing Stags" /></a> <a href=""><img src="" alt="Standing proof" /></a></p> <p><a href="">Roo</a> lived up to his role as Best Man last weekend by organising a Surfing trip to <a href=";ie=UTF8&amp;z=14&amp;ll=51.17337,-4.206305&amp;spn=0.025292,0.069094&amp;om=1">Woolacombe</a> for my Stag weekend.</p> <p>I have surfed a few times in the last couple years but it was good to have a day of properly organised tuition - thanks to Matt from <a href="">Nick Thorn</a>’s surf school for that. Thanks also to the guys at <a href="">Extreme Board Shots</a> for photographing the day in the water.</p> <p>Whilst not everyone I had hoped could make the weekend, fun was definitely had by all.</p> <div style="clear:both;"></div> Ruby Cubes 2006-10-04T00:00:00+00:00 <p><a href="/blog/content/2006/10/pieces.jpg"><img src="/blog/content/2006/10/pieces.thumbnail.jpg" alt="Ruby Cube Pieces" /><br /> The 6 Pieces</a></p> <p>At a recent get-together with friends, I found myself playing with a wooden puzzle consisting of 6 pieces that were claimed to be able to come together to form a solid cube.</p> <p>Lucy, the puzzle’s owner, said she had owned it for years and never successfully solved it. Not to be detered, I spent the whole evening failing to get anywhere with it.</p> <p>Continuing not be detered, I decided to do the honorable geek thing and write a computer program to solve it.</p> <p>Herewith how I approached the problem and got started with ruby.<!--more--></p> <h2 id="investigating-the-problem">Investigating the problem</h2> <p>I decided early on to go for an educated brute-force approach; educated in the sense that there were bound to be some assumptions that I could make to simplify things. To get a sense of the full scale of the problem, I wanted to calculate the number of combinations that the pieces could be arranged in.</p> <p>Consider each piece to consist of a number of 1x1x1 cubes and that there is a key cube within each piece. The key cube of each piece can be located in one of the 3x3x3=27 different locations in the solution cube (aka box). Each piece can then be rotated to 24 different orientations. This leads to having 24*27=648 arrangements of just one piece. We could crudely say this leads to 648^6 arrangements of all six pieces - 74,037,208,411,275,264.</p> <p>However, once the first piece is placed, we know that the key cube of the second piece cannot be located in the space the key cube of the first piece is. This means there are 27*26*25*24*23*22=213127200 arrangements of the pieces with regard position only. To factor in rotation, as there is no dependency between piece’s rotation, there are 24^6=191102976 arrangements of the pieces. Combining these gives: 213127200*191102976=40,729,242,186,547,200 combinations. Whilst not an order of magnitude less than the previous value, the figure is headed in a good direction.</p> <p>However, even if these combinations were to be consided at 1000 per second, it would take over 1291515 years to complete. It’s clear there needs to be some further refinement to the problem.</p> <p>The trick to simplifying the problem is realising that we do not need to consider every combination of every piece. To see this, the constraints that describe a ‘good’ placement of a piece need to be defined:</p> <ol> <li>The piece must fit entirely within the 3x3x3 box</li> <li>Once placed, the piece must not leave any voids in the solution cube that are 3 or less cubes in size (as all of the pieces consist of at least 4 cubes)</li> </ol> <p>Given a partial solution that has 2 pieces currently placed, if we find it impossible to place the third piece into the box without violating any of the constraints then it is not worth even trying to place any of the other 3 pieces. By considering each piece in turn like this, it is possible to make life even easier by considering the more complex pieces first - ie the ones with 5 cubes in them.</p> <p>The next simplification comes by considering the arrangements of the very first piece; we have already decided that it will be a 5-cube piece. All such pieces are 3-cubes long in one of the axis - in other words, the key-cube of the piece can be moved in only 2 axis; moving it in the third axis would lead to the piece not fitting entirely in the box - violating the first constraint.</p> <p>Furthermore, it is not necessary to consider any of the rotations of the first piece. This comes from the fact that we could just as easily rotate the box around a fixed first piece and get to an equivalent arrangment. Armed with these facts, it can be seen that a well-selected first piece only has 4 potential positions - that is 2 orders of magnitude less than the originally quoted 648 positions.</p> <p><img src="/blog/content/2006/10/pieces3.jpg" alt="Piece 1 options" /></p> <p>The four positions for piece one</p> <p>Enough of the theoretical.</p> <h2 id="implementing-it-in-ruby">Implementing it in ruby</h2> <p>Some of the problems that had to be solved include:</p> <ul> <li>Representing the box and pieces</li> <li>Rotating the pieces</li> <li>For a given location in the box and rotation on each axis, does a piece fit?</li> <li>For a given location in the box and rotation on each axis, actually placing a piece</li> <li>Checking the box is ‘ok’ - whilst a piece may fit in the box, if it leaves a 3 (or less) cube void then the box is not ‘ok’</li> <li>Iterating over the pieces</li> <li>Detecting that a solution has been reached</li> </ul> <h3 id="representing-the-box-and-pieces">Representing the box and pieces</h3> <p>I decided to represent the box as a multi-dimensional array:</p> <pre> box = [[[0,0,0], [0,0,0], [0,0,0]], [[0,0,0], [0,0,0], [0,0,0]], [[0,0,0], [0,0,0], [0,0,0]]] </pre> <p>A <code class="language-plaintext highlighter-rouge">0</code> represents an empty space in the box. Any other value represents piece number occupying that spot.</p> <p>The pieces are represented in a hash, indexed by the piece number, with the cubes of each piece listed as offsets from the key-cube.</p> <pre> pieces = { 1 =&gt; [[0,0,0],[0,0,1],[0,1,1],[1,0,1],[1,0,2]], ... } </pre> <h3 id="rotating-the-pieces">Rotating the pieces</h3> <p>This led me to re-learn some basic geometry to do with rotation matrices. Given a matrix M representing a point in space and a matrix R representing the rotation to be applied, the resultant point is simply the product of the two. Ruby comes in very handy here as a simple <code class="language-plaintext highlighter-rouge">require 'matrix'</code> gave me all the matrix functionality I could wish for.</p> <p>To help with the efficiency of the program I implemented a simple caching function that meant I didn’t have to perform the multiplication every time a particular rotation was needed. I didn’t do any quantative measurements of what gain this gave, but it felt good to do.</p> <h3 id="checking-a-piece-fits">Checking a piece fits</h3> <p>For each cube in the piece, I knew its offset from the ‘origin’ location. If this location was outside of the box (ie &lt;0 or &gt;2) then the piece didn’t fit. If the location was already occupied (ie non-zero in the box array) then, again, the piece didn’t fit.</p> <h3 id="placing-a-piece">Placing a piece</h3> <p>Assuming the piece fits, placing it in the box is simply a question of iterating over its cubes and updating the box array in the appropriate locations to the piece number.</p> <h3 id="checking-the-box-is-ok">Checking the box is ‘ok’</h3> <p>Whilst the constraint to be checked is that there are no voids of 3 cubes or less, I found it was far easier to initially check for 1 cube voids. This is done by checking each cube in the box that is empty and seeing whether it has an empty neighbour - if so, it is ‘ok’ in this looser definition of the constraint.</p> <h3 id="iterating-over-the-pieces">Iterating over the pieces</h3> <p>I wrote a function that, for a given position and rotation, finds the next location for the piece by iterating over the remaining positions and rotations. To do this I defined a natural ordering of these properties so that locations are iterated over prior to rotations.</p> <p>If the function finds a suitable placement for the piece it returns an array of the position and rotation of the piece. This allows the code to store these values on a stack and progress to the next piece to try and place it in the current box.</p> <p>If the function fails to find a placement it returns an empy array that triggers the code to step back to the previous piece, obtain that piece’s current position and rotation from the stack and call the placement function with these values so the next location can be found and evaluated.</p> <p>This recursive method required the most thought, debug and rewriting.</p> <h3 id="detecting-a-solution">Detecting a solution</h3> <p>I ended up implementing a number of checks for a solution in paranoia that I had got a piece of logic wrong somewhere. One involved counting the number of pieces successfully placed into the box - once all were placed, a solution would have been found. Another was within the function to check the box for voids of appropraite size; if it found no voids whatsoever, then by definition the box is complete.</p> <p>Clearly there are plenty of implementation details I am leaving out here. The logic of the approach has been covered, which ought to be enough and is frankly the more interesting part.</p> <h2 id="finding-the-solution">Finding the solution</h2> <p>It is only fair to point out that I didn’t do a full pen and paper analysis of the problem before I started coding. This is demonstrated by the fact that the first version of the code ran for hours on end without getting anywhere. The single biggest efficiency saver was the code to detect one-cube voids in the current partial solution - this led to it taking minutes to cover ground it had previously taken all night to do.</p> <p>To help track it’s progress I had the code report how many positions it had considered for each piece. The theory was that the code should not have to consider any more than 4 positions for the first piece. However the code was getting to the 5th position for the first piece without a solution being found. This led me to find and fix quite a few bugs in the implementation. I also verified that the first positions considered for the piece were actually the right ones.</p> <p>Even with this, a solution was not being found. I decided to test the program with a set of pieces that I knew had a solution; the <a href="">Soma cube</a>. It did take a few modifications due to the different number of pieces etc, but it didn’t take long for it to find a solution - which, considering there are 240 solutions to the Soma cube, shouldn’t have been too hard. This helped to validate the overall correctness of the code.</p> <h2 id="the-solution">The solution</h2> <p>With this in mind I re-ran the original pieces through the program, but still without a solution being found. At this point I felt safe to declare that there <em>simply is not a solution for this particular set of pieces</em>.</p> <p><a href="/blog/content/2006/10/pieces2.jpg"><img src="/blog/content/2006/10/pieces2.thumbnail.jpg" alt="Mirrorable Pieces" /><br /> Alternative Pieces</a></p> <p>But I wasn’t going to stop there. I decided to investigate whether any of the pieces had been ‘altered’.</p> <p>Taking each piece in turn I mirrored it and ran it through the program. For two of the pieces, solutions were found.</p> <h2 id="going-forward">Going forward</h2> <p>Whilst I have some reasonably generic ruby code that can find solutions to this type of problem there are plenty of next steps that I could take. The first few that come to mind include:</p> <ul> <li>The existing code is very procedural - barely scratching the surface of what can be done with ruby. The ‘box’ and ‘piece’ concepts of the puzzle do lend themselves to a more object-oriented design and implementation. Ruby certain allows for this.</li> <li>The void-constraint checking can be improved to spot the 2- or 3-cube voids.</li> <li>The order the pieces were placed into the box was selected by-hand based on my own judgement of piece complexity. It would be interesting to programmatically identify piece complexity</li> </ul> <p>How much I actually do now that I have broken the bad news to Lucy that her puzzle isn’t solvable is another matter.</p> Little Moo has delivered 2006-10-03T00:00:00+00:00 <p>Little Moo, the print robot of <a href=""></a> has delivered. As <a href="/2006/09/21/moo-they-love-to-print/">previously</a> blogged, I ordered my free sample of moo-cards 10 days ago and they arrived today.</p> <p><a href=""><img src="" alt="Little Moo delivers - the packaging" /></a></p> <p>The cards arrived in a nice compact holder with a tear-drop hole in the front. This provides a great way to slide cards out with your thumb. The holder itself was gum-attached to a cool Moo postcard that I’ll be sure to add to my postcard-wall at work.</p> <p><a href=""><img src="" alt="Little Moo delivers - my free cards" /></a></p> <p>The problem with ordering 10 cards is that you have to be very selective of the photos you pick. As I mentioned in the last post, the Moo website has a great interface for doing this. I ended up going for a random collection of my photos - but I did pick a couple out that favoured the long and thin shape of the cards.</p> <p><a href=""><img src="" alt="Little Moo delivers - flight tag" /></a></p> <p>The cards came with a luggage-tag-esque card for my order; a nice way to include the order number with the delivery. I wonder if I could quote the number to Moo and get the order repeated - assuming it wasn’t one of the free 10-card packs.</p> <p>One piece of feedback I have for Moo is to be careful in their escaping of characters on the tag. Its amazing how often I find myself called Mr O'Leary.</p> Moo - they love to print 2006-09-21T00:00:00+00:00 <p><a href=""><img src="" alt="Moo - they love to print" /></a></p> <p>There’s a lot of buzz around at the moment for <a href="">Moo’s MiniCards</a> - business cards using your photos from <a href="">Flickr</a>. What’s even better is the offer to all Pro users to get 10 cards free - mine should arrive in the next 10 days.</p> <p>The site has been very well designed and is very simple to use. It took all of about 5 minutes to get my cards ordered - once I had picked the photos of course.</p> <p>At only $19.99 for a pack of 100, its a great way to get business cards with a very personal touch. Considering I don’t often have a reason to hand out a business card, 100 would last me a long time.</p> 2 Laps of the Earth 2006-09-17T00:00:00+00:00 <p>I’ve driven just over twice around the equator and its only taken me 5 years to do it.</p> <p><img src="/blog/content/2006/09/50k_speedo.gif" alt="50k Speedo" /></p> Planetary Matters 2006-09-10T00:00:00+00:00 <p>I have added a new section to the site - <a href="">Planet Knolleary</a>. This is a collection of the feeds I read on a regular basis. The software used is Sam Ruby’s <a href="">Venus</a> refactoring of the <a href="">Planet</a> software. It’s also currently using Sam’s own theme - something I will be replacing with a more in-keeping style. The list of feeds it’s subscribed to will vary as I get the balance right.</p> <p>Oh, and in the same style of my links, there are now the top 5 entries from the planet in the sidebar.</p> Overlapping Text 2006-09-06T00:00:00+00:00 <p>The parent hosting site for knolleary is <a href="">BlogsDollocks</a> - a hosting service that I have a hand in.</p> <p>Last night I spent some time reworking the BD homepage so that it includes a selection of blog entries from the hosted sites. This was easily achieved using the same <a href="/2006/08/30/changing-dates/">mechanism</a> as I used for the links in the sidebar.</p> <p>One of the existing design features of the page that <a href="">Kyb</a> put together was the BD text at the top of the page. On my laptop, running Ubuntu Dapper with Firefox, it looks like this:</p> <p><img src="/blog/content/2006/09/bds_logo_2.gif" alt="BlogsDollocks Logo on Laptop" /></p> <p>I was never quite sure about this overlapping design, but it wasn’t until this morning that I looked at the site with my desktop machine, running Debian with Firefox 1.0.4 that I saw it rendered like this:</p> <p><img src="/blog/content/2006/09/bds_logo_1.gif" alt="BlogsDollocks Logo" /></p> <p>This does lead me to wonder which is the correct rendering of the text.</p> <p>The html for the logo is simply:</p> <pre><div class="title"> <h1>Blogs Dollocks</h1> <h2>every blog has his day</h2> </div></pre> <p>and the css is:</p> <pre> div.title { font-size: 28pt; width: 6em; margin: auto; } h1 { font-family: perpetua, serif; font-weight: normal; font-size: 28pt; text-align: center; line-height: 0px; padding-top: 20px; } h2 { font-style: italic; font-weight: normal; font-size: 7pt; text-align: right; line-height: 0px; } </pre> <p>It seems that the <code class="language-plaintext highlighter-rouge">width</code> property of <code class="language-plaintext highlighter-rouge">div.title</code> is what is being handled differently. It all comes down to the magical <code class="language-plaintext highlighter-rouge">em</code> unit of measurement.</p> <p>The differing versions of firefox is a complete red herring. The key is the different resolutions the two machines run at - the laptop at a lowly 1024x768 and the desktop at 1600x1200.</p> <p>If I set the <code class="language-plaintext highlighter-rouge">width</code> to a more rigid value of <code class="language-plaintext highlighter-rouge">200px</code> then the text is rendered more consistently between the two browsers - in this case with the words overlapping as <code class="language-plaintext highlighter-rouge">200px</code> is not sufficient for the text to be layed out fully at its default font size. However this doesn’t work with zooming the page (Ctrl++/Ctrl+- on firefox).</p> <p>Having conferred with Kyb, the desired rendering is the properly spaced out version. As such, I have changed to css to:</p> <pre> div.title { font-size: 8em; width: 3em; margin-left: 200px; } </pre> <p>This keeps the text looking sane at both resolutions, and gracefully handles enlarging the text.</p> <p>Note I have also moved away from <code class="language-plaintext highlighter-rouge">margin: auto</code> so that the title stays in position as the screen size changes.</p> Right-Handed Bullets 2006-09-01T00:00:00+00:00 <p>In the endless process of refining the aesthetics of the site, I would like to have the list items in the sidebar have their bullets on the right-hand side of the item text rather than the default left.</p> <p>A quick google reveals the <code class="language-plaintext highlighter-rouge">direction</code> CSS property which <em>almost</em> does the necessary. Given a simple list like this:</p> <pre> <ul> <li>A</li> <li>B</li> <li>C</li> </ul> </pre> <blockquote> <ul> <li>A</li> <li>B</li> <li>C setting the <code class="language-plaintext highlighter-rouge">direction</code> property to <code class="language-plaintext highlighter-rouge">rtl</code> - ie right-to-left - generates this:</li> </ul> </blockquote> <pre> <ul style="direction:rtl;"> <li>A</li> <li>B</li> <li>C</li> </ul> </pre> <blockquote> <ul> <li>A</li> <li>B</li> <li>C</li> </ul> </blockquote> <p>On the face of it, this is ideal. But if you have a punctuation character in there, such as the post counts against the catergory links, you hit a problem:</p> <pre> <ul style="direction:rtl;"> <li>A(1)</li> <li>B+</li> <li>C?</li> </ul> </pre> <blockquote> <ul> <li>A(1)</li> <li>B+</li> <li>C?</li> </ul> </blockquote> <p>This is a direct result of how right-to-left text works in CSS. There are a couple of possible work-arounds:</p> <ul> <li>Set the <code class="language-plaintext highlighter-rouge">list-style</code> to <code class="language-plaintext highlighter-rouge">none</code> and set the background image of the element to include a bullet point on the right-hand side.</li> <li>Apply <code class="language-plaintext highlighter-rouge">direction: rtl;</code> to the <code class="language-plaintext highlighter-rouge">&lt;li&gt;</code> elements and have a <code class="language-plaintext highlighter-rouge">&lt;span&gt;</code> element with its <code class="language-plaintext highlighter-rouge">direction</code> set to <code class="language-plaintext highlighter-rouge">ltr</code> surrounding the content in order to pull it back into place. The first option feels dirty and I would rather not go near it. The second option seems easier if it wasn’t for the way Wordpress generates the list of categories. The function call to <code class="language-plaintext highlighter-rouge">wp_list_cats()</code> doesn’t allow for custom HTML in the output, so it isn’t possible to add the <code class="language-plaintext highlighter-rouge">&lt;span&gt;</code> element. I could write my own version of the function that did what I wanted - but that seems overkill.</li> </ul> <p>For now I will live with the left-hand bullets - any other suggestions for moving the bullet over are welcome.</p> Changing dates 2006-08-30T00:00:00+00:00 <p>Being the tinkerer that I am, rather than use one of the available <a href=""></a> plugins for Wordpress, I decided to write my own; the result of which you can see over in the sidebar - unless of course you are reading this via one of the feeds.</p> <p>There is a simple php script sat on that uses <a href="">MagpieRSS</a> to fetch and cache the feed for my bookmarks. When a page of the site is loaded, a little bit of javascript sends an async request to the script to fetch the html for the bookmarks. This prevents any slowness in fetching the feed from impacting the page load time.</p> <p>The principle of it works very nicely, however there appears to be an issue with it; on occasion it gets the 5 most recent entries wrong - not randomly wrong, but consistently the wrong 5 entries are shown.</p> <p>Having spent some time looking at this, I have found the problem - MagpieRSS uses the <code class="language-plaintext highlighter-rouge">&lt;dc:date&gt;</code> field of the feed to determine the order of the items. For a selection of the items in the feed, the correct values are:</p> <pre> GStreamer : 2006-08-29T08:22:15Z MVC : 2006-05-16T20:16:00Z Cheat : 2006-05-07T11:48:41Z Dojo : 2006-05-24T13:21:15Z Styles : 2006-04-22T10:25:02Z </pre> <p>and the rss feed agrees - most of the time. However, it ocassionally gets the date field wrong, as you can see for the same selection of items taken from a feed fetched seconds after the previous one:</p> <pre> Styles : 2006-08-29T14:00:00Z MVC : 2006-08-29T14:00:00Z Cheat : 2006-08-29T14:00:00Z Dojo : 2006-08-29T14:00:00Z GStreamer : 2006-08-29T11:13:20Z </pre> <p>Its clear that there is a problem with the feed. I have emailed their support reporting the issue - hopefully it’ll get resolved soon.</p> <p>For now, enjoy the (pseudo-random) links.</p> <p><em>Update</em>:</p> <p>I was slightly skeptical of the potentially nebulous means of the reporting the issue to via email to a generic support address. However, I can happily report that it took all of 30 minutes to get a reply. No fix yet, but will look at it soon - apparently today is pretty busy… I wonder what is about to emerge ;)</p> Take 2006-08-29T00:00:00+00:00 <p>Welcome to my latest attempt at a web presence that will hopefully stick around for sometime. Its early days here, but the site is beginning to take shape. I’m hoping I won’t feel the need to change the theme anytime soon… but you can never tell. I haven’t decided what this site is for yet but it will be interesting to see where it does lead.</p> <p>I will be adding more content soon - there are still a few little style/theme tweeks I want to play with.</p>