Moving Node-RED to a monorepo with multiple modules21 Mar 2019
How we tore apart the internals of Node-RED and glued it back together without users knowing
There are two logical parts of Node-RED; the runtime where flows run and the editor where flows are edited. Ever since the start of the project, these two parts have been bundled together in a single blob of npm module.
With the 0.20 release we’ve just published, the internals of Node-RED have now been
split apart into 6 separate npm modules, along with the original
that now has the task of pulling those modules back together so the user doesn’t
know what we’ve done.
This post describes how we went about doing that and some of the challenges we faced along the way. If you want to see it for yourself, the code is here.
This blog post was presented at LNUG in September 2019.
When I started looking at how to structure the code to support this approach, I had to decide whether to keep all the code in one repo or to split it into one repo per module.
Splitting it out would have made it clear what code belongs to each module and make it easier for new developers to follow the structure.
But it would also greatly increase the administrative burden; multiple repositories to manage, with multiple issue lists and the need to carefully co-ordinate pull-requests when a new feature spanned multiple modules.
Keeping the code in one place made the most sense. The question was then how to do that in practice.
I looked around at other projects that maintain multiple modules in a single repository. There seemed to be a split between ones that chose to use established tooling, such as Lerna and those that rolled their own solution.
Not wanting to needlessly reinvent the wheel, I spent some time playing with Lerna.
Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.
The problem I found was Lerna has grown over time and does a lot. That isn’t necessarily a bad thing, but I found it hard to visualise how we’d migrate into using it. I got too caught up bouncing between the different options it provides without settling on one approach.
I also found this post from Nolan Lawson on why PouchDB moved away from Lerna that gave some hands-on perspective - albeit from a while ago now.
Ultimately I decided I wanted to understand the code structure and consequences of the split, rather than instant delegate that to the tooling. It wouldn’t preclude us from adopting Lerna in the future - but we’d be able to do that better informed.
Laying out the code
The existing code structure was already split between the node.js based runtime and the browser-based editor:
├── editor // All editor src and resources ├── nodes // The default core nodes ├── red // The node.js runtime code └── test // All test material
The primary target for repackaging was all of the node.js code under the
directory. The code was already reasonable well componentised under there, but it
wasn’t perfect. It was littered with
require statements with relative paths
that made assumptions about where particular files were.
The main challenge was figuring out a code layout that would allow the
statements to be updated to the new module structure, whilst still just working
when run in the development environment.
This is where I came across an approach linked to in Nolan’s post - the “Alle” model.
First a quick detour into how node does module loading. When you call
with a relative path, node loads that file directly. For example, given a pair
of files in adjacent directories:
. ├── a │ └── index.js └── b └── index.js
The code in
a/index.js can use the following code to load
const moduleB = require("../b/index.js");
Now, lets say
b are properly formed npm modules - so they include a
package.json file. Rather than require using a relative path, we want to require
using the name of the module:
const moduleB = require("b");
When you pass the name of a module to
require, node will check the current
directory for a
node_modules directory and look in there for a module with that
name. If it doesn’t find one, it then checks the parent directory for
and so on until it reaches the root of the filesystem.
We could take advantage of that in layout out the code - by adding a
directory in the structure:
. └── node_modules ├── a │ └── index.js └── b └── index.js
Now, when module
require("b"), node will search up the directory structure
node_modules directory and then find module
b in there.
Putting this into practice led to a structure of:
├── packages │ └── node_modules │ ├── @node-red │ │ ├── editor-api │ │ ├── editor-client │ │ ├── nodes │ │ ├── registry │ │ ├── runtime │ │ └── util │ └── node-red └── test ├── editor ├── node_modules │ └── nr-test-utils ├── nodes └── unit ├── @node-red │ ├── editor-api │ ├── registry │ ├── runtime │ └── util └── node-red └── lib
You can see the seven new modules under
packages/node_modules directory. They
require each other just as they will when properly installed with npm.
The test material was also restructured to match the layout. You may also spot
the same trick was used there to make a module called
to the test material.
This module provides two functions:
resolve. They can be used by
the test material to require a particular file from the source tree without having
to hardcode the relative path from the test material tree into the source tree.
Getting GitHub to not ignore node_modules
One downside of this approach is that many IDEs are told to ignore
directories as they don’t typically contain code a developer is expected to edit.
Adding some rules to
.gitignore to not ignore these directories seemed to fix it
for my preferred editor, atom.
node_modules !packages/node_modules !test/**/node_modules
We also found that GitHub would not generate diffs when showing changes to any files
under those directories, so we had to add
.gitattributes file containing the following:
That works on the desktop view, but the mobile view still suppresses diffs - not found a solution for that yet.
Each of the module directories has its own
package.json listing its dependencies
as normal. There is also a
package.json file at the top level of the project
that lists all dependencies (including development dependencies). This means
we don’t have to run
npm install in each module directory - in fact, we actively
avoid doing that because those dependencies include references to our other modules
that npm won’t be able to install itself.
This does mean there is an overhead around managing the dependencies.
Any new dependency needs adding in two places; the top-level package.json file so it gets installed in the development environment and the module’s own package.json file so it gets installed when the published module is installed.
As soon as you have the same piece of information in two places, you raise the risk of them getting out of sync.
The ‘Alle’ method I linked to earlier talks about automating the generation of the individual package.json files - something we haven’t adopted.
Instead, to help manage this, a new script was added,
that checks that every dependency listed in individual module package.json files
is also listed in the top-level package.json and that the version specifier matches.
The default set of tests now includes this check, so a build will fail if a mismatch
is found. The script can also be run with
--fix to automatically update the versions
in the module package.json file to match the top level one.
The one scenario this doesn’t catch is when a new dependency is added to the top-level file, but not to the individual module. The unit tests will still pass because the module is installed at the top level - but the published module will be missing it. We’ll have to be careful around that until we plug the gap.
Another important design decision was how to manage the version numbers of the individual modules. For example, if a fix was needed in one module would we publish a new version of just that module, or would we bump the version of all the modules.
npm makes it easy to take either approach. If we wanted to be able to publish
modules individually, we could set the dependency version numbers to
they would always get the latest version available in a given minor release.
The alternative would be to set them to a specific version to tie all the modules at
a particular level.
The problem with not keeping the versions closely aligned is what happens when a user hits a problem. The main goal of this entire refactoring was to hide the internal details of the split.
If a user hits a problem today, they can tell us what version of Node-RED they
have installed with a single number. It simply wouldn’t meet our goal if they
had to provide the versions of all seven modules so would know exactly what they
had installed. It would also be confusing to say they need to update their
install to get a fix but the
node-red module version doesn’t change.
So for now we’re going to keep all the modules in sync.
To help with that task, another script was added,
that can be used to update all of the individual
package.json files with the
right version number.
Building a release
We already had a build task,
grunt release, that took the source tree and built
the module that can be published to npm along with a zip file to upload to the
That task has been updated to now build the 7 individual modules, this time packed as tgz files.
.dist ├── modules │ ├── node-red-0.20.3.tgz │ ├── node-red-editor-api-0.20.3.tgz │ ├── node-red-editor-client-0.20.3.tgz │ ├── node-red-nodes-0.20.3.tgz │ ├── node-red-registry-0.20.3.tgz │ ├── node-red-runtime-0.20.3.tgz │ └── node-red-util-0.20.3.tgz └── node-red-0.20.3.zip
Those tgz files can be published one at a time to npm - ensuring the
module is done last. Currently that’s a manual task and something that is ripe
for automating in the future.
On reflection, having published the 0.20 release and a handful of subsequent maintenance releases, I’m pretty happy with the approach we took. Aside from a few more bits of task automation we could add, I think we have created a project structure that is well defined and easy to work with.
The main success has been that we did this without a single issue from a user related to the module structure or packaging. Users have no idea we made these changes - unless they read the release notes where we keep talking about it.