Automating data collection

Collecting data

Collecting data comes in many shapes and forms, there are many options out there from coding it yourself to using commercial tools. After doing some coding myself, which turned out to be difficult to maintain and sometimes a long process (more than 60 minutes) of adding a new data source, i decided not to the the coding by hand, but started to look for other free and readily available options.


I ended up using Node-RED, a flow based application, written in Node.js by Nick O'Leary. It is a very easy to use tool, which requires only a little javacript/json knowledge to get started with. It allows you to wire up a simple diagram to act with, in my case, public web api's in no time.

It supports a set of build in 'nodes' or logic blocks which can be configured through the GUI. Each node can be linked to other nodes to create the logic flow required to collect the data.
There are already a lot of nodes build-in which should get you a long way in setting up flows.

These nodes can be dragged-n-dropped from the toolbox on the left.

An example of the flow i use for retrieving data from efergy :

An example of the http request node configuration :

As you can see it is pretty easy to set and wire up a task that will read data and store it.


Node-RED also has a modular based design, which allows you to add any node or functionality that isn't already present by creating a couple of js & html files. More information on this can be found in the Node-RED documentation. But before creating a node one should check the already available community nodes that can be found on npmjs. Nodes are available under the naming suggested convention of node-red-node for built in nodes and node-red-contrib for community nodes.

Setting up Node-RED

To install Node-RED you will need to install Node.js first. After installing Node.js, create a folder where you want to install Node-RED, open a command prompt in that folder and invoke the npm package manager (which should be installed together with Node.js) :

npm install node-red

If you get any errors, see the troubleshooting section at the bottom of this page.

When Node-RED is installed you can start Node-RED from the command prompt by going to the node_modules\node-red folder and execute the following command :

node red

It will then start the webserver and display the port it is running on :

29 Nov 12:10:50 - [red] Server now running at

Open a web browser to the url specified by the server and you can start creating the flows :

Another way of installing Node-RED is by downloading the latest release, more information on this can be found on the Node-RED installation page.

To install additional contrib nodes, open the command prompt in the nodejs folder (where red.js is) and use npm to install a node package :

npm install node-red-contrib-postgres

Troubleshooting Node.js

It is possible that you get MSBUILD errors that Microsoft.Cpp.Default.props was not found, or other MSBUILD errors, which is required by Node.js to rebuild some c++ dependency modules of Node-RED, then either no Visual Studio is installed or it is not using the installed version.

To install Visual Studio, download and install an express edition from Microsoft, any edition which includes c++ will do.

If the installation of Node-RED still gives MSBUILD errors, try the following command :

npm install node-red --msvs_version=2013

This can also be used if you already have a version of Visual Studio installed, only replace the 2013 with the version that you have installed.

More information on this can be found on the node-gyp page.

comments powered by Disqus