Background on my REST web service using Node.js: One of my REST web service applications, originally written in PHP several years ago, needed to be cleaned up. I wanted to re-factored it to make it easier for me to maintain. I had picked a PHP framework, and I was going to re-code.
For my own education, I decided to use this project as a trigger to learn Node.js.
I had been following the progress of the node community and some of the organizations at work had started using node, so I decided to give it a try. I had been using JavaScript for quite awhile doing front end programming in my browser applications and felt that it shouldn’t be too hard for me to learn.
The notes that follow capture some of my learnings and record some of the steps I followed to design and setup my new web service.
Install Node.js on my Fedora Server
I am a system admin of my own Fedora server, and I was looking to do a full install of nodejs. If you have a new Fedora sever (Fedora 18+), you just use “yum install nodejs” — but I have an older Fedora server so I had to dig around; I found several useful Howto’s. What worked best for me was the following steps:
From root login:
$ cd /usr/local
$ wget http://nodejs.org/dist/v0.10.25/node-v0.10.25.tar.gz
$ tar zxvf node-v0.10.25.tar.gz
$ cd node-v0.10.25
$ ./configure
$ make
$ make install
Source: Ask Fedora Project
I then put node into default login profile:
$ cd /etc/profile.d
$ vi node.sh
#insert following lines
export PATH=/usr/local/bin:$PATH
export NODE_PATH=/usr/local/lib/node_modules
ZZ
From the root home directory and from my user log in id, I verified that node worked:
$ node -v
v0.10.25
The in my user account, I wrote a hello world style test application:
$ cd $HOME
$ mkdir helloworld
$ cd helloworld
$ vi app.js
// Insert the following files
// I forget which howto website I borrowed this from
var http = require('http');
var server = http.createServer(function(req, res) {
console.log("web page accessed");
res.writeHead(200);
res.end('Hello Http');
});
server.listen(8000);
ZZ
$ node app.js
Note: the app.js is listening on port 8000. To make this work on my server, I needed to go into iptables and open port 8000. I am not going to show how to do that here, but it is easy to do and easy to overlook. It is worth the trouble to get this little app working because you’ll need port 8000 working for the next step.
Once port 8000 is open and the app.js script running, do the following command in a different terminal window.
From user login:
$ curl -i http://localhost:8000
Output:
HTTP/1.1 200 OK
Date: Thu, 13 Feb 2014 16:27:00 GMT
Connection: keep-alive
Transfer-Encoding: chunked
Hello Http This turned out to be harder than I thought. The nodejs website offers Linux Binaries; I tried them and they didn’t work. I assume I did something wrong. Instead, I did the install from source setup. On my machine, it took about 10 minutes to compile and build. (The make step was longer than I thought).
Restify Module
The node.js community has a large library of packaged modules. I picked the Restify module to build my package upon. Since I am new to node, I don’t really know what the best choice would be, I just wanted to leverage something that had some good howto articles.
To setup restify, I ran the following steps from my root login:
$ npm install restify -g
Then from my development login, I wrote a barebones test application:
$ cd $HOME
$ mkdir restify
$ cd restify
$ vi app.js
// insert the following lines
// borrowed from stack overflow article:
// http://stackoverflow.com/questions/17589178/why-should-i-use-restify
var restify = require('restify');
var server = restify.createServer();
function respond(req, res, next) {
res.send('hello ' + req.params.name);
}
server.get('/hello/:name', respond);
server.listen(8000, function() {
console.log('Listening on port 8000');
});
ZZ
$ node app.js
Like the previous hello world test, this one verifies that basic restify works. I ran “node app.js” in one terminal window and in the other, I ran the following:
$ curl -i http://localhost:8000/hello/jack
Output:
HTTP/1.1 200 OK Content-Type: application/json Content-Length: 12 Date: Thu, 13 Feb 2014 17:03:19 GMT Connection: keep-alivehello jack
I make this all look so easy, but along the way I had several gotcha type problems. I originally didn’t have the environment variables setup right. And, my initial installation of restify, I forgot to use the -g option. And for my initial setup, I did a couple of things in my root login that should have been done in my user login, etc. It is worth it to get the basic hello world done before you start.
Reference: Creating a basic node.js api with restify and save by Dom Udall
My REST Design
My web application edits and reports on some resources stored on my server. I want to grant access to these resource through a REST-ish API. The two key resources are charts and jefiles — it turns out that these are actually text files stored on my server. There is one chart file per year, so in my REST URI I use the notation: chart/YY. The jefile files are monthly, with 12 months per year, so I picked the notation jefile/YY/MM. In my REST URIs, MM=00-12 and YY is any two digits representing a year (eg YY==12 implies 2012, YY==98 implies 1998).
So, my design is for the following “GET” operations return data from the server to my web application:
# returns contents of chart file for the year YY, # where YY is 13 for 2013 http://localhost:8000/chart/YY # returns contents of jefile for the year YY, Month MM # - where YY is 13 for 2013 # - where MM is 01-12, for a month http://localhost:8000/jefile/YY/MM # returns concatenation of all jefiles for the year YY # - where YY is 13 for 2013 (used for year to date reports) http://localhost:8000/jefile/YY
Also, for debugging purposes, I want both plain text and json formatted data returned. I picked the above syntax to represent plain text response. Here’s the syntax for json responses:
http://localhost:8000/chart.json/YY
http://localhost:8000/jefile.json/YY/MM
http://localhost:8000/jefile.json/YYI have experimented with using MIME types or other header mechanisms to control plain text vs json. I like this method because it lets me debug in any a normal browser; no special REST browser plugins needed to be installed. This approach is also valuable to me because it maintains continuity with my existing, several year old, PHP design approach.
So with this design, here’s the basic restify application.
From the file app.js:
var restify = require('restify');
var server = restify.createServer();
server
// Allow the use of POST
.use(restify.fullResponse())
// Maps req.body to req.params so there is no switching between them
.use(restify.bodyParser());
server.get('/chart/:YY', getChart);
server.get('/chart.json/:YY', getChart);
server.get('/jefile/:YY', getJefileForYY);
server.get('/jefile.json/:YY', getJefileForYY);
server.get('/jefile/:YY/:MM', getJefile);
server.get('/jefile.json/:YY/:MM', getJefile);
server.put('/jefile/:YY/:MM', putJefile);
server.listen(8000, function() {
console.log('Listening on port 8000');
});
The server.get() callback functions (eg getJefile() ) all follow a common pattern:
function getJeFile(req, res, next) {
console.log(params.req.YY);
console.log(req.route.name);
...;
res.send(...);
next();
}
The restify framework’s server.get() function takes the incoming REST request, parses the URI and matches it to the first parameter of the server.get() function, and runs the callback function with the request neatly parsed into the object req. Since in REST APIs, the GET method nominally has an empty message body, the only real input are the parameters parsed out of the URL. In my code above :YY and :MM ids are the names of the URI parameters.
These parameters are parsed by restify and stored in the object params.req. For all of my server functions, I always have params.req.YY as in input. Restify API document covers this, but again, this design pattern is found throughout nodejs code as a way for a chain of functions to all share the same input and output objects.
The callback functions above (eg getJeFile()) generates the content that gets stored into the response object, the 2nd callback parameter res. When the callback function is complete, it calls next(), the 3rd parameter.
So here’s what my 3 server.get() callback functions look like.
From app.js:
getChart = function getChart(req, res, next) {
console.log(req.route.name);
var year = req.params.YY;
// validate year
// get contents of chart file
// if I find any errors: return next( Error object )
res.send(chartFile);
return next();
}
getJefile = function getJefile(req, res, next) {
console.log(req.route.name);
var year = req.params.YY;
var month = req.params.MM;
// validate year, month
// read contents of jefile file
// if I find any errors: return next( Error object )
res.send(jefileFile);
return next();
}
getJefileForYY = function getJefileForYY(req, res, next) {
console.log(req.route.name);
var year = req.params.YY;
// validate year
// get contents of all of the jefile files, concatenate
// If I find any errors: return next( Error object )
res.send(jefileFileForYY);
return next();
}
Note: the above code is simplified to better help explain what I am doing.
For the case where my web application wants to write content to the server, I use the REST PUT method. The PUT method writes or overwrites the full contents of a resource. Since I am working with whole files on the web server, PUT is appropriate. POST would be OK, if I was adding blocks of data to an existing file; but I am not.
So to make a PUT work in the restify framework, my server.put() callback function needs to parse and validate the message body, convert it from JSON to a native format then save the contents to a file on the server. Restify makes this pretty easy. Here’s the basics:
From app.js:
putJefile = function putJefile(req, res, next) {
console.log(req.params);
console.log(req.route.name);
var year = req.params.YY;
var month = req.params.MM;
// validate year
// validate requerst body, named jefilejson, is valid
jefileFile = JSON.pars(req.params.jefilejson)
// write contents of jefileFile to server, directory YY, file-MM
res.send(200)
return next();
}
Note: the message body holds a json object; in my case, the web application names the json object jefilejson. So to access the contents of the message body, you access req.params.jefilejson.
Validating the MM and YY ids
The server.get() routing function’s syntax makes it easy to match roughly correctly formatted URIs, but for me, it didn’t have enough checking power to verify that the YY and MM ids were correct. Server.get() just matches any character between the slashes; it doesn’t check that YY is 2 numeric digits or that MM is 01-12 only with the leading zero required. I had to add my own validation logic in each of my callback functions.
It turns out the code for verifying YY and MM ids was several lines of coded repeated multiple times. Node.js and restify design patterns give you the option of inserting additional callback functions into the server.get() function call. I added validation functions as follows:
From app.js:
server.get('/chart/:YY', validateYY, getChart);
server.get('/chart.json/:YY', validateYY, getChart);
server.get('/jefile/:YY', validateYY, getJefileForYY);
server.get('/jefile.json/:YY', validateYY, getJefileForYY);
server.get('/jefile/:YY/:MM', validateYY, validateMM, getJefile);
server.get('/jefile.json/:YY/:MM', validateYY, validateMM, getJefile);
server.put('/jefile/:YY/:MM', validateYY, validateMM, putJefile);
Now, I liked this because it let me re-factor a bunch of common code out of each of my callback functions. I have seen other node.js code snippets that take advantage of this coding pattern.
Validation Functions and Error Handling.
Within a nodejs or restify program, there’s a common way of handling errors. Since everything is asynchronous and chained together with next() functions, there needs to be a way to break the chain on error conditions. The key technique is to call the next() function with an Error object as a parameter. Let me show how I coded the validateYY() function.
From app.js:
/*
** Validate YY id.
** -- syntax: 2 digits, 00-99
** -- directory nfroot+YY must exist
*/
function validateYY(req, res, next) {
var ret = 500;
var year = req.params.YY;
console.log('validateYY:'+req.route.name);
var isValidYY = year.match(/^\d\d$/);
if (!isValidYY) {
ret = new Error("The field YY-'"+year+"' is not valid. YY is a two digit year where YY is 00-99");
ret.statusCode = 400;
return next(ret);
}
/* Verify that YY exists */
if (!fs.existsSync(nfroot+year)) {
ret = new Error("The field YY-'"+year+"' is not valid. The diretory-'"+nfroot+year+"' doesn't exist on server.");
ret.statusCode = 404;
return next(ret);
}
/* Verify that YY is a directory */
if (!fs.lstatSync(nfroot+year).isDirectory()) {
ret = new Error("Server side problem: The path -'"+nfroot+year+"' is a file, but must be a directory.");
ret.statusCode = 500;
return next(ret);
}
return next();
}
The key is the Error object. When I first saw code snippets using the Error object, I didn’t really know what’s going on. I Googled it and found there’s a rich design pattern built into node.js around how errors are handled. If you just jump into node.js like me, there’s lots of things you need to learn. Restify just leverages Error, assuming coders already know how to work node.js.
Notice in the code above, I tried really hard to follow best practices for mapping http error codes into my REST API definitions. For the case of someone trying to access a resource like a particular jefile file within a year (eg ..jefile/YY/MM): If the syntax was wrong, I returned a Bad Format (400) error; if the format of the REST URI was right but the file requested file didn’t exist I returned a Not Found (404) error. Also, I even verified that YY in the path was really a directory on the server; if not, that’s shouldn’t ever happen — I returned a Server Error (500).
The way the function chaining works: if next() is passed an Error object, the chaining stops and a json formated response is generated with Error information. This powerful construct is built into node.js, but if you didn’t know it, you might start inventing it yourself, especially if you come from a PHP background, like me.
Splitting Node applications into multiple files – Module Loading with Require
When I first started my node project, I put everything into one big app.js file. This got too big and clumsy for me. I am used to working on projects where coding is split across multiple source files. The node documentation defines a method for split source code across multiple files… similar to how multiple modules are handled in web browser applications.
I decided to dedicate a source file to each of my main REST resources. For this writeup, those would be chart.js and jefile.js. I still have a main app.js. The way to link them together is to use the require() functions at the top of each file:
Contents of app.js:
var restify = require('restify');
var chart = require('./chart');
var jefile = require('./jefile');
...
server.get('/chart/:YY', validateYY, chart.getChart);
server.get('/chart.json/:YY', validateYY, chart.getChart);
server.get('/jefile/:YY/:MM',validateYY,validateMM, jefile.getJefile);
server.get('/jefile.json/:YY/:MM',validateYY,validateMM,
jefile.getJefile);
server.put('/jefile/:YY/:MM',validateYY,validateMM,jefile.putJefile);
Contents of jefile.js:
exports.putJefile = function putJefile(req, res, next) { ... }
exports.getJefile = function getJefile(req, res, next) { ... }
Contents of chart.js:
exports.getChart = function getChart(req, res, next) { ... }
Note a few things: the require() function for restify does not need a path. I installed the restify NPM globally (-g option). The require() function is only needed in the main file, app.js. The restify routing functions (eg server.get()) references callback functions from the other source files, by prefixing with chart or jefile. And, the source code for those callbacks are written as exports (eg. exports.getJefile = …).
Note how I managed, syntactically, my REST GET API methods for returning plain or JSON responses. ../chart/YY returns the contents of the chart file for year YY; ../chart.json/YY returns the same file, but in JSON format. Visually, I would rather see the contents of the chart file in native format, but for web page development, I want all responses in JSON format. The server.get() callback for both of these cases is getChart(). The way the getChart() function knows to return json or native format is by looking at the req.route.name parameter; that tells me how the getChart() was called.
The way source files are modularized is a pattern is written about in the nodejs documentation, but it turns out there are many more advanced ways of managing modules. This one worked great for me. It also made it easier for me to debug and test each resource without wrecking, accidentally, the others.
Reference: Node API Docs: Modules
Run Node and Apache both on Port 80
As I get my REST web service using Node working, I need to integrate it into the web service infrastructure already installed on my server. Port 80 today runs Apache serving blogs, weather stations, and a couple of web applications — all intertwined with Apache and I cannot just start over.
So I found a way to get Apache to work with node; I keep node on port 8000 and Apache on port 80 and configure an Apache “ProxyPass” for URLs to the node-based REST service.
In /etc/httpd/conf/httpd:
<VirtualHost 64.53.181.XXX 192.168.100.XXX >
ServerName mydomain.com
...
ProxyPass /node http://localhost:8000/
...
</VirtualHost >
Any URI with mydomain.com/node in it gets redirected to my node service. This is good, but part of the reason for moving to node is that it is supposed to give much improved performance — not that Apache is slow, but because it has years of historic features built into it that take longer to execute.
Well someday, I’ll run only node. But for now, for me, node must coexist and this method works for me.
Conclusions
Working with Javascript on the server side was nice. For me, the debugging skills that I use for web page development carried over nicely to node development. Nodejs’s asynchronous design pattern was new to me, but once I caught on, it wasn’t bad. The way nodejs standardizes error handling is much better than the adhoc approaches that I have been using with PHP.
I found that I missed some of the PHP built in functions. For example, to access files on the server, nodejs has a very robust File System module. It behaves differently from PHP, but I was able to figure it out — they are all based on the same underlying C libraries. I resisted the temptation to use one of the frameworks that help bridge PHP developers into nodejs; they aren’t bad, but they worked against my objective to learn.
Some next steps: I would like to do some performance comparisons. Superficially, my REST API re-written in node is faster than the existing PHP implementation. I’d like to quantify. Also, I have written server side code, that I ought to be able to use inside the web client applications; that would be a nice thing to clean up. Also, I’d like to get my node application to do logging at least equal to the logging I get from Apache — restify has some nice built in features for that.
There’s a big library of NPM modules out there. I’d like to learn more about them. This time around, all I really needed was restify. Maybe next time, I’ll try to use more of the features inside of it, or branch out and learn express. Anyway, thanks to the greater nodejs community for helping me get started.
