Production Ready Node: Configuration

onfiguration is an important piece to not only making your application deployable, but flexible and accommodating to change. We can't predict how an application will need to accept configuration between a local setup, build servers, staging and production. More over, if any of those things were to change, large parts of the application would need to change how it handles configuration. Some systems prefer files, some prefer environment variables some prefer command line arguments.

The Basics

A production ready application should be able to collect configuration from multiple sources and reconcile them into a single entity that is referenced throughout the application. We're going to walk through a sensible way to do this. The first thing we need to consider is sources of configuration, and the order in which we want to combine them. I've found that the following order seems to hold up pretty well:

System Defined overrides
Command Line Arguments
Environmnet Variables
Configuration File(s)
System Defaults.

The second thing that we will want to consider is how people can actually define their settings. Because we are dealing with JavaScript, it makes sense to define our configuration files as JSON. This allows developers create name-spaces and hierarchies of settings. Luckily, there are a number of NPM modules that do much of this word for us. In this situation, we're going to use nconf. With nconf, this is actually really easy to stub out.

// conf.js
var nconf = require('nconf')

module.exports = nconf
				.overrides( { /* something that must always be this way */} )
                .argv()
                .env({separator:'__'})
                .file('path/to/a/file')

This little snippet of code covers most of our uses cases. As long everything in the application that needs to use some kind of configurable value does so by pulling it from this module, we won't need to change much when change happens. The really great thing about nconf is that it stores everything as JSON and lets you access nested properties with a colon( : ). For example, if our config file looked like this:

{
	foo:{
    	bar:{
        	baz: 1
        }
    }
}

Our config module would let us change that value by doing one of the following

# cli flag
node app --foo:bar:baz=2

# environment variable
foo__bar__baz=2 node app

# predefined environment
export foo__bar__baz=2
node app

These all resolve to the same thing with in nconf and allow you to do the following

var conf = require('./packages/conf')
conf.get('foo')  // { bar: { baz: 2 } }
conf.get('foo:bar') // { baz: 2 }
conf.get('foo:bar:baz') // 2

We are able to use the double underscore ( __ ) in place of the colon because to specified it in the .env() call. We do this because bash doesn't like periods or colons in environment vars, they already have meaning.

Configuration Files

We run into a couple of problems when it comes to files. Our snippet only deals with a single file, and it is hard coded. Which means a file would need to exist in the exact location, with the exact same name in every deployment - production or otherwise. If we look back to the previous post on project structure, this also doesn't take into account any of the pluggable packages. Which would mean our configuration file would become the only place people can put configuration, and really nasty to maintain. We'll want to code defensively here and come up with a better way to deal with this.

Defensive Coding

Operating under the assumption that anything that can change. Applications that are coded defensively will making room for and accomidations for change to happen.

We'll want to modify our config module to take into account default settings defined by other packages in one or more files that can be a javascript or json file. We'll also want to search a number of different locations for our primary config file as well as allow the location of where it lives to be passed in so it can be specified if none of the other locations we've specified are acceptable by the person doing deployments. This is going to require a little bit of work on our part.

Default File Locations

Let's start by tackling default file locations. We want a predefined set of file paths to look in for a config file, search for package configuration, as well as allow a file path to be passed in. The latter is going to be the tricky one.

// conf.js
var nconf   = require('nconf');
var fs      = require('fs');
var startup = nconf
         .argv()
         .env({separator:'__'})
         .defaults( require('path/to/defaults') );
         
// get a conf
var configFile = path.resolve( startup.get( 'conf' ) || 'project.json' );      

// purge the start up config
startup.remove('env');
startup.remove('argv');
startup.remove('defaults');
startup = null;

var conf = nconf
			.overrides( { /* something that must always be this way */} )
            .argv()
            .env({separator:'__'})
                
if(  fs.existsSync( configFile ) ){
   if( fs.statSync( configFile ).isDirectory() ){
     // if it is a directory, read all json files
     fs
        .readdirSync( configFile )
        .filter( function( file ){
           return (/\.json$/).test( file );
        })
        .sort( function( file_a, file_b ){
           return file_a < file_b;
        })
        .forEach( function( file ){
           var filepath = path.normalize( path.join( configFile, file ) );
           conf = conf.file( file, filepath );
        })  
   } else{
    // if it is a file, read the file
    conf = conf.file( configFile );
   }
}

// set up defaults
conf
	.defaults( require('path/to/defaults') );

The first time the config module is required, it will load the bare bones environment mainly to determine if a primary config file location has been specified. If one has, it will attempt to read it. If that location it a directory, filter out json files, sort them by name, and normalize the paths. So rather than having a single massive file, it can be broken up into multiple files. We also sort them and apply files in an over-riding fashion. This will allow people to use simple file name conventions to determine how they are loaded.

+
|-- configuration/
|   |-- 10-defaults.json
|   |-- 20-core.json
|   `-- 30-mail.json

This would load files in order of 10-defaults < 20-core < 30-core where the values in 30-core have the final say in what is what. And this directory can be loaded with a similar invocation as above.

node app --conf=./configuration --foo:bar:baz=2

We have accounted for a few of the primary problems with configuration files. Config files no longer need to be in a static location, or kept under version control. Additionally, we can have an entire directory of smaller configuration files that have a narrow focus rather than a single monolithic file. The next thing that we want to account for is additional, common directories that configuration files might live. This is in an effort to account for places that automated configuration management tools like chef might be set to generate configuration for your application. Common places for config files in automated deployments might be:

/etc/
in a .config directory of the user's home directory
the project root

This is pretty easy to add in to our conf module.

// conf.js
var nconf   = require('nconf');
var fs      = require('fs');

// order matters, otherwise this could be an object
var lookuppaths =[
   ['project', path.normalize( path.join(PROJECT_ROOT,"project.json") )]
 , ['home',path.normalize( path.join(( process.env.USERPROFILE || process.env.HOME || PROJECT_ROOT ),'.config', "project.json") ) ]
 , ['etc', path.normalize('/etc/proejct.json')]
]

var startup = nconf
         .argv()
         .env({separator:'__'})
         .defaults( require('path/to/defaults') );
         
// get a conf
var configFile = path.resolve( startup.get( 'conf' ) || 'project.json' );      

// purge the start up config
startup.remove('env');
startup.remove('argv');
startup.remove('defaults');
startup = null;

var conf = nconf
			.overrides( { /* something that must always be this way */} )
            .argv()
            .env({separator:'__'})
                
if(  fs.existsSync( configFile ) ){
   if( fs.statSync( configFile ).isDirectory() ){
     // if it is a directory, read all json files
     fs
        .readdirSync( configFile )
        .filter( function( file ){
           return (/\.json$/).test( file );
        })
        .sort( function( file_a, file_b ){
           return file_a < file_b;
        })
        .forEach( function( file ){
           var filepath = path.normalize( path.join( configFile, file ) );
           conf = conf.file( file, filepath );
        })  
   } else{
    // if it is a file, read the file
    conf = conf.file( configFile );
   }
}

// loop over paths and load them in
lookuppaths.forEach(function( lp ){
   conf = conf.file( lp[0], lp[1] )
});

// set up defaults
conf
	.defaults( require('path/to/defaults') );

Pretty simply, we just make an array of normalized paths that point to locations, and try to load it in. If the file doesn't exist, nconf will ignore it. There isn't anything particular about these paths, feel free to change them as you need.

Now we have tackled a number of our problems with config files. We can deal with multiple file, from multiple locations, and accept an additional directory of files or file from either cli input or the host environment.

Packages Specific Configuration

The last thing we want to account for is configuration located in our internal, pluggable packages. This is the one place where we can feel OK with configuration being javascript modules rather than just json files. Some times configuration requires some path resolution or other calculation before final values can be set. Luckily require makes this pretty simple to do.

// conf.js
var nconf   = require('nconf');
var fs      = require('fs');

// order matters, otherwise this could be an object
var lookuppaths =[
   ['project', path.normalize( path.join(PROJECT_ROOT,"project.json") )]
 , ['home',path.normalize( path.join(( process.env.USERPROFILE || process.env.HOME || PROJECT_ROOT ),'.config', "project.json") ) ]
 , ['etc', path.normalize('/etc/proejct.json')]
]

var startup = nconf
         .argv()
         .env({separator:'__'})
         .defaults( require('path/to/defaults') );
         
// get a conf
var configFile = path.resolve( startup.get( 'conf' ) || 'project.json' );      

// purge the start up config
startup.remove('env');
startup.remove('argv');
startup.remove('defaults');
startup = null;

var conf = nconf
			.overrides( { /* something that must always be this way */} )
            .argv()
            .env({separator:'__'})
                
if(  fs.existsSync( configFile ) ){
   if( fs.statSync( configFile ).isDirectory() ){
     // if it is a directory, read all json files
     fs
        .readdirSync( configFile )
        .filter( function( file ){
           return (/\.json$/).test( file );
        })
        .sort( function( file_a, file_b ){
           return file_a < file_b;
        })
        .forEach( function( file ){
           var filepath = path.normalize( path.join( configFile, file ) );
           conf = conf.file( file, filepath );
        })  
   } else{
    // if it is a file, read the file
    conf = conf.file( configFile );
   }
}

// loop over paths and load them in
lookuppaths.forEach(function( lp ){
   conf = conf.file( lp[0], lp[1] )
});


packagepaths = fs
               .readdirSync( PACKAGE_PATH )
               .filter( function( dir ){
                  return /project-/.test( dir )
               })
               .map(function( dir ){
                  return path.join(PACKAGE_PATH, dir, 'conf' )
               })

// collector for all package defaults 
var defaultCfg = {};

packagepaths.forEach(function( pconf ){
   var config;
   try{
      config = require( pconf )
      defaultCfg = merge( defaultCfg, config )
   } catch( e ){
      console.log('unable to load %s: %s', pconf, e.message )
   }
});

// set up defaults
defaultCfg = merge( defaultCfg, require('path/to/defaults') )

conf
   .defaults(defaultCfg);

This will now run through our packages directory looking filtering out folders that do not adhere to our naming convention of project-<NAME> and tries to require(project-<NAME>/conf). If required returns something, we merge it into our defaults object. And that is it. That means packages contain either:

conf.js
conf.json
conf.node
conf/index.js

+ packages/
  |-- project-foo/
  |   `-- conf.json
  |-- project-bar
  |   |-- conf/
  |   |   |-- a.js
  |   |   |-- b.js
  |   |   `-- index.js

Packages can contain some complex configuration setup, and because we are letting require do the heavy lifting, packages authors can do some interesting things to get the set up they need.

Recap

This is a fairly simple module provides a rather robust way to get configuration into your node apps. This config module will collect data from the host environment, multi configuration files, command line arguments, and event complex package specific configuration through a single unified interface. Using a simple module like this will allow developers and admins configure your application any way they see fit for any environment.