File Uploads With Node Tastypie & Hapi

ealing with files and handling uploads is an ugly reality in web applications. It can be even more unpleasant if you application is driven by REST APIs. In days passed, it often came down to flash uploaders talking outside the api and someone having to link multiple data sets together. I was always partial to the fantastic FancyUploader Library. Fortunately, things have gotten better. Node, and Hapi make dealing with incoming files much easier. More over, Tastypie & Hapi make this exceptionally easy to do this in a single Api Endpoint. To illustrate this we are going to build up a small Api to store and save some binary data.

To accomplish this, we need to do 4 things:

Define A Model ( to store data in Rethinkdb )
Define A Resource
Define A Custom Route
Define A Custom Field

Define A Model

We are going to need a place to store information about the files people are uploading. For this case, I'm going to use RethinkDB and the rethink resource type for thinky

npm install tastypie@^0.5.0 thinky --save

Let's say we want to store videos, which of course can be rather large. Even though rethink can store binary data, we don't want to but all of that data in the DB. We just want to store some kind of file path or uri and use a dedicated server to serve files. A simple video model might look like this:

// models/video.js
var rethink = require('thinky')({db:'vlogger'});
var type = rethink.type;
var Video;

module.exports = Video = rethink.createModel('vlogger_video',{
   name: type.string().required()
  ,video:type.string()
  ,created:type.date().default(function(){ return new Date() })
  ,views: type.number().min(0).default(0)
  ,file_size: type.number().min(0).default(0)
  ,file_type: type.string()
  ,active: type.boolean().default( true )
  ,screenshot:type.string()
  ,description: type.string()
});

Define A Resource

The Rethink Resource Does most of the heavy lifting. All we need to do is give it a query to work with, and specify the data points we want to expose.

// resources/video.js
var Resource = require('tastypie/lib/resource/rethink');
var Video = require('../models/video');

module.exports = Resource.extend({

  options:{
    queryset: Video.filter({active:true})
  }

  ,fields:{
     created:{ type:'date', readonly: true }
    ,name: { type:'char', required: true }
    ,description:{ type:'char', required: true }
    ,video:{ type:'char' }
  }
});

Add Custom Route

There isn't to much to making a resource deal with uploads. It is really a matter of telling Hapi, that we are expecting multi part data. You can either add a new route to do this or override the list endpoint to do it. If you are a URL snob like me, then you will probably want to use POST /<resource> rather than POST /<resource>/<upload>.

module.exports = Resource.extend({
  /* fields */ 

  , base_urls: function base_urls( ){
     return [
       // The other default routes ... 
       , {
            path:'/api/v1/video'
          , handler: this.dispatch_list.bind( this )
          , name: 'list'
          , config:{
              payload:{
                 output:'stream',
                 maxBytes: 3 * Math.pow( 1024, 2 ) 
              }
          }
      }];
  }

});

Custom File Field

We need a field that can deal with a data stream. Basically what we want it to do is shuttle the data to a known place, and return a uri/path to that location. The hydratemethod on fields handles the incoming field data. This is the place to do that. Here is an abbreviated version:

var FileField = new tastypie.Class({

  inherits: tastypie.fields.ApiField

  // convert is the last step in dehydration before data
  // is returned to the client. i.e., don't return
  // sensitive information
  , convert: function( value ){
     return "<RELETIVE/PATH/TO/FILE"> 
  }

  , hydrate: function( bundle ){
      // call parent to get the actual field data
      var value = this.parent('hydrate', bundle )
      var fpath = path.join(
                      '/tmp'
                    , 'uploads'
                    ,  value.hapi.filename
                  );

      var out = fs.createWriteStream( fpath );
      value.pipe( out );
      return fpath;
  }
});

Hydrate

The hydrate function handles incoming field data

All the hyrate function needs to do here is get a handle on the field value, which will be a stream, pipe it to a known location, and return that location. Now that we have a field to handle the incoming binary stream, all we need to do is replace the default field on the video property with our new field

// resources/video.js
module.exports = Resource.extend({
  fields:{
     created:{ type:'date', readonly: true }
    ,name: { type:'char', required: true }
    ,description:{ type:'char', required: true }
    ,video: new FileField({ }) // <-- Our new field
  }
});

That is pretty much it, you can hit your endpoint with a video file, in addition to the other data, and it will all be saved. You could use curl, or an html form or anything else that can send multipart form data. You could even use the new FormData and FileList to upload files straight from a browser using plain old Ajax!

I Have a more robust and working version of this upload API set up on bitbucket. view project