Using giter8 to build Scala REST projects with Unfiltered, Netty and Gatling load testing

At Mind Candy we have a number of different Scala REST services to provide common aspects to our games, for example authentication and moderation.

A standard we have developed is to use an Unfiltered Netty standalone server wrapped with the apache-commons daemon library. Also, we want our services to be well tested, easily deployable, and stable under heavy load. For ease of deployment we use Fabric scripts, and for load testing we use Gatling to hammer the services under different stress-tests to check they are up to standard.

We have quite a few services that have a similar setup, and it was getting tedious to create and configure a new project. So, we created a giter8 script to do it all for us!

Giter8

Giter8 is an awesome command line tool which can create a template project structure for you with example files and sbt configuration.

There are quite a few templates contributed by the community already but none did exactly what we want, so we created our own which we’d like to share.

The template will create an example sbt project that:

  • Uses unfiltered and netty to give you a very simple starting endpoint service which will output ‘Hello, <project>’ when hit
  • Gives you some basic test stubs using Specs2
  • Is setup with gatling load testing with an example scenario
  • Is configured with the sbt-idea plugin (IntelliJ IDEA is our preferred IDE)
  • Is configured with the sbt-assemblysbt-release, and sbt-dependency-graph plugins
  • Has a simple fabric deploy file, unix start/stop scripts, and an example init script

The generated project is structured with 3 sub projects:

  • project-example
    • example-core // Business logic
    • example-resources // Containing the Unfiltered Netty request plan
    • example-server // To control the server hosting the plan

Usage:

First make sure you have setup giter8 using the instructions here. Then fire open a terminal and run:

g8 mindcandy/unfiltered-rest-gatling

There are some options to enter, but most can be left as default (See here for more info). The important ones are:

name => The main projects name
project => The name prepended to each of the sub projects
organization => Used as the basis for packages

Once you’ve entered the properties, giter8 will do its magic and whip you up a full sbt project. After it completes, you can cd to the directory, and run sbt.

Test it:

sbt test

Run it:

sbt run

Now open up your favourite browser and go to http://localhost:8889/<project> where <project> is whatever you specified above, and you should get a “hello, project” message back.

Load test it:

Start the server running in one terminal. In another terminal do:

cd yourproject
sbt
project <project>-server
test-load:run

You should get presented with a choice of different classes to run:

  • Engine → Executes the gatling Engine in interactive mode prompting you which scenario to run
  • BatchEngine → Runs all simulations available in a batch with no user prompts
  • Recorder → Starts the pretty cool Gatling recorder which you can use to create scenarios. Basically, you setup your web browser to use the recorder’s proxy, and then just browse as normal on your webpage. Gatling records all of your actions as a scenario that can be replayed and customised.
Hit the option for BatchEngine, then load the results in your browser to see something like:

Results from Gatling are put in the gatling/results directory. In the example scenario in the giter8 skeleton we simulate 10 users hitting the simple endpoint, ramping up over a 3s duration. The results are rendered in a nice html page with graphs showing exactly what happened during the simulation. The full gatling feature set is quite extensive and worth checking out.

Create IntelliJ IDEA project files:

sbt gen-idea

Run from start script:

First setup jsvc. Then :

sbt assembly
cd bin
chmod +x start.sh
./start.sh

This has the same affect as ‘sbt run’, but using the apache-commons daemon wrapper. Try opening the service in your browser again and it should work as before. Don’t forget to stop the server when you’re done with the ../stop.sh script :)

Other pieces:

There is also a sample init script that is configured for the skeleton in the bin folder, which is a good starting point for creating a complete init script.

Last to mention is there is a small fabric file that provides a (very) basic setup for copying over the built assembly onto a test/integration server. You’d need to configure your host and change the paths as appropriate to use it, but it’s useful as a starting point.

Let us know if you find this useful, and thanks to n8han for providing a great tool!

Building testing tools against REST APIs with node.js and Coffeescript

During the dev cycle here at Mind Candy it is useful to have tools to help automate some of the more repetitive tasks, for example, registering a user. The great thing about test tools is that they are a great excuse to try out new technologies too! In this blog post, I’ll be telling you about a tool we wrote to register new users. It uses node.js, the non-blocking I/O server side JavaScript platform that is powered by Google’s V8 VM.

The reasons why I chose node.js for this project are as follows:

  1. It makes working with I/O operations an absolute breeze. Every I/O operation is required to be non-blocking, so its perfectly valid to keep a request open from a client such as a web browser while you make a bunch of (sometimes concurrent) calls off to REST APIs, databases, external systems etc, wait for them to call back to your code and then return a response. All of this happens without occupying a thread per request, unlike your standard java servlet. In fact, node.js is single threaded and uses an event loop, so as long as your expensive I/O operations are non blocking, the event loop just keeps ticking over and your system stays responsive.
  2. While node.js is a comparatively new technology, it has a HUGE and vibrant community behind it. There are so many 3rd party modules that if you need to interface with any other thing, there is most likely a module already written! Node also has an awesome package manager in npm, which makes declaring and downloading modules super easy.
  3. The holy nirvana – the same language running on the server and the client. Since the app is going to be deployed for the web, and therefore going to be powered by javascript, there are no problems with things such as serialising and deserialising objects between client and server (they all natively speak JSON), or different paradigms between languages giving you an impedance mismatch.

The whole project is actually written in a language called Coffescript. For those who haven’t heard of it, Coffeescript is a language that compiles down to javascript. It abstracts away some of the more grizzly parts of javascript, has a nice, clear and concise syntax and has a bunch of useful syntax features built in, such as loop comprehensions, conditional and multiple assignment and a simple class system. It’s like a bunch of best practices for javascript!

So let’s have a look at some of the code. For example, here is how we talk to the Adoption endpoint:

request = require('request')
xmlbuilder = require('xmlbuilder')

class Adoption
    constructor: (@host) ->

    start: (username, password, email, cb) ->
        adoption = xmlbuilder.create()

        adoption.begin('adoption')
            .ele('email').txt(email).up()
            .ele('password').txt(password).up()
            .ele('username').txt(username).up()

        adoptionXml = adoption.toString({pretty: true})

        request({
            method: 'POST',
            uri: "http://#{@host}/my/rest/service",
            body: adoptionXml,
            headers: {
                'Content-type': 'application/xml'
            }
        }, (err, resp, body) ->
            cb(err) if err

            if(body.search(/<error name="username" rule="is_not_unique"\/>/) > 0)
                usernameError = {
                    message: 'username is already taken'
                }

            cb(usernameError)
        )

module.exports = Adoption

First thing to note is indenting and whitespace is important! You MUST use spaces instead of tabs here, otherwise the compiler complains! Here we’re creating a ‘class’ called Adoption. Javascript doesn’t really have classes, but Coffeescript translates that into some javascript that gives you class-like behaviour. At line 5, we declare the class constructor function. In Coffeescript, functions are a very small declaration: just the function arguments in brackets and then an arrow. Anything after the arrow is the function body. The constructor is very simple, all it’s doing is setting the arguments of the function as member variables of that object. Looking at the javascript generated from the Coffeescript compiler illustrates this:

function Adoption(host) {
    this.host = host;
}

The start function (line 7) takes a bunch of parameters of data from the user and a callback function as the last argument. In node.js, if we are doing any async operation such as calling a REST endpoint, we cannot return the response data from that function since that would block the event loop. Instead, we are provided with a callback function which we can then call with the response once the server responds.

On line 10 we build up the xml payload to the Moshi Monsters REST endpoint, using a module called xmlbuilder. It would be a lot simpler if the endpoint accepted JSON! Next (line 17) we send the request to the endpoint itself. Here, we use the excellent request module. If you are familiar with how you perform ajax requests with jQuery, this should look quite familiar to you! Its another example of how node.js makes full stack development a lot less trouble for your developers as so many of the techniques used on the client side can be applied to your server side code.

The request function takes an object with the options for that request, and a callback that gets called upon error or success. The convention in node.js is to always expect the first argument to your callback function as a possible error, since if the async operation fails, you can check that parameter for the exception. On line 25 there is an example of Coffeescript’s postfix form of the if operator.

We then check for some response xml with a regular expression (line 27) and call the callback with the possible error object if the regex matches. Notice we do not have to declare variables before we use them. If we did this in raw javascript, they would end up becoming global variables, but Coffeescript handily declares them up front for us.
The last line is how we expose our class to the rest of the program. Node.js uses the CommonJS module system, so every file loaded is self contained as a module. We can expose our class by assigning it to the module.exports variable. This allows us to instantiate an Adoption object in another file:

Adoption = require('./path/to/adoption.coffee')

I used the brilliant express http server to serve this webapp. It has the concept of ‘middleware’ – effectively a bunch of functions that every request and response pass through. This means it is super easy to add functionality to express like caching, static file serving, and even asset pipelining as we will see later! We can set up a handler for our adoption request like so:

express = require('express')
Adoption = require('./adoption') #note that the file extension is optional!
app = express.createServer()

app.set('view engine', 'jade')

app.use(express.bodyParser())

adoption = new Adoption('www.moshimonsters.com')

app.post('/adopt', (req, res) ->
    username = req.body.username
    password = req.body.password
    email = req.body.email
    adoption.start(username, password, email, (err) ->
        if(err)
            res.json(err.message, 500)
        else
            res.send()
   )
)

app.listen(3000)

We’re using a template language called jade for the client side HTML markup. Jade is a simple, lightweight alternative to html. Here’s an example straight from their website:

doctype 5
html(lang="en")
  head
    title= pageTitle
    script(type='text/javascript')
      if (foo) {
         bar()
      }
  body
    h1 Jade - node template engine
    #container
      if youAreUsingJade
        p You are amazing
      else
        p Get on it!

Jade is nice and lightweight, but you can also do more hardcore stuff like scripting in the templates if you wish. It supports layouts and partials and all kinds of other nice stuff. Check out the website to learn more about it!

The client side javascript for this tool is written in Coffeescript too! But how does the browser understand it? The answer is that it doesn’t – we have to compile it first into javascript. You could do this as part of the build, but we have a better solution available to us.

There is a middleware module for express called connect-assets. This middleware adds asset pipelining to connect, so that you can write your code in Coffeescript and it will compile it on the fly and serve it to the browser, without you having to do anything! It can even minify the resulting javascript. You add it like this:

connectAssets = require('connect-assets')

...

# set build to true to minify the js
app.use(connectAssets({build: false}))

…and then we add a macro into our jade template:

doctype 5
html(lang="en")
  head
    // add macro in the head of your html document
    != js('adoption')

    // rest of your markup below

…passing in the name of your Coffeescript file (minus the .coffee) extension.

Obviously this is not the whole of the source code of the tool, but hopefully it has been a taster of how awesome modern javascript development can be! In the last few years, javascript has gone from being an unliked toy language into something a lot more powerful and expressive. Here at Mind Candy we hope to leverage amazing new tools like node.js and coffeescript in our future work to allow us to become a more happy and productive development team!