A Puppet module for Dynamic DynamoDB

As my colleagues have said in other posts, we make an extensive use of Amazon Web Services at Mind Candy. Recently we decided to use the AWS NoSQL offering DynamoDB for a specific use case in one of our products.

Whilst DynamoDB provides us with a highly distributed NoSQL solution, it works based on telling Amazon what read and write capacity you require via their API. If you find that you go over either of these value you begin to, potentially at least, lose queries if you have not factored in some sort of caching layer using, for example, Amazon SQS.

In the ideal world, Amazon would offer auto scaling features for DynamoDB, however at time of writing they don’t. Instead they advise people to use an independently developed tool called Dynamic DynamoDB written by Sebastian Dahlgren.

Dynamic DynamoDB is a tool written in Python that allows us to effectively auto scale our provisioned reads and writes. It use CloudWatch metrics to establish current usage and then based on the configuration option either scales up or down your provisioned capacity on a per table basis.

As I’ve posted before here, we use Puppet at Mind Candy, so the first point of call whenever a new tool comes along is to see if anyone has written, or started to write, a Puppet module for it. Sadly it didn’t look like anyone had, so we quickly wrote up our own, which is available on Github here.

Event Processing at Mind Candy

At Mind Candy we want to build great games that are fun and that captivate our audience. We gather a great deal of data from all of our products and analyse it to determine how our players interact with our games, and to find out how we can improve. The vast majority of this data consists of ‘events’; a blob of json that is fired by the client or server in response to an interesting action happening in the game.

This blog post is about the approach that we have taken at Mind Candy to gather and process these events, and scale the systems into the cloud using fluentd, akka, SQS, Redshift and other AWS Web Services.

What is an event?

From our point of view, an event is any arbitrary valid json that is fired (sent) to our Eventing service via a simple REST api.

When an event is received, it is enriched with some additional fields which includes a ‘fired_ts’ of when the event was received, a unique uuid and, importantly, the game name, version, and event name taken from the endpoint. These three together form what we call the ‘event key’.

This service is extremely lean, and does not itself expect or enforce a rigid taxonomy. It simply writes the enriched events to disk. As a result, the service is incredibly easy to scale and to achieve high availability.

Validation and processing

We then use fluentd, an open source data collector and aggregator, to take the enriched data written to disk and place it onto an SQS queue. Currently, we use a single queue (per environment) which receives data from many different eventing servers.

Once all that data is arriving on the queue, we need to do something useful with it! This is where our home grown event processing engine, Whirlpool, comes into play.

Whirlpool is a scala and akka based service which retrieves messages from SQS and processes and validates them accordingly. It uses a variant of the akka work-pull pattern with dedicated workers for pre-fetching, processing, and writing events, communicating with a master worker. The number of workers and other parameters can be tweaked for maximum throughput.

Where does the metadata for processing come from? We have a shared ‘data model’ which contains information on what an event should look like for a specific game and version. This is essentially a scala library that reads from a backing Postgres store.

The structure of that schema is (simplified):

Screen Shot 2014-07-25 at 15.49.50

An event field is a single field to be found in the json of the sent event. It has a number of different properties, for example whether it is mandatory or not, and whether it should be expanded (exploded out into multiple events), and the json path to where that field should be expected. The point of the eventversion table is to provide a history, so that all changes to all events are recorded over time so we have a rollback, as well as an audit trail for free.

An event destination configures where an event should end up in our warehouse. It can be copied to any number of schemas and tables as we require.

Whirlpool retrieves the metadata for an event based on the extracted event key. It then passes the event through a series of validation steps. If it fails at any level, the reason why is recorded. If it completes all validations, the event can be processed as expected.

The processMessage function looks like this:

Screen Shot 2014-07-25 at 16.49.28

We use Argonaut as our JSON processing library. It is a fully functional library written in Scala that is very nice to work with, as well as having the added benefit that our resident Mind Candy, Sean, is a contributor!

After our events have been validated, they are either a successful event for a particular game and version, or a failure. At this point we make use of fluentd again with a modified version of the Redshift plugin to load them into our Redshift data warehouse. Here they are available for querying by our data scientists and data analysts. Typically, the period from an event being received to being queryable within the data warehouse is measured in seconds, and in any case within a couple of minutes in normal cases.

Configuring events

To actually setup the metadata for what constitutes an event, we have created a simple GUI that can be accessed by all game teams. Any changes are picked up within a few minutes by Whirlpool, and those events will start to flow through our pipeline.

We also needed to solve one large problem with the configuration, namely: “How do you avoid having to create a mapping for every single game version when the events haven’t changed, and how do you accommodate for changes when they do occur?”

It took us a while to find a nice balance for solving this, but what we have now is a mapping from any POSIX regex which is matched against an incoming game version, to a specific version that should be used for retrieving the metadata (this is the purpose of the ‘configmapping’ table in the schema). So, when we release 1.0 of our game, we can create metadata that applies to “1.x”. If in version 1.5 we introduce a new event, we can create a new config at that point to apply to all later versions, while still having versions 1.0-1.4 processed correctly.

Handling Failure

Events can fail for a large variety of reasons. Currently there are 17 specific types of these, with a couple being:

  • The event is malformed; it does not contain the fields that we expect
  • The event is unknown

A failure is captured by the following class:

Screen Shot 2014-07-25 at 16.49.46

The FailureType here is another case class corresponding to the specific failure that was generated, and the fields contain some additional attributes which may or may not be extracted from the failure.

We treat failures separately from processed events, but they still make their way into Redshift in a separate schema. Each failure contains enough information to identity the problem with the event, which can then be fixed in most cases in the metadata; typically, event failures occur during development, and are a rare occurrence in production.

Scaling our infrastructure

We make heavy use of AWS at Mind Candy, and the eventing pipeline is no exception. All the eventing servers are described via Cloud Formation, and setup in an autoscale group fronted by an ELB. As a result, the number of servers deployed scales up and down in response to rising and waning demand.

The use of SQS also separates out our event gathering and event processing infrastructure. This means that Whirlpool instances do not have to scale as aggressively, as the queue provides a natural buffer to iron out fluctuations in the event stream due to peaks of traffic. For Redshift, we have a 6XL node cluster which we can scale up when required, thanks to the awesome features provided by Amazon.


We’ve benchmarked each of our eventing servers comfortably processing 5k events/sec, on m1.medium instances.

Whirlpool does a little more work, but we are currently running a configuration offering a sustained rate of just over 3k events/sec per instance, on c1.medium instances, with a quick ramp up time.

Instances of both Eventing and Whirlpool operate independently, so we scale horizontally as required.

Screen Shot 2014-07-25 at 16.24.54

The Future

We have real-time dashboards that run aggregations against our event data and display it on screens around the office. It’s very useful, but is only the first incarnation. Currently we’re working on streaming processed events from Whirlpool into Spark via Kafka, to complete our lambda architecture and greatly reduce the load on our Redshift cluster. We’re also improving the structure of how we store events in Redshift, based on our learnings over the last year or so! At some point when we have more time, we would also like to open-source Whirlpool into the community.


Using giter8 to build Scala REST projects with Unfiltered, Netty and Gatling load testing

At Mind Candy we have a number of different Scala REST services to provide common aspects to our games, for example authentication and moderation.

A standard we have developed is to use an Unfiltered Netty standalone server wrapped with the apache-commons daemon library. Also, we want our services to be well tested, easily deployable, and stable under heavy load. For ease of deployment we use Fabric scripts, and for load testing we use Gatling to hammer the services under different stress-tests to check they are up to standard.

We have quite a few services that have a similar setup, and it was getting tedious to create and configure a new project. So, we created a giter8 script to do it all for us!


Giter8 is an awesome command line tool which can create a template project structure for you with example files and sbt configuration.

There are quite a few templates contributed by the community already but none did exactly what we want, so we created our own which we’d like to share.

The template will create an example sbt project that:

  • Uses unfiltered and netty to give you a very simple starting endpoint service which will output ‘Hello, <project>’ when hit
  • Gives you some basic test stubs using Specs2
  • Is setup with gatling load testing with an example scenario
  • Is configured with the sbt-idea plugin (IntelliJ IDEA is our preferred IDE)
  • Is configured with the sbt-assemblysbt-release, and sbt-dependency-graph plugins
  • Has a simple fabric deploy file, unix start/stop scripts, and an example init script

The generated project is structured with 3 sub projects:

  • project-example
    • example-core // Business logic
    • example-resources // Containing the Unfiltered Netty request plan
    • example-server // To control the server hosting the plan


First make sure you have setup giter8 using the instructions here. Then fire open a terminal and run:

g8 mindcandy/unfiltered-rest-gatling

There are some options to enter, but most can be left as default (See here for more info). The important ones are:

name => The main projects name
project => The name prepended to each of the sub projects
organization => Used as the basis for packages

Once you’ve entered the properties, giter8 will do its magic and whip you up a full sbt project. After it completes, you can cd to the directory, and run sbt.

Test it:

sbt test

Run it:

sbt run

Now open up your favourite browser and go to http://localhost:8889/<project> where <project> is whatever you specified above, and you should get a “hello, project” message back.

Load test it:

Start the server running in one terminal. In another terminal do:

cd yourproject
project <project>-server

You should get presented with a choice of different classes to run:

  • Engine → Executes the gatling Engine in interactive mode prompting you which scenario to run
  • BatchEngine → Runs all simulations available in a batch with no user prompts
  • Recorder → Starts the pretty cool Gatling recorder which you can use to create scenarios. Basically, you setup your web browser to use the recorder’s proxy, and then just browse as normal on your webpage. Gatling records all of your actions as a scenario that can be replayed and customised.
Hit the option for BatchEngine, then load the results in your browser to see something like:

Results from Gatling are put in the gatling/results directory. In the example scenario in the giter8 skeleton we simulate 10 users hitting the simple endpoint, ramping up over a 3s duration. The results are rendered in a nice html page with graphs showing exactly what happened during the simulation. The full gatling feature set is quite extensive and worth checking out.

Create IntelliJ IDEA project files:

sbt gen-idea

Run from start script:

First setup jsvc. Then :

sbt assembly
cd bin
chmod +x start.sh

This has the same affect as ‘sbt run’, but using the apache-commons daemon wrapper. Try opening the service in your browser again and it should work as before. Don’t forget to stop the server when you’re done with the ../stop.sh script :)

Other pieces:

There is also a sample init script that is configured for the skeleton in the bin folder, which is a good starting point for creating a complete init script.

Last to mention is there is a small fabric file that provides a (very) basic setup for copying over the built assembly onto a test/integration server. You’d need to configure your host and change the paths as appropriate to use it, but it’s useful as a starting point.

Let us know if you find this useful, and thanks to n8han for providing a great tool!

Migrating a Play 1.2 website to Play 2.0

At Mind Candy there are a number of internal websites used for reporting and communication. For example – reading automated build/test status via some REST APIs and turning this into a nice visual status display for large ‘build screens’. These have been authored in Scala using the Play Framework 1.2.x.

Recently, version 2.0 of the Play Framework came out of beta and so I wanted to convert the existing Play 1.2 websites over to it. Amongst other reasons, the ability to build to a .jar file makes for simpler deployment, which I was keen to have.

I couldn’t find a good guide for migrating from 1.x -> 2.0 so I am sharing my experiences of porting a Scala Play 1.2.x application. I expected that it wouldn’t be too hard as I was already using Scala Templates and controllers. That was mostly true, with a couple of issues.

I did find a handy conversion guide on the Play 2 wiki which can help converting some uses (it has some unfortunate formatting though so I’d recommend copying and pasting it)

Starting the project

I’d previously installed Play via brew so an upgrade was as simple as
brew update
brew upgrade play

This installed Play 2 whilst leaving Play 1.2.4 also installed for easy roll-back for developing older stuff if I need it.

For simplicity and because there were various code changes that I knew I needed to make, I created a new site in a new location using play new. However I then used svn copy to copy over source assets and code before modifying. This means in the future I can pull updates down using “svn merge” as work continued on the old site whilst I was porting it over!

All of the public files which aren’t built – javascript, css, images – were simply copied across. In the future I’ll check Play’s support for LESS CSS and the Google Closure JS compiler but for now I just want to get things working.

Initially I copied across all the controllers, models and views and configuration files, though I expected to have to fix those up as the syntax had changed.

After a quick run of play compile I had a whole host of build errors, unsurprisingly. So to cut these down I commented out a lot of the logic – all the routes except the home page /, all the templates and all the controllers except for those needed by the home page.

I fixed up controller in turn and gradually re-enabled the routes, until I hit an issue with Database models that stopped me from migrating the rest of my application (see below)

Migrating dependencies

There were a number of java libraries used by the web app, for example to talk to a Subversion server. (This is one of the reasons I really like Scala – it’s easy to pull in useful Java libraries) In Play 1 that was defined in dependencies.yml, e.g.:

- org.tmatesoft.svnkit -> svnkit 1.3.7
- org.tmatesoft.svnkit -> svnkit-javahl16 1.3.7

- tmatesoft:
  type: iBiblio
  root: "http://maven.studio:8081/nexus/content/repositories/tmatesoft"
  - org.tmatesoft.svnkit -> *</code>

Play 2 now uses sbt as its build system and thus uses the standard sbt syntax for dependencies, so I edited the the project/Build.scala file to add dependencies:

object ApplicationBuild extends Build {
  // some definitions omitted

  val appDependencies = Seq(
    "org.tmatesoft.svnkit" % "svnkit" % "1.7.4",
    "org.tmatesoft.svnkit" % "svnkit-javahl16" % "1.7.4",

  val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
    resolvers += "TmateSoft Repository" at "http://maven.tmatesoft.com/content/repositories/releases/",

Once the sbt syntax is understood it’s a reasonably simple process to convert dependencies across.

The original play run / test / start / stop command still work, which is helpful. It’s also quite useful to simply run play to get an sbt console and then run ~test which runs test and then re-runs them anytime a source file changes, which is very useful.

For IDE integration, I use IntelliJ IDEA and happily there is an idea command in Play’s SBT which generates an IntelliJ module which has any library dependencies included. However, your code has to build correctly before this works and you probably need to remember to re-run this every time you modify a dependency. If you don’t the IDE won’t show types correctly – typically you will get more ‘false negatives’ of errors showing up in the IDE which will not happen within Play itself.

Migrating the routes file

For some reason the old Routes file had an explicit 404 for the for the favicon. There didn’t seem to be an obvious pure route-based replacement for this and I didn’t need that behaviour so removed the route.

I had to add controllers. to all routes as the full package reference seems to be required now.
e.g. home page changed from
GET / Application.index
GET / controllers.Application.index

The syntax for ‘public’ files has changed so I replaced
GET /public/ staticDir:public
GET /public/*file controllers.Assets.at(path="/public", file)

I was also serving a single file elsewhere – multiple Assets.at() in the routes file didn’t go well, but I was able to call to add a new method to the Application e.g. ‘serveMyFile’ and then set the routing to be controller.Application.serveMyFile in routes and then have that call the Assets.at() method
GET /myfile controllers.Application.serveMyFile
in the Application controller:
def serveMyFile = controllers.Assets.at(path="/path/to/it", file="myfilename.json")

The syntax for parameters in routes has changed and I found that I needed to move the default parameter definitions from the Controller out to the Routes file, using the ?= syntax instead of the Option[T] syntax I had before. That wasn’t hard to do but was slightly irritating in some ways – I’m still not sure if I like having so it in the routes file.

Minor changes to the syntax for parsing out ids from the URL, for example:
GET /versionone/stories/{projectId} versionone.Printer.storiesById
changed to
GET /versionone/stories/:projectId controllers.versionone.Printer.storiesById(projectId: String, timeboxId: String ?= "", template: String ?= "")

Note the additional default parameters being passed. Also in this instance the Id is actually a string, not an Int.

Migrating Controllers

A lot of the package names have changed e.g. play -> play.api so some of the imports needed to be fixed up with the new ‘api’

import play._
import play.mvc._

import play.api._
import play.api.mvc._

With the paths to generated views also changed, I removed lines like import views.Application._

Then with imports fixed up I could then fix up the actual controller code to return an Action and
adding Ok(views.html. … )
e.g. from
def index = {

def index = Action {

As I mentioned earlier, some changes to controllers were needed because the routes file now supplies default parameters, and I took out some use of the Option[T] type — though as this was generally a String, the empty string works very well as an equivalent to None.

In some controllers I’ve implemented an OAuth callback to read from a Google Calendar – eventually I’d like to play with the built-in OAuth support in Play 2 but in the interest of getting things working quickly I wanted to port over my current code. In order to send a url for Google auth to use as a callback I am calling Router.getFullUrl() but this doesn’t work in Play 2, instead one calls the automatically generated route classes instead. To turn this into a full url you call .absoluteUrl(), but you must remember to add the implicit request to your controller code in order to give this the context needed to generate the url.
e.g. the original code was approximately:

object MyController extends Controller {
  def request = {
    val callbackUrl = Router.getFullUrl(“MyController.callback”)
    // send web request using callbackUrl

  def callback = {
    // handle request

the new Play 2 code is:

object MyController extends Controller {
  def request = Action { implicit request =>
    val callbackUrl = routes.MyController.callback().absoluteURL()
    // send web request using callbackUrl

  def callback = Action {
    // handle request

This is more robust because that reverse-lookup is now checked at compile time, rather than runtime. However if your code doesn’t build, your IDE will give you errors as the reverse routes don’t exist. I did occasionally have to do a clean build to force regeneration of reverse-lookups, if the code structure had changed significantly.

Migrating Views

I had to replace @asset() with @routes.Assets.at()

Old references to actions can now be replaced with the compile-time checked generated routing objects. But the snag that isn’t documented is that if you are using a sub-project, the routes are generated within the scope of that sub-project.

e.g. replace @action(svn.Sizes.index()) with @controllers.svn.routes.Sizes.index()
This was mentioned on the Play discussion group here.

Migrating Tests

Unfortunately the bundled Play test framework has changed from scalatest to specs2. I could possibly have set up scalatest to work but decided to migrate the tests instead as the new functional tests look useful, as is the ‘property’ support in specs2.

A few imports change – I had to get rid of the scalatest imports and add in:

import org.specs2.mutable._
import play.api.test._
import play.api.test.Helpers._

The test class now simply extends Specification and I had to fiddle around with the syntax, replacing it with a noun and adding some more braces. should be(x) was replaced with mustEqual x

For example:

it should "retrieve svn server prefix" in {
  val svnServers = SvnServer.find().list()
  svnServers.length should be > (0)


"SvnServer" should {
  "retrieve svn server prefix" in {
    val svnServers = SvnServer.find().list()
    svnServers.length must be_>(0)

There is a good list of all the matchers in specs2 at the specs2 site which I found very useful.

I migrated some very basic selenium tests to the new functional test framework – documented here. The advantage of this is that you can separately test the controller, view and the whole thing together with the router.

The old selenium test was

#{selenium 'Home page'}
// Open the home page, and check that no error occured
assertNotTitle('Application error')

So a simple test that the home page works is:

class ApplicationTest extends Specification {

  "The Application" should {
    "render template correctly " in {

      val html = views.html.Application.index()

      contentType(html) must equalTo("text/html")

      val content = contentAsString(html)
      content must contain ("Welcome to the Tools website")
      content must not contain ("Application error")

    "have a working controller" in {
      val result = controllers.Application.index()(FakeRequest())

      status(result) must equalTo(OK)
      contentType(result) must beSome("text/html")
      charset(result) must beSome("utf-8")

      val content = contentAsString(result)
      content must contain ("Welcome to the Tools website")
      content must not contain ("Application error")

    "run in a server" in {
      running(TestServer(3333)) {

        await(WS.url("http://localhost:3333").get()).status must equalTo(OK)


Migrating the Database Models – problems!

Now for the bad news… our database models were making heavy use of the Magic[T] class and that simply isn’t present in Play 2.0 at the moment (Apparently it featured in a beta version but was removed). What Magic[T] did was generate a lot of the code for the CRUD operations from a simple case class. For reference in the old documentation for the play-scala-0.9.1, this is what Magic did.

Whilst I could write all of this code myself, I have about 40 classes to do this for, so this is a non-trivial amount of work.

I also discovered that the DB evolution hashes are different, so play wanted to revert and re-apply all of my DB schema changes, which would have trashed the DB contents. Apparently this is now fixed in the trunk in Play 2 so hopefully that change will be in a release soon.

Because I don’t have the time to rewrite all the DB classes, I decided to split my application into two parts, as there are a number of very independent reports in the site which do not use the DB. The reports will be in a play 2 site and the older code that uses the DB will stay as a Play 1.2 site.

In the future I would like to port everything to Play 2. I’m hoping that someone will reintroduce Magic[T], either into the core of Play 2 or as an additional bit of code / plugin. Alternatively I’ll have to do all the grunt work myself, which would be a shame. It’s the kind of work I like to avoid simply because there is very little value I can add – it will either keep working or (more likely) I will break something! So the best option is to leave this old ‘legacy’ site as play 1.2 until there is an easier/safer upgrade path.

However, I will definitely be using Play 2.0 for new development!

Google and HTML5 – Simplicity, Security and Speed


HTML5 has been around for a while, but only since recent times however, has it emerged as the major web technology of choice moving forward. Browsers are catching up, increasingly adding HTML5 feature support and there is an emergence of tools offering near-comprehensive compatibility with older browsers and platforms. Not to mention the massive support and investment from tech powerhouses such as Google and Apple, HTML5 really is fast becoming the defacto standard for building powerful, cross-compatible web apps.

This new age of HTML5 powered web apps means we can recreate a seamless native app experience from within the browser. However this time, there’s no flash, no plugins, nor any downloads. Instead, with the help of new HTML5 features such as drag and drop, local storage, multimedia (canvas, svg, video and audio), geolocation, offline and real time communication (webRTC), along with CSS3 and various Javascript API’s, we now have one technology stack that is usable across multiple platforms and devices.


Google have pioneered in this area for a while, having led the way with a plethora of powerful, game-changing, web apps such as Docs and Sites to name but a few. They recently invited a group of geeks/web developers to their London offices in Victoria for a talk about HTML5 and some of the things they’ve been working on. I was part of this group.

The offices are pretty cool. From the famous Google ‘don’t be evil’ posters to the life-size London red bus to the fully catered kitchen and astro-turfed meeting room. It all supported the fun, creative and inspiring vibe floating around the place and this is what Google is about from a cultural point of view. What Google are all about technologically, can be loosely categorised into 3 areas: simplicity, security and speed. These are the primary focuses for the Chrome OS and also for their user-centric designs.


New breed of Web apps: One technology stack that is interoperable between multiple devices and platforms without the need for downloads or cross platform and OS adaptations. Add powerful features such as offline use, canvas and touch event API’s, and the categorisation between a website and web app becomes more converged. FT and Kindle are good examples of this new breed of HTML5 web app.

Chromeframe:  There are a lot of cool new HTML5 features which aren’t compatible with older browsers such as Internet Explorer 6 & 7 and this can be costly to support for developers. One option to reduce this headache is to prompt users to install Google Chromeframe which enables Chrome to work inside Internet Explorer. Instant compatibility.

WebIntents: Web Intents make it easier to share data and web services across apps. E.g. ‘share a link with Twitter’ or ‘edit a photo with image editor before sending’. Instead of integrating each and every one of these web services with your app, the user is empowered with their own choice. The idea is that your service registers an intent to handle an action such as sharing or editing, and the system will learn to find the appropriate services for the user to use based on the user’s preferences. Less integrating, more sharing.


Content Security Policy (CSP): CSP tackles malicious cross site scripting by allowing web pages to limit where external content can be loaded from. Additionally, inline javascript and CSS are not allowed. As well as providing a good security measure, this also helps maintain good semantics.

3Dtin: This is an entirely web based 3D modelling app where anyone can jump in and start creating 3D models. If extra functionality is required, it offers an openID signup using Google Identity Toolkit which adds a layer of security onto the login part of the process. This ‘try now, sign in after’ approach enhances the user experience by allowing users to first engage, then commit after.

Browsers: A good browser makes all the difference. Chrome pushes updates automatically which ensures that the user always has the latest version with all the latest security patches. It also offers comprehensive sandboxing and plugin security for extra layers of protection.


Offline: HTML5 provides a better way of implementing offline availability and local storage. Previously this was done using cookies and offline emulation (gears). Now, it can be done using application caching and HTML5 storage. Application Cache is like a beefed up implementation of the browser cache, storing references to all required resources in a cache manifest file. HTML5 storage includes API’s such as Web Storage and IndexedDB, which store user state (such as state of play in a game) on the user’s local database for later use.

Frameworks/Templates: The idea of using frameworks and templates isn’t new, nor are they a requisite for building HTML5 web apps, but the ‘do more with less code’ benefits are certainly worth a mention. For Javascript, popular MVC frameworks include Backbone.js and Javascript MVC. SASS and LESS are some of the more well known CSS frameworks. Underscore and Mustache are great for templating. If you want to start building an HTML5 website from a template, then HTML5 Boilerplate is the daddy.

Bleeding Edge Technologies: This is what the Google developers refer to when talking about work on more-cutting-than-cutting-edge technologies (i.e. not well supported!) These include:

  • pageVisibility API: This determines if your web app is actively in view of the user. If not (i.e. user is on another tab), then the non-active pages are paused, hence optimising performance.
  • prerendering API: This predicts where the user will navigate next and pre-loads the page so that it appears to load instantly when it is clicked.
  • Fullscreen API: Fullscreen without the need for flash. All individual DOM elements are styleable which leads to lots more control and customisation.
  • Gamepad API: With powerful multimedia capabilities such as canvas and SVG gaining momentum, games running in the browser are only going to get more sophisticated. This API maps gamepad button events and empowers the user with a more traditional gaming experience.
  • webRTC: This is real time communication through browser alone i.e. plugin-free. It’s based on p2p and a handshake through to the server, and uses the getUserMedia API to talk to the user’s camera/microphone. This has the possibility to create much richer web-browser social video based experiences than was possible before.
  • Speech Input API: With the ability to transcribe voice into text, this will be a game-changer… once the voice recognition technology improves. An extremely exciting prospect for the near future.
  • Video subititles: Using the <track> tag and a linked .vtt file, video subtitles can be implemented. Great for accessibility, with potential for timed metadata and deep-linking into exact points on a video. Perhaps, on-the-fly localised subtitles using multiple .vtt files and the speech input API in the near future?


Technology moves fast. In my time at Google, buggy demos were shown, Chrome extensions were written on the fly and new features were released that the top developers didn’t even know about. The motto of the day seemed to be ‘There are a few bugs, but it works’ and ‘That’s a bug, I’ll fix it’. It’s what happens when you’re at the front of a new technology. But it’s through innovation and fast-paced iterations in which cool things develop…

So it’s a wonder why Microsoft have been so slow in providing HTML5 support for Internet Explorer. With its large market share, mainstream adoption of HTML5 has been massively held back. However, this is all set to change, with the imminent arrival of IE10. Less time will be spent dealing with cross platform and device issues, messing with legacy browser woes and hacking polyfiller shims together. More time will be spent innovating, and building richer, more sophisticated web apps that run through the browser alone. Exciting times to be a web developer! Developers are just scratching the surface of what is possible and as the HTML5 standard matures, as will our web apps. And if we adhere to the 3 S’s and acknowledge that users want things simple, secure and speedy, the future of the web is already in better shape. Exciting times ahead indeed!

Many thanks to Google and the team Sam Dutton, Paul Kinlan, Ido Green and Alex Russell

Useful Links

Testing Unity’s Flash Export on a Large Project – Part 1

Over the last weekend, I’ve been hard at work trying to get an unannounced Mind Candy project (made in Unity) to export to Flash. I thought it would be useful to share some details from the experience since most of the issues I’ve encountered would probably be avoidable if your project is architected in a way that lends itself to Flash export.

During the Christmas holidays, I made a game for Unity’s Flash in a Flash contest. It wasn’t the most exciting game, but it worked. The core mechanic of that game (including new 3.5 features such as nav mesh) exported to Flash well. The reason this game worked is because I had been paying close attention to the Flash export and knew what features wouldn’t work at that time. I avoided anything overly complicated and developed the game with the limitations in mind. Fundamentally, I decided to make a new, simple game rather than trying to port an existing one.

Now, I’m doing the opposite. I’m trying to get an existing game to publish to Flash. This project is a relatively large one. The game has been in development for quite a long time. It contains a lot of complex C# code and most importantly: a lot of features that don’t yet work in the Flash export. Trying to get this game to export to Flash is no easy task. I’ve spent numerous weekends on this since the 3.5 beta was made available and I still haven’t got it to work.

Despite the export (currently) not working for our project, there are a lot of lessons to be learned. Hopefully these will be of use to other people attempting the same task, and will be a good reference point for myself when I inevitably try to export the game again at a later date.

Currently Unsupported Features

Unity have already listed some of the unsupported features in the Flash export as part of the 3.5 preview faq. Some of these features (and the ones that have proved most problematic for me) are:

  • LINQ
  • Terrains
  • Asset Bundles
  • WWW
  • Raknet networking

If you’re using these features, then you’ll encounter a lot of errors as soon as you try to get Unity to build to Flash. Some example errors I’ve seen are:


  • error CS0246: The type or namespace name `ConnectionTesterStatus’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `NetworkView’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `BitStream’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `WWW’ could not be found. Are you missing a using directive or an assembly reference?


  • error CS0246: The type or namespace name `MovieTexture’ could not be found. Are you missing a using directive or an assembly reference?

These errors are effectively a checklist of all the classes you’re using that aren’t yet supported and there’s only one thing you can do: remove them from your build. There are numerous ways to do this (depending on what you’re trying to achieve), from brute force deletion to telling Unity to skip these sections in a Flash build. You can do the latter by using platform dependant compilation.

All you need to do is wrap your Flash specific code in a platform check such as:

Debug.Log(&amp;quot;Flash build&amp;quot;);

In my case, the first thing I had to do was to try and remove these unsupported features. MovieTextures were easy to take out, as they’re not vital to our game. Networking, however was more problematic. And this is my first (and most important) lesson…

Lesson 1 – Separation of Networking Code

Our game currently uses the inbuilt RakNet networking solution. These networking elements are fundamental to our game, and as such, the networking code exists in many different areas of our codebase. When publishing to the web player or a standalone app/exe build this is fine. For Flash export, this suddenly creates a big problem when the networking solution isn’t yet supported.

As an example, if your game uses RPCs across clients to update core data in your game, then you’re going to have problems. I’m sure that there are other solutions which are better suited to Flash export, but this doesn’t fix my immediate problem: we have a game, where our chosen networking solution won’t publish to Flash. Unity suggest that you can use Flash networking instead of RakNet, but since I’m doing this export with tight time constraints (self imposed, by the mere fact it’s a weekend), that solution is not feasible for this test.

This has left me with one option in my mission to get our game working: rip out RakNet. This is not ideal, but luckily, our game copes with it ok.

This raises an interesting point in that the networking code should be as decoupled from the core mechanic of your game as possible. In some cases this can’t be done, but if you can find a way to make your networking layer easily removed/changed, then you’ll be in a much better place than I was regarding Flash export. It will also help you if you ever decide to switch to a different networking solution.

At this point, I’m going to gloss over about 10 other failed builds. It takes a few attempts at building to clear up this first wave of errors. Once you’ve cleared that first wave, you can breathe a sigh of relief and ready yourself for wave two: Attempting an actual build…

Attempting a Build

Once you’ve fixed/removed/hacked-out all the unsupported features, you’ll get to a point where the build process will now try to publish your game to Flash. The type of errors you get now will be more complex than those in wave one. Below is a screenshot of one of my build attempts at this stage:

You’ll note that these errors are more complicated than the “you can’t use ClassX because it’s unsupported” ones. In the case of these errors, it’s up to you to go into each of these classes and try to simplify your code as much as possible.

Some areas where our build failed were where we’d used generics. For example, we had fairly complex code to randomise the order of elements in an array. It wasn’t vital, so it went in the bin. This seems to be a common trend in trying to get this project to build to Flash. I’m slowly, over time, discarding features to the point where it’s a very stripped-down version of the game.

There are a couple of errors regarding our audio library in the above screenshot. This library wouldn’t convert at all (I got multiple waves of errors). My only solution at present has been to remove it.

The last item in that list is log4net. This caused a lot of issues. Rather than spending ages resolving them for this test, I decided it should also be removed. Since we used the logging in a lot of our code, I’ve ended up writing my own logging classes based on the log4net interfaces. This meant that I only had to fix up the imports in the class and our existing logging would still work using Unity’s own Debug.Log etc.

A few more iterations and build attempts occurred before wave 2 was complete. All in all, the first two waves have taken out large chunks of our features, and as a result the game feels somewhat unstable.

Akin to a game of [insert zombie survival FPS game of your choice here], we’ve just about survived the first few waves. We’re battererd, we’re bruised, but most importantly, we’re not defeated! We’re now ready for the next wave. Bring on the boss; the tank; the last major hurdle in the flash export – the conversion of your code to ActionScript.

Converting your code to ActionScript

At this stage, when you try to build, Unity will attempt to convert your source to ActionScript. Having previously spent years as a Flash developer, I find this part of the build rather exciting. The guys at Unity have done a fantastic job of getting this process to the stage it’s at.

That said, this is probably the toughest part of the process. Ripping out features and (to some extent) fixing the errors in the previous stage is easy. Trying to work out why the generated ActionScript doesn’t work is much more difficult. Luckily, when a build fails, you can find all the AS classes in a temp folder in your project (/Temp/StagingArea/Data/ConvertedDotNetCode/global/). This will enable you to look at them (if you wish) and try to understand where it might be going wrong, such that you can adjust your C# or js accordingly.

In my first attempt at this stage, I was left with 87 errors. The following are a small selection of these to give you an idea of the kind of problems I’ve seen:

Conversion Error 1

  • Error: Access of possibly undefined property $Type through a reference with static type Class.

This error seems to be very common and occurs when reflection is used (and probably in other situations). Unfortunately, a lot of our core libraries use reflection, and as such, this is a large problem to try and fix.

Conversion Error 2

  • Error: Call to a possibly undefined method IComparable$1_CompareTo_T through a reference with static type Number.

This has occurred because we’re trying to compare two values whose classes implement IComparable. In our case, this could be worked around relatively easily.

Conversion Error 3

  • Error: Type was not found or was not a compile-time constant: ReadOnlyCollection$1

In some of our classes we’re providing access to ReadOnlyCollections. It seems that we can’t use these at present and we could work round this by simply returning a standard Collection.

Conversion Error 4

  • Error: Call to a possibly undefined method String_Constructor_Char_Int32 through a reference with static type String.

A common style of conversion error that’s quite tricky to work out. I saw a lot of errors similar to this one.

These are just 4 of the 87 errors which need fixing. I expect that if/when all 87 are resolved, I’d have another wave or two to get through before the game would actually build. For now though, it’s Sunday night and I’ve run out of time to work on this test.

Next Steps…

My next challenge in this Flash export test is to go through the aforementioned 87 conversion errors and try to resolve them. I’m hoping that I’ll be able to get the game to build after another solid few days working on the export.

If that task proves too difficult then I will try a different approach of starting from a clean project and adding features one by one. In theory, that should be easier to get working, although that’s not how we’d want to export to Flash in the long run.

If I do get the export to work, I shall write a follow-up post with a walkthrough of some conversion errors. For these, I’ll include (where possible) the raw C#, the converted AS, and examples of how the errors can be avoided/solved.

For now though, I’m going to give up and play some well-earned Killing Floor! :D

Scala at Mind Candy

Reading some recent negative commentary about Scala with interest I felt like it would be good to share our experiences with Scala.

The Good.

Scala is an expressive language – It often results in a lot less code getting in the way of what you want to do or achieve. The simple example for something like this would be a simple bit of code like this:

case class Person(id: Int, name: String)
def lookupPerson(id: Int): Person = {
  new Person(id, "Test" + id)
} // Token implementation for the example.
val ids = Seq(1,10,100)
ids.filter(id => id < 50).map(lookupPerson)

To write this in Java would require a whole load of boilerplate, which includes the case class generating the stock class methods like toString, multiple lines to create a collection and then transforming a collection.

More powerful ways to write tests – This can fall under the Spiderman grouping of power admittedly, but libraries like specs2 and scalacheck make it easier to create the kind of tests we wanted. Scalacheck is the stand-out example of this where a generator for creating Person objects from above is as easy as this:

object TestGenerators {
  val personGenerator = for {
    id <- arbitrary[Int]
    name <- alphaStr
  } yield new Person(id, name)

That’s all it takes and that object can be imported into all of your test code that needs to generate Person objects.

Less magic – A lot of libraries like Spring and Hibernate need to use byte code modification or reflection to achieve what they do, which means that when they fail it can be really hard to diagnose or fix the problems. We’ve even seen some of these do things non-deterministically, which has caused hours of bemusement. Contrary to this, Scala libraries just tend to use the type system to achieve these ends which means that in our case we catch problems at compile-time, not at run-time.

The REPL – The idea scratchpad and general utility will be your friend in times of need. Use this as an education tool to step through an idea with some else. Use it to test some code interactively if you want to confirm something but you’re not quite sure how to code it or what results you’ll get. Use it to solve those gnarly Project Euler problems without having to create a whole new build for each one.

SBT – Controversial one this may be, but it manages to give you the sensible build model and plugin system that Maven has while allowing you to easily create custom tasks. If nothing else being able to run a command, for example the ‘test’ task, on each save is the most useful thing I’ve seen in a while.

POWAH! – There’s an elegance that comes with time when using Scala, in much the same way that it does with a lot of languages, that means code slots together so cleanly and with little friction. For me personally the Option class was the beginning of this change in thinking, where I realised that representing the possible lack of something without using a null made a lot more sense.

The Bad.

SBT – It’s a double edged sword in you’ll need to understand a bit of Scala to be able to do non-trivial configuration in it. Documentation for this has improved massively in recent times, it can still be somewhat impenetrable, especially to someone new to Scala.

Somewhat idiomatic libraries – Databinder Dispatch is a good example of this, writing a custom handler to parse a HTTP response is just unnecessarily puzzling. As with all libraries how easy they are to use and extend should be evaluated, so don’t be blinded by those libraries just because they’re written in Scala. It’s better to pimp a stock Java library that already works well than to use one that is badly written in Scala.

Binary compatibility – This is the stock issue that is often complained about, fortunately SBT does notice when two versions of the same library that relate to two different Scala versions are pulled into the dependencies. The way others have presented this is as a major pain point, it’s generally only so much of an issue as it is with Maven dependencies with a little more granularity. Also if you’re using SBT it’s possible to create dependencies that tie to the Scala version used automatically.

Knowledge – There’s a couple of aspects to this. The first is that Scala is a “new” language and as such there is one learning curve which relates to the language, SBT, the libraries and how to use them all effectively. Beyond this is that some functional programming concepts are foreign to a lot of programmers and this can be a wall that isn’t scalable in a short period of time for a lot of people. Hopefully with time this will become less of an issue but at the moment there aren’t a lot of Scala developers that can hit the ground running.

The Ugly?

As with all new things, there is a learning curve with Scala, which can be problematic, but the benefit of the design is that it’s possible to do something the “wrong way” as the language is very flexible. People with a history in languages like Java can start out writing code that looks not that much different but still get benefits like better collections. Then with time progress onto using more the powerful features in the language like pattern matching and implicits. For the foreseeable future Scala is a tool we intend to keep using, as it’s been of great benefit to us (this week I parsed a 37GB log file with a couple of lines of code in the REPL), maybe you should too…

99 Bottles of JMeter on the wall

I’ve recently had to do some performance testing on a couple of our new web services. I know of a few handy open source tools available for this. Tsung, Grinder and JMeter spring to mind. I find that I can get up and running in JMeter quicker than I can with the other tools and it tends to be the one I use most. One of the web services I wanted to test required some dynamic content to be generated in the body of the HTTP POST. Normally I’ve been able to get away with using the standard counters and config variables provided by JMeter, but this time I needed a bit more control. Luckily, JMeter allows you to do some scripted preprocessing of HTTP requests and I was able to use this to generate dynamic content within JMeter. I found the JMeter documentation to be a bit lacking in this area, so I created this blog post to give a quick explanation of how I managed to do this.

I’ve created a demo project that you can follow along with to see how it all works. Download the source code here: https://github.com/groodt/99bottles-jmeter Follow the README on GitHub to get everything setup and running. All you need is git, Python and JMeter. Open up the file “Test Plan.jmx” in JMeter to follow along.

The demo
The demo project is a simple web service that parses JSON payloads and prints a modified version of the “99 Bottles of beer on the wall” song onto the console. The JSON payload looks something like this:

{“drink”:”beer”, “bottles”:”99″, “date”:”1321024778956″, “thread”:”4″}

The server then parses these payloads and prints them out to the console:

JMeter then aggregates the response times in the summary report:

If you navigate to the “HTTP Request” node in the example you can see the JSON POST body being constructed:

The variables ${drink}, ${bottles}, ${date} and ${thread} are generated dynamically by a script that JMeter executes for each request.

The BSF PreProcessor
The BSF PreProcessor is run before each HTTP request to generate the dynamic content mentioned earlier. The BSF PreProcessor allows you to run Javascript, Python, Tcl and a few other languages inside JMeter. I decided to write my script in Javascript.

If you navigate to the “BSF PreProcessor” node in the example you can see the script that is used:

The Javascript
The simple Javascript basically places 4 variables in scope that are then available for JMeter.

// Calculate number
var count=vars.get("count");
var bottles=99-count;

// Calculate drink
var random=new Packages.java.util.Random();
var number = random.nextInt(4);
var drink = vars.get("drink"+number);
vars.put("drink", drink);

// Calculate date
var date=new Packages.java.util.Date().getTime();

// Calculate thread
var thread=ctx.getThreadNum();
  • In lines 1 to 4, the counter is read from JMeter then 99 is subtracted and the value is placed into scope under the name “bottles”.
  • In lines 6 to 10, a random number from 0 upto 4, it then uses this number as a lookup into the names of drinks (beer, wine, mead, cider) defined in the JMeter General Variables. It then stores this value in a variable named “drink”. It makes uses of java.util.Random to generate the random integer.
  • In lines 12 to 14, java.util.Date is used to generate a timestamp in milliseconds. This value is stored in a variable named “date”.
  • In lines 16 to 18, the JMeter thread number is read from the JMeter context and then stores this value into a variable named “thread”.

Executing Java libraries within the scripts
If you noticed in the scripts above, the Java libraries are exposed in JMeter under Packages.*. This allows you to execute any of the Java standard libraries or Java code in the classpath of JMeter. I think you can also write your own Java code and place it in the JMeter classpath and expose it in the same way.

Putting it all together
Putting all of that together gives you a handy way of doing reasonably complex performance testing in JMeter and I hope you find it useful.

NOSQL Exchange

This is a quick run through of the NOSQL exchange that Ciaran & I attended on Nov 2 at SkillsMatter, which featured 8 speakers and links to all talks are included.

A lot of people were asking which NoSQL solution to use?

This was the advice given by the speakers…. There is no silver bullet. Is there a need for reading/writing lots of Big data? Think about the shape of the data and how are you going to query your data to help understand which NOSQL solution fits best. Also understand the trade-offs when you choose your solution. Finally at the talks there was a lot of evidence of people using NOSQL solutions when a SQL solution would have sufficed.

This was the best talk of the day and anyone interested in NOSQL should watch the talk.

NOSQL stands for Not Only SQL.

Main types of NOSQL:

  1. Key-value originated from Amazon’s paper on Dynamo e.g. Riak, Voldemort (used in Linkedin)
  2. Column Family e.g. Cassandra, Hbase, Hyper table
  3. Document databases (most popular) descended from Lotus notes. e.g. CouchDb & MongoDb
  4. Graph Databases (nodes with properties) originated from Euler and Graph theory. e.g. infinitegraph, Neo4J

Documents are superset of Key-values. Graphs are supersets of documents and thus all others. Does this imply you should use Graph NOSQL solutions for all your NOSQL concerns? The graph NOSQL advocates think so.


  • Acidity is increasing e.g. MongoDb adding durable logging storage, Cassandra adding stronger consistency
  • More query languages – Cassandra -CQL, CouchDb UnQL, Neo4J – Cyper, Mongo.
  • Potentially more schemas?

NoSql challenges:

  • Mindshare
  • Tool support
  • Middleware support

Oracle now adopting NOSQL with a KeyValue solution despite debunking NOSQL in May this year. NOSQL seems to be following similar historical trends to SQL. SQL which had many vendors to begin with, over time resulted in 4 large vendors. Could NOSQL result in a similar situation in the near future?

Key quote from this talk: “Large systems are always in some degree of failure”

Problem: According to CAP: Consistency, Availability & Partition tolerance – you can’t have all 3. Have to compromise by picking 2.
In the case of a partition (P), trade availability (A) for consistency (C)
Else (E) trade latency for consistency (C)

Riak inspired by Dynamo. Built in Erlang/OTP. Has features such as MapReduce, links, full text search. Uses vector clocks not timestamps. Statebox for automation of resolving conflicts.
Uses a wheel for storing clustered data.


MongoDb is not suited for highly transactional applications or ad-hoc intelligence that requires SQL support. MongoDb resolves around memory mapped files. Mongo has an autosharding system.

Things to remember:
The datastore is a servant to the application not vice-versa
Don’t frankenshard

4) REAL LIFE CASSANDRA by Dave Gardner (from Cassandra user group)

  • Elastic – Read/Write throughput increases as scale horizontally.
  • Decentralised no master node.
  • Based on Amazon’s Dynamo paper
  • Rich data set model
  • Tunable
  • High write performance

If your requirements are big data, high availability high number of writes then use Cassandra.
When data modelling, start from your queries and work backwards.
Has expiring columns.
Avoid read before write & locking by safely mutating individual columns
Avoid super columns, instead use composite columns
Use Brisk (uses hadoop) for analysing data directly from Cassandra cluster.

5) DOCTOR WHO AND NEO4J by Ian Robinson
Although it was a fairly slick presentation it seemed to focus too much on modelling Doctor Who and his universe as a working example of graphs & Neo4J. Could this be to hide some shortcomings in Neo4J?

  • Neo4J is a fully ACID replacement for Mysql/Oracle.
  • Neo4j is a NOSQL solution that tries to sell itself as the most enterprise ready solution.
  • Has master/slave nodes.
  • Has 3 licenses: Community/Advanced/Enterprise.

With mentions of 2 phase commits, other than the advantage of modelling relationships such as social networks, there seemed little benefit from moving away from a relational database.
Having spoken to the Neo4J guys afterwards, it seems that the DB loses its ACIDity once you cluster it, and becomes another eventually-consistent store!



  • Written in Erlang has Lift support (Scala framework)
  • Exposes REST/JSON endpoints
  • Eventually consistent
  • Versioning appends only updates
  • Mapreduce for querying using views

A muddled presentation talking about mixing graph NOSQL solution with a document based one.

Rather than using the out of the box storage engines for NOSQL solutions, there can be dramatic throughput gains for using alternative storage engines such as Tokutek and Acunu (Castle).

Flash on the Beach 2011

Last month I went to Flash on the Beach in Brighton. It was the first time I’ve to this particular event and I really enjoyed a lot. For the few of you that don’t know, FOTB is an annual conference full of code, design and inspiration.

The three days in Brighton were full of inspiring stuff, like Carlos Ulloa new WebGL project, Lights, the Seb Lee-Delisle live experiment mixing OpenFrameworks, HTML5 and Phones, as well as lots of new people of the Elevator Pitches. Just check out the impressive main titles video created by Gmunk, creator of some of the FX for Tron Legacy.

Adobe showed some of the new stuff like EDGE, a Flash-like productivity tool to create HTML 5 canvas animations, and talked about Molehill/Stage 3D. They also talked about a new hardware accelerated framework for 2D animation in Flash recently released under the name of Starling. But all that doesn’t mean that that the “Flash is over” controversy wasn’t in the air, just check out the slide of Adobe’s official position on the question “When must I use Flash and when not?”.

Adobe posture on the Flash-HTML debate

Adobe posture on the Flash-HTML debate

Remy Sharp talked about this question too in “HTML5: Where Flash isn’t needed anymore”. He spoke about the fallback solutions when some features don’t work (usually using Flash), support of different parts of the specification in different browsers/platforms and the pitfalls that we will find on our path embracing this new standard.

In the closing speech John Davey, the creator and main organiser of the event, revealed that this is going to be the last Flash on the Beach! But don’t panic yet! It’s just a change of title and (maybe) format, to allow the conference to not be related with a single technology (i.e. the Flash platform) and be more based on design and creative coding (maybe something like the move of Flash Belt to The Eyeo Festival?)

Of course I’m leaving out a lot of things, like the interesting chat about kids games playability done by Jon Howard, or the demos of the new Away 3D using the new Stage 3D API, so don’t forget to check out the list of speakers and check out their individual sites.