A lazy adventure in Haskell

Monday-26
We’re seeing more and more Haskell pop up around Mind Candy. Time to get more familiar with it. Haskell has non-strict semantics, which in practice means a lot of Haskell is lazily evaluated (it’s not the same though).

For example, look at this piece of Scala code. It squares all the numbers in a list:

val l = List(1,2,3,4)
l.map(x => x*x)
// --> List(1, 4, 9, 16)

In Haskell this would look something like this:

let l = [1,2,3,4]
map (\x -> x*x) l

Only in Haskell, this would generally not do anything! We’re not using the result of the map anywhere. Haskell only evaluates the expression when it needs to. When you want to print it for instance:

let l = [1,2,3,4]
putStrLn $ show $ map (\x -> x*x) l
// --> [1,4,9,16]

Don’t worry about these ‘$’ signs. It’s basically the same as putStrLn (show (map (\x -> x*x) l)). It just makes the code less lisp-y. And show is more or less just .toString.

Actually, Haskell only evaluates only as much as it needs. So in the following example it would only evaluate the first element, but leave the rest of the list ‘unprocessed’. You could compare this a bit to boolean short-circuit evaluation in languages like Java/C++.

// Omitting the putStrLn but pretend it's there
let l = [1,2,3,4]
head $ map (\x -> x*x) l
// --> 1

So what good does that do us? In some cases this can actually improve performance. Let’s imagine that you have to get the x smallest numbers in a list of n numbers. (Maybe the scores of the winners in a golf tournament? Or the ages of the youngest people in your organisation?) Here’s a Haskell function that takes an x and an n, generates a list of n random numbers, then sorts it, and finally takes the first x numbers.

someFunc :: Int -> Int -> IO ()
someFunc x n = do
  -- this is just to show all IO 'as it happens'
  hSetBuffering stdout NoBuffering

  numbers <- time "Generating" $ getData n
  firstX <- time "Sorting" $ take x $ sort numbers
  putStrLn $ show firstX

We use a sort from the base library. The function getData looks like this:

getData:: Int -> [Int]
getData n = take n $ randoms $ mkStdGen n

There’s actually a neat thing in there! The function randoms takes a random number generator and produces an infinite list of random numbers! How come this doesn’t blow up your computer? Well, it produces it lazily, and since we only need the first n, all is well.

The function time just takes an arbitrary computation and prints how long it takes. The code is at the end of this post because it’s not really important right now. In case you’re wondering though, it does force the evaluation of the computation passed to it. Otherwise we would not be able to time it :)

These are the results of calling this code with n=2000000 and increasing xs:
What’s going on!? We’re obviously seeing lazy evaluation in action here. If we only need the first 10 ‘winners’ the sort seems super fast. If we need the first million, i.e., half the list, then sorting becomes much slower. This is simply because we don’t need to sort the whole list just to get the first x results. If we take x=2000000 the code takes ~12 seconds to complete (not shown in graph).

We can force the evaluation of an expression in Haskell by using force or seq (or BangPatterns). Let’s make this small change and try again:

  firstX <- time "Sorting" $ take x $ force $ sort numbers


As expected, this code has the same performance for all xs, the full 12 seconds as when we were asking for the completely sorted list.

What now?

The sort algorithm probably has O(n log n) time complexity, which is great. But can we do better, knowing that we only need a small number of values? Here’s an implementation of selection sort.

selSort :: (Ord a) => [a] -> [a]
selSort [] = []
selSort xs = let x = minimum xs in x : selSort (remove x xs)
  where remove _ [] = []
        remove a (y:ys)
          | y == a = ys
          | otherwise = y : remove a ys

Selection sort has horrible time complexity O(n^2). This particular implementation can probably be improved and it is not even in-place. Selection sort might be good for our use case though, since it basically builds up the sorted list with the values we want first.

An explanation of selection sort:

© http://www.cs.rmit.edu.au

© http://www.cs.rmit.edu.au

Here are the results alongside the results for the standard sort, for the first 100 xs.

This shows that selection sort beats the standard one for small values of x. Very small values of x

Conclusion

The lesson is of course that you should still use the optimised sort. Selection sort is faster only if you need just the smallest one in the list, but you might as well use the minimum function! The interesting observation is that due to the laziness, selection sort seems to actually perform as if you’re calling the linear time complexity minimum function! Even though this is an oversimplified example, it is clear that Haskell’s evaluation strategy can have interesting benefits on performance.

Note: micro benchmarks like these are a bit evil. Timings might be different on your own machine and performance of code within a larger program might exhibit very different behaviour.

The code for the time function for completeness sake:

time :: NFData a => String -> a -> IO a
time s a = do
    putStr $ s ++ ": "
    start <- getCPUTime
    let !res = force a
    end <- getCPUTime
    let diff = (fromIntegral (end - start)) / 
       (fromIntegral ((10::Integer) ^ (12::Integer)))
    printf "%0.3f sec\n" (diff :: Double)
    return res

Utilising AWS Lambda to migrate 25,000,000+ images S3 bucket

When AWS announced AWS Lambda at last year’s re:Invent, we were really excited about it here at Mind Candy. The concept of a zero-administration compute platform, that is very scalable, cheap and so easy to use AND at the same time integrates with so many AWS services through triggers is pretty exciting and potentially – very powerful.

Since then, we started using AWS Lambda in some of our products – PopJam being one of them. We use it to near-instantly generate thumbnails of all the amazing creations users of PopJam share through the app.

Recently, a quite interesting story surfaced on our sprint – we were to migrate one of the AWS S3 buckets PopJam uses, from US to EU (to bring it closer to the backend and users) without any downtime for users.

Now, you’ll think – “why that would be interesting?”

The answer is – 25,000,000+ – scale of this task.

The aforementioned AWS S3 bucket stores over 25,000,000 files (mostly images) and this number is growing faster every single day. Just running ‘s3cmd du’ on the bucket, took almost a day. When I tried to perform ‘s3cmd ls’ to count the number of keys in the bucket, I got bored before it finished (I had to write a simple Python script that utilises multi-processing and split the process of counting into 256 threads; only then would it finish within few minutes).

Obviously, any form of existing CLI command like s3cmd sync or AWS CLI s3 commands is out of question as before it finishes (after many, many hours), the source bucket will have tens of thousands of new files which haven’t been copied across and we’d have to re-run it again which would lead to the same situation.

I mentioned, AWS Lambda functions can be triggered by other AWS services; one of them being AWS S3. Essentially, we can configure an AWS S3 Bucket to invoke a Lambda function whenever a new object (key) is being created.

Given this, we could create a Lambda function on the old bucket that will be triggered whenever a new key is created (ObjectCreated event) that would copy over new keys to the new bucket. Then, we’d have to only sync the old bucket to the new one without having to worry about missing some keys on the way.

The proposed plan looked like this:

  1. Create new S3 bucket in EU
  2. Set up AWS Lambda Copy function and configure it to be triggered whenever a new key is added
  3. Run aws s3 sync command in background
  4. Wait, wait, wait…
  5. Reconfigure CDN to use the new bucket as origin
  6. Switch backend application to upload all images from now on, to the new S3 bucket in EU

This plan, also meant there should be zero downtime during the whole migration. Everyone likes zero downtime migrations, right?

The actual implementation, while not very painful, did uncover a few issues with the plan that had to be dealt with. These issues resulted in some learnings which I wanted to share here.

AWS Lambda copy object function

The Lambda function code to perform the copy happens to be pretty trivial.

var AWS = require(‘aws-sdk’);
var util = require(‘util’);

exports.handler = function(event, context) {
        var s3 = new AWS.S3(options = {region: “eu-west-1”});

        var params = {
                Bucket: ‘popjam-new-bucket’,
                CopySource: event.Records[0].s3.bucket.name + ‘/‘ + event.Records[0].s3.object.key,
                Key: event.Records[0].s3.object.key,
                ACL: ‘public-read’
        }

        s3.copyObject(params, function(err, data) {
                if (err) console.log(err, err.stack);  // an error occurred
                else     context.done();  // successful response
        });
};

It just works, but there’s one small culprit…

… what happens to S3 object ACLs should they be changed in the meantime?

We needed ACLs for particular objects to be in-sync (for various reasons, one of them being moderation).

Given the AWS Lambda function is triggered on ObjectCreated event (there sadly isn’t a way to trigger it on ObjectModify), should you need to change ACL there’s no way to do it through AWS Lambda at this stage.

We worked around this problem by writing a Python script that basically iterates through the S3 buckets, compares ACLs and tweaks them if there’s a need (as before, we had to parallelise it otherwise it’d take ages).

Beware of AWS Lambda limits!

While being pretty scalable, AWS Lambda has got some limits. We were bitten by the “Concurrent requests per account” and “Requests per second per account” limits a few times (fortunately we did just enough with AWS Lambda to get the attention of AWS Lambda product team and they kindly raised these limits for us).

For most of the use cases those limits should be fine, but in our case, when on top of the AWS Lambda copy function we were also triggering a series of functions to generate thumbnails, we hit these limits pretty quickly and had to temporarily throttle our migration scripts.

AWS Lambda is still pretty bleeding edge technology

AWS Lambda will work great for you most of the time. However, when it fails, troubleshooting can be quite … inconvenient to say the least.

Remember you can now view all AWS Lambda logs through CloudWatch – make use of them and don’t shy away from placing debug statements in your code.

The deployment of AWS Lambda is pretty tricky, too. While there are some options, it’s still in early stage and it feels like even AWS is still trying to figure it out (potentially through feedback from customers – if you use AWS Lambda do make sure to feedback to AWS).

The most interesting tool that I found out to support deployment and integrating with AWS Lambda in general is kappa

And all of this for what?

Let the graph speak for itself…

(the graph represents upload time to S3 bucket in US – green line, and S3 bucket in EU – orange line – after migration)

ReaderT 101

This blog post is about dependency injection (d.i.) using the Reader monad in Scala. I won’t explain what a monad is nor will I explore any category theory (mostly because I don’t know how to explain any of that). In this post I just want to show the mental model I have when using monadic style with ReaderT.

Note: this post turned out to be quite big. It’s not very dense though! Especially if you’re familiar with Scala you should be able to whisk through most of it.

Dependency injection

Code needs other code. That’s what d.i. is for me. We write separate pieces of code. Often one bit needs to use the other. I’ll use the following example source for this post:

case class Hero(name: String)

// imagine this is a database
case class Ooo(importantSettings: Unit) {
  private[this] val finn = Hero("Finn")
  private[this] val jake = Hero("Jake")

  def findHero(name: String): Hero = {
    // Imagine all kinds of database processing here
    finn
  }

  def friendsRegistry(): Map[String, Hero] = {
    // Moar processing
    Map(finn.name -> jake)
  }

  def evalAdventure(hero1: Hero, hero2: Hero): String = {
    // Jake always saves the day, he's a magic dog!
    if (hero1 == jake || hero2 == jake) "awesome" else "disappointing"
  }
}

// The instance of Ooo we want to inject everywhere
val ooo = Ooo(()) 

// This is a piece of 'business' logic
object AdventureTime {
  def getHero(name: String): Hero = ooo.findHero(name)

  def getBestFriend(hero: Hero): Hero = ooo.friendsRegistry()(hero.name)

  def goOnAdventure(hero1: Hero, hero2: Hero): String = {
    val result = ooo.evalAdventure(hero1, hero2)
    s"Adventure time with ${hero1.name} and ${hero2.name} was $result!"
  }
}
Adventure Time - Land of Ooo

Adventure Time – Land of Ooo

Instead of a stuffy real-world example I’m using Adventure Time. Think of Ooo as a database repository and AdventureTime as some piece of business logic. I assume this code is relatively simple and understandable. The problem is this: how does AdventureTime get a reference to Ooo? In other words, we want to inject Ooo into AdventureTime and possibly other parts of the code.

First, an example of how one could have an adventure:

import AdventureTime._

val hero1 = getHero("Finn")
val hero2 = getBestFriend(hero1)
val result = goOnAdventure(hero1, hero2) 

// result -> "Adventure time with Finn and Jake was awesome!"

A global variable and/or the Singleton

The example above illustrates one of the easiest ways of doing this: use a global variable and refer to that. This works great for small programs but when your program gets a bit larger, or your codebase is a bit older, this becomes very painful. Globals are difficult to maintain, they’re not very flexible, and they make code difficult to unit-test. You can also see in the example that the dependency is kind of hidden.

DI frameworks

Thankfully the industry has moved on from globals (right?) and frameworks like Spring and Guice have been invented to help. I won’t go into details about how they work, but they’re usually similar to constructor injection.

Constructor injection

In OO languages we can use the constructor of an object to provide it with the needed dependency. The AdventureTime object is now a class.

class AdventureTime(ooo: Ooo) {
  def getHero(name: String): Hero = ooo.findHero(name)

  def getBestFriend(hero: Hero): Hero = ooo.friendsRegistry()(hero.name)

  def goOnAdventure(hero1: Hero, hero2: Hero): String = {
    val result = ooo.evalAdventure(hero1, hero2)
    s"Adventure time with ${hero1.name} and ${hero2.name} was $result!"
  }
}

val at = new AdventureTime(ooo)

val hero1 = at.getHero("Finn")
val hero2 = at.getBestFriend(hero1)
val result = at.goOnAdventure(hero1, hero2)

This is a bit better than using global variables. Note that we still need some way to actually get ooo to where we create our at object, but in this post I want to focus on where the dependency is used. You can see that AdventureTime now has an explicit dependency on Ooo.

One caveat of this approach is that your class file should not become too large, otherwise you’re basically back to using a global variable! Constructor injection is not bad, it’s been used to create large systems. It’s fairly flexible, although you usually can’t change the dependency after it’s set. In order to test this you’d need to create a mock implementation or use a mocking library to mock the dependency.

What we actually want

We actually would like to pass the dependency as a parameter to every function that might need it.

object AdventureTime {
  def getHero(ooo: Ooo, name: String): Hero = ooo.findHero(name)

  def getBestFriend(ooo: Ooo, hero: Hero): Hero = {
    ooo.friendsRegistry()(hero.name)
  }

  def goOnAdventure(ooo: Ooo, hero1: Hero, hero2: Hero): Unit = {
    val result = ooo.evalAdventure(hero1, hero2)
    s"Adventure time with ${hero1.name} and ${hero2.name} was $result!"
  }
}
import AdventureTime._
val ooo = Ooo(())

val hero1 = getHero(ooo, "Finn")
val hero2 = getBestFriend(ooo, hero1)
val result = goOnAdventure(ooo, hero1, hero2)

This is a very flexible approach, we could change the dependency with each function call. We don’t need an instance variable to hold the dependency which makes this approach very suitable for, well, functions. We obviously see a pattern in these functions, but we can’t really abstract over it to remove the repetition.

Monads

Let’s see how we can use some functional programming and the Reader monad to improve this. Before we do that though, let’s quickly refresh how monads work. We use an all time favourite, the Option monad. Feel free to skip this explanation if you’re familiar with it.

The example code is actually not very null-safe.

val hero1 = getHero(ooo) // <- hero1 could be null
// which would probably make getBestFriend throw an NPE
val hero2 = getBestFriend(ooo, hero1)
// hero2 can also be null...
val result = goOnAdventure(ooo, hero1, hero2)

One way to handle this would be something like:

val hero1 = getHero(ooo, "Finn")
if (hero1 != null) {
  val hero2 = getBestFriend(ooo, hero1)
  if (hero2 != null) {
    val result = goOnAdventure(ooo, hero1, hero2)
  } else {
    println("No adventure today")
  }
} else {
  println("No adventure today")
}

This kind of clutters up things and distracts from what the code is actually trying to do. The Option monad represents the possibility that something can be null. We can encode this optional behaviour into the types. The monad then let’s us concentrate on the actual happy-path of the code while handling the boiler-plate around null-checking for us.

case class Ooo(importantSettings: Unit) {

  // It's possible the hero can't be found, so it's optional
  def findHero(name: String): Option[Hero] = {
    Some(finn)
  }

  def friendsRegistry(): Map[String, Hero] = {/* same as before */}

  def evalAdventure(hero1: Hero, hero2: Hero): String = {
    /* same as before */
  }
}

object AdventureTime {
  // Another Option here.
  def getHero(ooo: Ooo, name: String): Option[Hero] = ooo.findHero(name)

  // Yet another one. Types tend to ripple through a codebase
  def getBestFriend(ooo: Ooo, hero: Hero): Option[Hero] = {
    ooo.friendsRegistry().get(hero.name)
  }

  def goOnAdventure(ooo: Ooo, hero1: Hero, hero2: Hero): String = {
    /* same as before */
  }
}
import AdventureTime._
val ooo = Ooo(())

val result: Option[String] = for {
  hero1 <- getHero(ooo, "Finn")
  hero2 <- getBestFriend(ooo, hero1)
} yield goOnAdventure(ooo, hero1, hero2)

println(result.getOrElse("There was no adventure :("))

The Option monad does exactly what we want. If there are no nulls, everything works as before. If there is a null somewhere in the process, it kind of ‘sticks’. I.e., no subsequent code is executed and a None is returned. It’s not exactly ‘as before’, we’ve obviously switched to a for comprehension.

We’ve enhanced the return types of our functions to deal with a kind of ‘secondary’ logic so we can focus on the main functionality that we’d like to express. That sounds familiar. What if we could encode our dependency into the return type as well?

Enter the Reader

The Reader monad basically encodes a simple function. It’s type definition is:

type Reader[E, A] = ReaderT[Id, E, A]

Let’s forget the right hand side of that type alias for now. Reader just expresses a function that takes a parameter of type E and returns a value of type A. Think of it as:

def func(e: E): A = {
  // create some A using e
}
// or
val func = (e: E) => {
  new A(e.foo())
}

You see how we could use that to express a dependency. The first type parameter E stands for ‘environment’. In our code E is Ooo and A is whatever our functions return. E.g., an Option[Hero] or a String. The type signature of getHero would become def getHero(name: String): Reader[Ooo, Option[Hero]]. Read: “getHero is a function that returns a function. When the returned function is supplied an Ooo it will return an Option of Hero“.

Let’s add this to our example. Note that all the functions in AdventureTime have the same dependency, so we make a little type alias for it. I’m assuming the reader is familiar with the various ways of creating lambda functions in Scala.

// Warning: this is not the final example, don't write code like this!
type OooReader[X] = Reader[Ooo, X]
object AdventureTime {

  def getHero(name: String): OooReader[Option[Hero]] = Reader{
    (ooo: Ooo) => ooo.findHero(name)
  }

  def getBestFriend(hero: Hero): OooReader[Option[Hero]] = Reader{
    _.friendsRegistry().get(hero.name)
  }

  def goOnAdventure(h1: Hero, h2: Hero): OooReader[String] = Reader{
  (ooo: Ooo) =>
    val resultOfAdventure = ooo.evalAdventure(h1, h2)
    s"Adventure time with ${h1.name} and ${h2.name} was $resultOfAdventure!"
  }
}
import AdventureTime._

val res = for {
  hero1 <- getHero("Finn")
  hero2 <- getBestFriend(hero1.get) // .get !? ick...
  result <- goOnAdventure(hero1.get, hero2.get)
} yield result

This looks similar to before, but we’ve managed to remove all the ooo parameters. Hang on, where are we injecting ooo now? Well, we’re not. This code seems to not do anything. If you inspect the type of res you’ll see it’s scalaz.Kleisli[scalaz.Id.Id,Ooo,String]. 😱

Remember that getHero returns an OooReader, i.e., a function taking an Ooo and returning an Option[Hero]. getBestFriend actually has the same signature. Just like Option, using Reader in a for comprehension sequences the monads into a ‘bigger’ one. For Option this means combining potentially absent values. For Reader it just means: “keep passing the dependency to the next function”. We’ve basically combined all three function calls into one big Reader.

If we want to execute the code we need to supply it with an Ooo using the run function of Reader.

res.run(Ooo(()))
// --> scalaz.Id.Id[String] = Adventure time with Finn and Jake was awesome!
Monad Transformer

Monad Transformer

We’ve run into a problem though. We had to resort to the evil get function for unwrapping our Options. So the Reader basically undid all the Option monad goodness. Ideally the code should handle both monads at once. Fortunately there is a monad transformer for Reader called ReaderT.

What was that weird type signature and what is this Id stuff? Remember the right hand side of the Reader type alias? It was ReaderT[Id, E, A]. It turns out that instead of working with functions of type E => A, we usually work with functions like E => M[A], where M is some kind of monad. ReaderT expresses just that. Reader is actually an alias for ReaderT where M is the Id monad. I see Id as the ‘does nothing’ monad.
ReaderT looks like this:

type ReaderT[F[_], E, A] = Kleisli[F, E, A]

What? Another type alias? Yes, ReaderT is actually equivalent to Kleisli, which is what scalaz uses. Kleisli also adds many convenience functions for combining Kleislis.

Let’s rewrite our example using Kleisli instead:

object AdventureTime {
  // Kleisli[Option, Ooo, Hero] 'represents' Ooo => Option[Hero]
  def getHero(name: String) = kleisli[Option, Ooo, Hero](_.findHero(name))

  def getBestFriend(hero: Hero) = kleisli[Option, Ooo, Hero]{
    _.friendsRegistry().get(hero.name)
  }

  def goOnAdventure(h1: Hero, h2: Hero) = kleisli[Option, Ooo, String]{
  (ooo: Ooo) => 
    val resultOfAdventure = ooo.evalAdventure(h1, h2)
    Some(s"Adventure time with ${h1.name} and ${h2.name} " +
         s"was $resultOfAdventure!")
  }
}
import AdventureTime._

val res = for {
  hero1 <- getHero("Finn")
  hero2 <- getBestFriend(hero1)
  result <- goOnAdventure(hero1, hero2)
} yield result

res.run(Ooo(()))

Before we had Reader just wrapping a function that matches the desired type. There is no such constructor for ReaderT, probably just because kleisli already does exactly the same. In other words, one can create a ReaderT using the kleisli function. The type parameters in order are: the monad of the return value, the environment of the function, and the type of the return value.

The Future

This all looks nice but we might not be convinced yet. Sit tight, I’ll show you a great advantage of using Reader. We’ll have to go even more functional though.

Our for comprehension should belong in some function in the logic layer of our program. We’ve abstracted the dependency on Ooo through the Reader but the sample code still strongly couples to AdventureTime. Let’s remove that by passing the necessary functions as parameters instead!

object SomeFancyLogic {
  def startEpicAdventure(
    getHero: (String) => ReaderT[Option, Ooo, Hero],
    getBestFriend: (Hero) => ReaderT[Option, Ooo, Hero],
    goOnAdventure: (Hero, Hero) => ReaderT[Option, Ooo, String])
   (name: String): ReaderT[Option, Ooo, String] = {
    for {
      hero1 <- getHero(name)
      hero2 <- getBestFriend(hero1)
      result <- goOnAdventure(hero1, hero2)
    } yield result
  }
}

// We usually 'wire up' the parameter group containing the
// functions first
val startEpicAdventureWired = SomeFancyLogic.startEpicAdventure(
                                          AdventureTime.getHero _,
                                          AdventureTime.getBestFriend _,
                                          AdventureTime.goOnAdventure _) _

startEpicAdventureWired("Finn").run(Ooo(()))

Let’s also make our ‘database’ a bit more realistic. In the server world we like to avoid blocking, so APIs for external services usually return Futures.

// The land of Ooo of the future
case class Ooo(importantSettings: Unit) {

  // findHero now returns a Future
  // for simplicity I'm ignoring the Option stuff.
  def findHero(name: String): Future[Hero] = {
    Future.successful(finn) // again, just simulating here..
  }

  def friendsRegistry(): Future[Map[String, Hero]] = {
    Future.successful(Map(finn.name -> jake))
  }
  
  def evalAdventure(hero1: Hero, hero2: Hero): Future[String] = {
    Future.successful{
      if (hero1 == jake || hero2 == jake) "awesome" else "disappointing"
    }
  }
}

// The rest of the code stays almost the same!
// Just change the Monad type parameter from Option to Future

object AdventureTime {
  def getHero(name: String) = kleisli[Future, Ooo, Hero](_.findHero(name))

  def getBestFriend(hero: Hero) = kleisli[Future, Ooo, Hero]{
    _.friendsRegistry().map(_(hero.name))
  }

  def goOnAdventure(h1: Hero, h2: Hero) = kleisli[Future, Ooo, String]{
  (ooo: Ooo) =>
    ooo.evalAdventure(h1, h2).map{result =>
      s"Adventure time with ${h1.name} and ${h2.name} was $result!"
    }
  }
}

object SomeFancyLogic {
  def startEpicAdventure(
    getHero: (String) => ReaderT[Future, Ooo, Hero],
    getBestFriend: (Hero) => ReaderT[Future, Ooo, Hero],
    goOnAdventure: (Hero, Hero) => ReaderT[Future, Ooo, String]
  )(name: String): ReaderT[Future, Ooo, String] = {
    for {
      hero1 <- getHero(name)
      hero2 <- getBestFriend(hero1)
      result <- goOnAdventure(hero1, hero2)
    } yield result
  }
}

/* wiring as before, snipped for brevity o_O */

val future = startEpicAdventureWired("Finn").run(Ooo(()))
Await.result(future, 2.seconds)

A pattern is emerging here! We can actually abstract out the monad! We can also abstract away the dependency on Ooo. It looks like this:

object SomeFancyLogic {
  def startEpicAdventure[M[_]: Monad, E](
    getHero: (String) => ReaderT[M, E, Hero],
    getBestFriend: (Hero) => ReaderT[M, E, Hero],
    goOnAdventure: (Hero, Hero) => ReaderT[M, E, String]
  )(name: String): ReaderT[M, E, String] = {
    for {
      hero1 <- getHero(name)
      hero2 <- getBestFriend(hero1)
      result <- goOnAdventure(hero1, hero2)
    } yield result
  }
}

E is now the generic type for the dependency. M[_] is a type that is actually a type constructor. Look at it as a type with a hole that needs another type to be whole. E.g., Option[String] or Future[Hero]. We also specify that there needs to be an implementation for the Monad type class for M.

The cherry on top

Wildberry is not a cherry but she is pretty.

Wildberry is not a cherry but she is pretty.

Testing this piece of logic now becomes pretty easy. Of course the logic is really simple here.

A unit test should only test the code-under-test. With our new function parameters this means we can easily instruct our test without using any mock libraries. We test Popjam using ScalaCheck to do extensive property based testing. Also note that while the database is using Futures, we don’t actually want to test the asynchronous behaviour of the code, just the logic. Moreover, creating tests with concurrency in them usually leads to brittle time-dependent tests.

Here’s how we could test our logic:

def testEpicAdventure() = {
  // our 'mocked' functions. Usually we would make them return
  // more useful results obviously
  val getHero = (name: String) => kleisli[Id, Unit, Hero]{
    _ => Hero(name)
  }
  val getBestFriend = (h: Hero) => kleisli[Id, Unit, Hero]{
    _ => Hero("Jake")
  }
  val goOnAdventure = (h1: Hero, h2: Hero) => kleisli[Id, Unit, String]{
    _ => "Test adventure"
  }
  
  val wired = startEpicAdventure(getHero, getBestFriend, goOnAdventure) _
  val result = wired("Finn").run(())

  result aka "how did the adventure test go" should equal("Test adventure")
}

We can just use Id for our monad and Unit for the database. I’ve found this way of testing to be a lot more fun than setting up complicated mock, stub, or spy objects.

There are a lot more things we can do with scalaz and ReaderT. Like MonadReader ask for instance. I encourage you to go on that adventure yourself!

Testing with Amazon SQS

We all know how great Amazon SQS is, and here at Mind Candy we use it extensively in our projects.

Quite recently, we started making some changes to our Data Pipeline in order to speed up our Event Processing, and we found ourselves with the following problem: how can we generate thousands of messages (events) to benchmark it? The first solution that came into our minds was to use the AWS Command Line Interface, which is a very nifty tool and works great.

The AWS Command Line Interface SQS module comes with the ability to send out messages in batches, with a maximum of 10 messages per batch, so we said: “right, let’s write a bash script to send out some batches”, and so we did.

Problem

It worked alright, but it had some problems:

  • It was slow; because messages were being sent in batches of up to 10 messages and not in parallel
  • The JSON payload had to contain some metadata along with the same message repeated 10 times (1 for each message entry)
  • If you needed to send 15 messages, you would have to have 1 message batch with 10 entries and another one with 5 entries (2 JSON files)
  • Bash scripts are not the best thing in the world for maintenance

So, what did we do to solve it? We wrote our own command line program, of course!

Solution: meet sqs-postman

Writing command line applications in Node.js is very very easy, with the aid of the good old Commander.js. Luckily, AWS has an SDK for Node.js, so that means that we don’t need to worry about: AWS authentication, SQS API design, etc. Convenient? Absolutely!

Sqs-postman was designed with the following features out of the box:

  • Sends messages in batches of up to 10 messages at a time (AWS limit)
  • Batches are sent out in parallel using a default of 10 producers, which can be configured using the –concurrent-producers option
  • A single message is read from disk, and expanded into the total number of messages that need to be sent out
  • It supports AWS configuration and profiles

In order to solve the “messages in parallel” problem, we used the async library. We basically split the messages into batches and we then use eachLimit to determine how many batches can be executed in parallel, which starts with a default value of 10 but can be configured with an option.

Can I see it in action?

Of course you can! sqs-postman has been published to npm, so you can install it by running:

 npm install -g sqs-postman

Once installed, just follow these simple steps:

  • Make sure to configure AWS
  • Create a file containing the message, i.e. message.json with a dummy content
    {
       "message": "hello from sqs-postman"
    }
  • Run it
    $ postman message my-test-queue --message-source ./message.json --concurrent-producers 100 --total 1000

If you would like to see more information, the debug mode can be enabled by prepending DEBUG=sqs-postman postman…

Text is boring, show me some numbers!

You are absolutely right! If we don’t share some numbers, it will be hard to determine how good sqs-postman is.

Messages aws-cli sqs-postman
100 0m 4.956s 0m 0.90s
1000 2m 31.457s 0m 4.18s
10000 8m 30.715s 0m 30.83s

As you can appreciate, the difference in performance between aws-cli and sqs-postman is huge! Because of sqs-postman’s ability to process batches in parallel (async), the execution time can be reduced quite considerably.

These tests were performed on a Macbook Pro 15-inch, Mid 2012 with a 2.6 GHz Intel Core i7 Processor and 16 GB 1600 MHz DDR3 of RAM. And time was measured using Unix time.

Conclusion

Writing this Node.js module was very easy (and fun). It clearly shows the power of Node.js for writing command line applications and how extensive the module library is when it comes to reusing existing modules/components (e.g. AWS SDK).

The module has been open sourced and can be found here. Full documentation can be found in there too.

As usual, feel free to raise issues or better yet contribute (we love contributors!).

A Puppet module for Dynamic DynamoDB

As my colleagues have said in other posts, we make an extensive use of Amazon Web Services at Mind Candy. Recently we decided to use the AWS NoSQL offering DynamoDB for a specific use case in one of our products.

Whilst DynamoDB provides us with a highly distributed NoSQL solution, it works based on telling Amazon what read and write capacity you require via their API. If you find that you go over either of these value you begin to, potentially at least, lose queries if you have not factored in some sort of caching layer using, for example, Amazon SQS.

In the ideal world, Amazon would offer auto scaling features for DynamoDB, however at time of writing they don’t. Instead they advise people to use an independently developed tool called Dynamic DynamoDB written by Sebastian Dahlgren.

Dynamic DynamoDB is a tool written in Python that allows us to effectively auto scale our provisioned reads and writes. It use CloudWatch metrics to establish current usage and then based on the configuration option either scales up or down your provisioned capacity on a per table basis.

As I’ve posted before here, we use Puppet at Mind Candy, so the first point of call whenever a new tool comes along is to see if anyone has written, or started to write, a Puppet module for it. Sadly it didn’t look like anyone had, so we quickly wrote up our own, which is available on Github here.

Event Processing at Mind Candy

At Mind Candy we want to build great games that are fun and that captivate our audience. We gather a great deal of data from all of our products and analyse it to determine how our players interact with our games, and to find out how we can improve. The vast majority of this data consists of ‘events’; a blob of json that is fired by the client or server in response to an interesting action happening in the game.

This blog post is about the approach that we have taken at Mind Candy to gather and process these events, and scale the systems into the cloud using fluentd, akka, SQS, Redshift and other AWS Web Services.

What is an event?

From our point of view, an event is any arbitrary valid json that is fired (sent) to our Eventing service via a simple REST api.

When an event is received, it is enriched with some additional fields which includes a ‘fired_ts’ of when the event was received, a unique uuid and, importantly, the game name, version, and event name taken from the endpoint. These three together form what we call the ‘event key’.

This service is extremely lean, and does not itself expect or enforce a rigid taxonomy. It simply writes the enriched events to disk. As a result, the service is incredibly easy to scale and to achieve high availability.

Validation and processing

We then use fluentd, an open source data collector and aggregator, to take the enriched data written to disk and place it onto an SQS queue. Currently, we use a single queue (per environment) which receives data from many different eventing servers.

Once all that data is arriving on the queue, we need to do something useful with it! This is where our home grown event processing engine, Whirlpool, comes into play.

Whirlpool is a scala and akka based service which retrieves messages from SQS and processes and validates them accordingly. It uses a variant of the akka work-pull pattern with dedicated workers for pre-fetching, processing, and writing events, communicating with a master worker. The number of workers and other parameters can be tweaked for maximum throughput.

Where does the metadata for processing come from? We have a shared ‘data model’ which contains information on what an event should look like for a specific game and version. This is essentially a scala library that reads from a backing Postgres store.

The structure of that schema is (simplified):

Screen Shot 2014-07-25 at 15.49.50

An event field is a single field to be found in the json of the sent event. It has a number of different properties, for example whether it is mandatory or not, and whether it should be expanded (exploded out into multiple events), and the json path to where that field should be expected. The point of the eventversion table is to provide a history, so that all changes to all events are recorded over time so we have a rollback, as well as an audit trail for free.

An event destination configures where an event should end up in our warehouse. It can be copied to any number of schemas and tables as we require.

Whirlpool retrieves the metadata for an event based on the extracted event key. It then passes the event through a series of validation steps. If it fails at any level, the reason why is recorded. If it completes all validations, the event can be processed as expected.

The processMessage function looks like this:

Screen Shot 2014-07-25 at 16.49.28

We use Argonaut as our JSON processing library. It is a fully functional library written in Scala that is very nice to work with, as well as having the added benefit that our resident Mind Candy, Sean, is a contributor!

After our events have been validated, they are either a successful event for a particular game and version, or a failure. At this point we make use of fluentd again with a modified version of the Redshift plugin to load them into our Redshift data warehouse. Here they are available for querying by our data scientists and data analysts. Typically, the period from an event being received to being queryable within the data warehouse is measured in seconds, and in any case within a couple of minutes in normal cases.

Configuring events

To actually setup the metadata for what constitutes an event, we have created a simple GUI that can be accessed by all game teams. Any changes are picked up within a few minutes by Whirlpool, and those events will start to flow through our pipeline.

We also needed to solve one large problem with the configuration, namely: “How do you avoid having to create a mapping for every single game version when the events haven’t changed, and how do you accommodate for changes when they do occur?”

It took us a while to find a nice balance for solving this, but what we have now is a mapping from any POSIX regex which is matched against an incoming game version, to a specific version that should be used for retrieving the metadata (this is the purpose of the ‘configmapping’ table in the schema). So, when we release 1.0 of our game, we can create metadata that applies to “1.x”. If in version 1.5 we introduce a new event, we can create a new config at that point to apply to all later versions, while still having versions 1.0-1.4 processed correctly.

Handling Failure

Events can fail for a large variety of reasons. Currently there are 17 specific types of these, with a couple being:

  • The event is malformed; it does not contain the fields that we expect
  • The event is unknown

A failure is captured by the following class:

Screen Shot 2014-07-25 at 16.49.46

The FailureType here is another case class corresponding to the specific failure that was generated, and the fields contain some additional attributes which may or may not be extracted from the failure.

We treat failures separately from processed events, but they still make their way into Redshift in a separate schema. Each failure contains enough information to identity the problem with the event, which can then be fixed in most cases in the metadata; typically, event failures occur during development, and are a rare occurrence in production.

Scaling our infrastructure

We make heavy use of AWS at Mind Candy, and the eventing pipeline is no exception. All the eventing servers are described via Cloud Formation, and setup in an autoscale group fronted by an ELB. As a result, the number of servers deployed scales up and down in response to rising and waning demand.

The use of SQS also separates out our event gathering and event processing infrastructure. This means that Whirlpool instances do not have to scale as aggressively, as the queue provides a natural buffer to iron out fluctuations in the event stream due to peaks of traffic. For Redshift, we have a 6XL node cluster which we can scale up when required, thanks to the awesome features provided by Amazon.

Performance

We’ve benchmarked each of our eventing servers comfortably processing 5k events/sec, on m1.medium instances.

Whirlpool does a little more work, but we are currently running a configuration offering a sustained rate of just over 3k events/sec per instance, on c1.medium instances, with a quick ramp up time.

Instances of both Eventing and Whirlpool operate independently, so we scale horizontally as required.

Screen Shot 2014-07-25 at 16.24.54

The Future

We have real-time dashboards that run aggregations against our event data and display it on screens around the office. It’s very useful, but is only the first incarnation. Currently we’re working on streaming processed events from Whirlpool into Spark via Kafka, to complete our lambda architecture and greatly reduce the load on our Redshift cluster. We’re also improving the structure of how we store events in Redshift, based on our learnings over the last year or so! At some point when we have more time, we would also like to open-source Whirlpool into the community.

 

Using giter8 to build Scala REST projects with Unfiltered, Netty and Gatling load testing

At Mind Candy we have a number of different Scala REST services to provide common aspects to our games, for example authentication and moderation.

A standard we have developed is to use an Unfiltered Netty standalone server wrapped with the apache-commons daemon library. Also, we want our services to be well tested, easily deployable, and stable under heavy load. For ease of deployment we use Fabric scripts, and for load testing we use Gatling to hammer the services under different stress-tests to check they are up to standard.

We have quite a few services that have a similar setup, and it was getting tedious to create and configure a new project. So, we created a giter8 script to do it all for us!

Giter8

Giter8 is an awesome command line tool which can create a template project structure for you with example files and sbt configuration.

There are quite a few templates contributed by the community already but none did exactly what we want, so we created our own which we’d like to share.

The template will create an example sbt project that:

  • Uses unfiltered and netty to give you a very simple starting endpoint service which will output ‘Hello, <project>’ when hit
  • Gives you some basic test stubs using Specs2
  • Is setup with gatling load testing with an example scenario
  • Is configured with the sbt-idea plugin (IntelliJ IDEA is our preferred IDE)
  • Is configured with the sbt-assemblysbt-release, and sbt-dependency-graph plugins
  • Has a simple fabric deploy file, unix start/stop scripts, and an example init script

The generated project is structured with 3 sub projects:

  • project-example
    • example-core // Business logic
    • example-resources // Containing the Unfiltered Netty request plan
    • example-server // To control the server hosting the plan

Usage:

First make sure you have setup giter8 using the instructions here. Then fire open a terminal and run:

g8 mindcandy/unfiltered-rest-gatling

There are some options to enter, but most can be left as default (See here for more info). The important ones are:

name => The main projects name
project => The name prepended to each of the sub projects
organization => Used as the basis for packages

Once you’ve entered the properties, giter8 will do its magic and whip you up a full sbt project. After it completes, you can cd to the directory, and run sbt.

Test it:

sbt test

Run it:

sbt run

Now open up your favourite browser and go to http://localhost:8889/<project> where <project> is whatever you specified above, and you should get a “hello, project” message back.

Load test it:

Start the server running in one terminal. In another terminal do:

cd yourproject
sbt
project <project>-server
test-load:run

You should get presented with a choice of different classes to run:

  • Engine → Executes the gatling Engine in interactive mode prompting you which scenario to run
  • BatchEngine → Runs all simulations available in a batch with no user prompts
  • Recorder → Starts the pretty cool Gatling recorder which you can use to create scenarios. Basically, you setup your web browser to use the recorder’s proxy, and then just browse as normal on your webpage. Gatling records all of your actions as a scenario that can be replayed and customised.
Hit the option for BatchEngine, then load the results in your browser to see something like:

Results from Gatling are put in the gatling/results directory. In the example scenario in the giter8 skeleton we simulate 10 users hitting the simple endpoint, ramping up over a 3s duration. The results are rendered in a nice html page with graphs showing exactly what happened during the simulation. The full gatling feature set is quite extensive and worth checking out.

Create IntelliJ IDEA project files:

sbt gen-idea

Run from start script:

First setup jsvc. Then :

sbt assembly
cd bin
chmod +x start.sh
./start.sh

This has the same affect as ‘sbt run’, but using the apache-commons daemon wrapper. Try opening the service in your browser again and it should work as before. Don’t forget to stop the server when you’re done with the ../stop.sh script :)

Other pieces:

There is also a sample init script that is configured for the skeleton in the bin folder, which is a good starting point for creating a complete init script.

Last to mention is there is a small fabric file that provides a (very) basic setup for copying over the built assembly onto a test/integration server. You’d need to configure your host and change the paths as appropriate to use it, but it’s useful as a starting point.

Let us know if you find this useful, and thanks to n8han for providing a great tool!

Migrating a Play 1.2 website to Play 2.0

At Mind Candy there are a number of internal websites used for reporting and communication. For example – reading automated build/test status via some REST APIs and turning this into a nice visual status display for large ‘build screens’. These have been authored in Scala using the Play Framework 1.2.x.

Recently, version 2.0 of the Play Framework came out of beta and so I wanted to convert the existing Play 1.2 websites over to it. Amongst other reasons, the ability to build to a .jar file makes for simpler deployment, which I was keen to have.

I couldn’t find a good guide for migrating from 1.x -> 2.0 so I am sharing my experiences of porting a Scala Play 1.2.x application. I expected that it wouldn’t be too hard as I was already using Scala Templates and controllers. That was mostly true, with a couple of issues.

I did find a handy conversion guide on the Play 2 wiki which can help converting some uses (it has some unfortunate formatting though so I’d recommend copying and pasting it)

Starting the project

I’d previously installed Play via brew so an upgrade was as simple as
brew update
brew upgrade play

This installed Play 2 whilst leaving Play 1.2.4 also installed for easy roll-back for developing older stuff if I need it.

For simplicity and because there were various code changes that I knew I needed to make, I created a new site in a new location using play new. However I then used svn copy to copy over source assets and code before modifying. This means in the future I can pull updates down using “svn merge” as work continued on the old site whilst I was porting it over!

All of the public files which aren’t built – javascript, css, images – were simply copied across. In the future I’ll check Play’s support for LESS CSS and the Google Closure JS compiler but for now I just want to get things working.

Initially I copied across all the controllers, models and views and configuration files, though I expected to have to fix those up as the syntax had changed.

After a quick run of play compile I had a whole host of build errors, unsurprisingly. So to cut these down I commented out a lot of the logic – all the routes except the home page /, all the templates and all the controllers except for those needed by the home page.

I fixed up controller in turn and gradually re-enabled the routes, until I hit an issue with Database models that stopped me from migrating the rest of my application (see below)

Migrating dependencies

There were a number of java libraries used by the web app, for example to talk to a Subversion server. (This is one of the reasons I really like Scala – it’s easy to pull in useful Java libraries) In Play 1 that was defined in dependencies.yml, e.g.:

require:
- org.tmatesoft.svnkit -> svnkit 1.3.7
- org.tmatesoft.svnkit -> svnkit-javahl16 1.3.7

repositories:
- tmatesoft:
  type: iBiblio
  root: "http://maven.studio:8081/nexus/content/repositories/tmatesoft"
  contains:
  - org.tmatesoft.svnkit -> *</code>

Play 2 now uses sbt as its build system and thus uses the standard sbt syntax for dependencies, so I edited the the project/Build.scala file to add dependencies:

object ApplicationBuild extends Build {
  // some definitions omitted

  val appDependencies = Seq(
    "org.tmatesoft.svnkit" % "svnkit" % "1.7.4",
    "org.tmatesoft.svnkit" % "svnkit-javahl16" % "1.7.4",
  )

  val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
    resolvers += "TmateSoft Repository" at "http://maven.tmatesoft.com/content/repositories/releases/",
  )
}

Once the sbt syntax is understood it’s a reasonably simple process to convert dependencies across.

The original play run / test / start / stop command still work, which is helpful. It’s also quite useful to simply run play to get an sbt console and then run ~test which runs test and then re-runs them anytime a source file changes, which is very useful.

For IDE integration, I use IntelliJ IDEA and happily there is an idea command in Play’s SBT which generates an IntelliJ module which has any library dependencies included. However, your code has to build correctly before this works and you probably need to remember to re-run this every time you modify a dependency. If you don’t the IDE won’t show types correctly – typically you will get more ‘false negatives’ of errors showing up in the IDE which will not happen within Play itself.

Migrating the routes file

For some reason the old Routes file had an explicit 404 for the for the favicon. There didn’t seem to be an obvious pure route-based replacement for this and I didn’t need that behaviour so removed the route.

I had to add controllers. to all routes as the full package reference seems to be required now.
e.g. home page changed from
GET / Application.index
to
GET / controllers.Application.index

The syntax for ‘public’ files has changed so I replaced
GET /public/ staticDir:public
with
GET /public/*file controllers.Assets.at(path="/public", file)

I was also serving a single file elsewhere – multiple Assets.at() in the routes file didn’t go well, but I was able to call to add a new method to the Application e.g. ‘serveMyFile’ and then set the routing to be controller.Application.serveMyFile in routes and then have that call the Assets.at() method
e.g.
GET /myfile controllers.Application.serveMyFile
in the Application controller:
def serveMyFile = controllers.Assets.at(path="/path/to/it", file="myfilename.json")

The syntax for parameters in routes has changed and I found that I needed to move the default parameter definitions from the Controller out to the Routes file, using the ?= syntax instead of the Option[T] syntax I had before. That wasn’t hard to do but was slightly irritating in some ways – I’m still not sure if I like having so it in the routes file.

Minor changes to the syntax for parsing out ids from the URL, for example:
GET /versionone/stories/{projectId} versionone.Printer.storiesById
changed to
GET /versionone/stories/:projectId controllers.versionone.Printer.storiesById(projectId: String, timeboxId: String ?= "", template: String ?= "")

Note the additional default parameters being passed. Also in this instance the Id is actually a string, not an Int.

Migrating Controllers

A lot of the package names have changed e.g. play -> play.api so some of the imports needed to be fixed up with the new ‘api’

replaced
import play._
import play.mvc._

with
import play.api._
import play.api.mvc._

With the paths to generated views also changed, I removed lines like import views.Application._

Then with imports fixed up I could then fix up the actual controller code to return an Action and
adding Ok(views.html. … )
e.g. from
def index = {
html.index()
}

to
def index = Action {
Ok(views.html.Application.index())
}

As I mentioned earlier, some changes to controllers were needed because the routes file now supplies default parameters, and I took out some use of the Option[T] type — though as this was generally a String, the empty string works very well as an equivalent to None.

In some controllers I’ve implemented an OAuth callback to read from a Google Calendar – eventually I’d like to play with the built-in OAuth support in Play 2 but in the interest of getting things working quickly I wanted to port over my current code. In order to send a url for Google auth to use as a callback I am calling Router.getFullUrl() but this doesn’t work in Play 2, instead one calls the automatically generated route classes instead. To turn this into a full url you call .absoluteUrl(), but you must remember to add the implicit request to your controller code in order to give this the context needed to generate the url.
e.g. the original code was approximately:

object MyController extends Controller {
  def request = {
    val callbackUrl = Router.getFullUrl(“MyController.callback”)
    // send web request using callbackUrl
  }

  def callback = {
    // handle request
  }
}

the new Play 2 code is:

object MyController extends Controller {
  def request = Action { implicit request =>
    val callbackUrl = routes.MyController.callback().absoluteURL()
    // send web request using callbackUrl
  }

  def callback = Action {
    // handle request
  }
}

This is more robust because that reverse-lookup is now checked at compile time, rather than runtime. However if your code doesn’t build, your IDE will give you errors as the reverse routes don’t exist. I did occasionally have to do a clean build to force regeneration of reverse-lookups, if the code structure had changed significantly.

Migrating Views

I had to replace @asset() with @routes.Assets.at()

Old references to actions can now be replaced with the compile-time checked generated routing objects. But the snag that isn’t documented is that if you are using a sub-project, the routes are generated within the scope of that sub-project.

e.g. replace @action(svn.Sizes.index()) with @controllers.svn.routes.Sizes.index()
This was mentioned on the Play discussion group here.

Migrating Tests

Unfortunately the bundled Play test framework has changed from scalatest to specs2. I could possibly have set up scalatest to work but decided to migrate the tests instead as the new functional tests look useful, as is the ‘property’ support in specs2.

A few imports change – I had to get rid of the scalatest imports and add in:

import org.specs2.mutable._
import play.api.test._
import play.api.test.Helpers._

The test class now simply extends Specification and I had to fiddle around with the syntax, replacing it with a noun and adding some more braces. should be(x) was replaced with mustEqual x

For example:

it should "retrieve svn server prefix" in {
  val svnServers = SvnServer.find().list()
  svnServers.length should be > (0)
}

becomes:

"SvnServer" should {
  "retrieve svn server prefix" in {
    val svnServers = SvnServer.find().list()
    svnServers.length must be_>(0)
  }
}

There is a good list of all the matchers in specs2 at the specs2 site which I found very useful.

I migrated some very basic selenium tests to the new functional test framework – documented here. The advantage of this is that you can separately test the controller, view and the whole thing together with the router.

The old selenium test was

#{selenium 'Home page'}
// Open the home page, and check that no error occured
open('/')
assertNotTitle('Application error')
#{/selenium}

So a simple test that the home page works is:

class ApplicationTest extends Specification {

  "The Application" should {
    "render template correctly " in {

      val html = views.html.Application.index()

      contentType(html) must equalTo("text/html")

      val content = contentAsString(html)
      content must contain ("Welcome to the Tools website")
      content must not contain ("Application error")
    }

    "have a working controller" in {
      val result = controllers.Application.index()(FakeRequest())

      status(result) must equalTo(OK)
      contentType(result) must beSome("text/html")
      charset(result) must beSome("utf-8")

      val content = contentAsString(result)
      content must contain ("Welcome to the Tools website")
      content must not contain ("Application error")
    }

    "run in a server" in {
      running(TestServer(3333)) {

        await(WS.url("http://localhost:3333").get()).status must equalTo(OK)

      }
    }
  }
}

Migrating the Database Models – problems!

Now for the bad news… our database models were making heavy use of the Magic[T] class and that simply isn’t present in Play 2.0 at the moment (Apparently it featured in a beta version but was removed). What Magic[T] did was generate a lot of the code for the CRUD operations from a simple case class. For reference in the old documentation for the play-scala-0.9.1, this is what Magic did.

Whilst I could write all of this code myself, I have about 40 classes to do this for, so this is a non-trivial amount of work.

I also discovered that the DB evolution hashes are different, so play wanted to revert and re-apply all of my DB schema changes, which would have trashed the DB contents. Apparently this is now fixed in the trunk in Play 2 so hopefully that change will be in a release soon.

Because I don’t have the time to rewrite all the DB classes, I decided to split my application into two parts, as there are a number of very independent reports in the site which do not use the DB. The reports will be in a play 2 site and the older code that uses the DB will stay as a Play 1.2 site.

In the future I would like to port everything to Play 2. I’m hoping that someone will reintroduce Magic[T], either into the core of Play 2 or as an additional bit of code / plugin. Alternatively I’ll have to do all the grunt work myself, which would be a shame. It’s the kind of work I like to avoid simply because there is very little value I can add – it will either keep working or (more likely) I will break something! So the best option is to leave this old ‘legacy’ site as play 1.2 until there is an easier/safer upgrade path.

However, I will definitely be using Play 2.0 for new development!

Google and HTML5 – Simplicity, Security and Speed

HTML5

HTML5 has been around for a while, but only since recent times however, has it emerged as the major web technology of choice moving forward. Browsers are catching up, increasingly adding HTML5 feature support and there is an emergence of tools offering near-comprehensive compatibility with older browsers and platforms. Not to mention the massive support and investment from tech powerhouses such as Google and Apple, HTML5 really is fast becoming the defacto standard for building powerful, cross-compatible web apps.

This new age of HTML5 powered web apps means we can recreate a seamless native app experience from within the browser. However this time, there’s no flash, no plugins, nor any downloads. Instead, with the help of new HTML5 features such as drag and drop, local storage, multimedia (canvas, svg, video and audio), geolocation, offline and real time communication (webRTC), along with CSS3 and various Javascript API’s, we now have one technology stack that is usable across multiple platforms and devices.

Google

Google have pioneered in this area for a while, having led the way with a plethora of powerful, game-changing, web apps such as Docs and Sites to name but a few. They recently invited a group of geeks/web developers to their London offices in Victoria for a talk about HTML5 and some of the things they’ve been working on. I was part of this group.

The offices are pretty cool. From the famous Google ‘don’t be evil’ posters to the life-size London red bus to the fully catered kitchen and astro-turfed meeting room. It all supported the fun, creative and inspiring vibe floating around the place and this is what Google is about from a cultural point of view. What Google are all about technologically, can be loosely categorised into 3 areas: simplicity, security and speed. These are the primary focuses for the Chrome OS and also for their user-centric designs.

Simplicity

New breed of Web apps: One technology stack that is interoperable between multiple devices and platforms without the need for downloads or cross platform and OS adaptations. Add powerful features such as offline use, canvas and touch event API’s, and the categorisation between a website and web app becomes more converged. FT and Kindle are good examples of this new breed of HTML5 web app.

Chromeframe:  There are a lot of cool new HTML5 features which aren’t compatible with older browsers such as Internet Explorer 6 & 7 and this can be costly to support for developers. One option to reduce this headache is to prompt users to install Google Chromeframe which enables Chrome to work inside Internet Explorer. Instant compatibility.

WebIntents: Web Intents make it easier to share data and web services across apps. E.g. ‘share a link with Twitter’ or ‘edit a photo with image editor before sending’. Instead of integrating each and every one of these web services with your app, the user is empowered with their own choice. The idea is that your service registers an intent to handle an action such as sharing or editing, and the system will learn to find the appropriate services for the user to use based on the user’s preferences. Less integrating, more sharing.

Security

Content Security Policy (CSP): CSP tackles malicious cross site scripting by allowing web pages to limit where external content can be loaded from. Additionally, inline javascript and CSS are not allowed. As well as providing a good security measure, this also helps maintain good semantics.

3Dtin: This is an entirely web based 3D modelling app where anyone can jump in and start creating 3D models. If extra functionality is required, it offers an openID signup using Google Identity Toolkit which adds a layer of security onto the login part of the process. This ‘try now, sign in after’ approach enhances the user experience by allowing users to first engage, then commit after.

Browsers: A good browser makes all the difference. Chrome pushes updates automatically which ensures that the user always has the latest version with all the latest security patches. It also offers comprehensive sandboxing and plugin security for extra layers of protection.

Speed

Offline: HTML5 provides a better way of implementing offline availability and local storage. Previously this was done using cookies and offline emulation (gears). Now, it can be done using application caching and HTML5 storage. Application Cache is like a beefed up implementation of the browser cache, storing references to all required resources in a cache manifest file. HTML5 storage includes API’s such as Web Storage and IndexedDB, which store user state (such as state of play in a game) on the user’s local database for later use.

Frameworks/Templates: The idea of using frameworks and templates isn’t new, nor are they a requisite for building HTML5 web apps, but the ‘do more with less code’ benefits are certainly worth a mention. For Javascript, popular MVC frameworks include Backbone.js and Javascript MVC. SASS and LESS are some of the more well known CSS frameworks. Underscore and Mustache are great for templating. If you want to start building an HTML5 website from a template, then HTML5 Boilerplate is the daddy.

Bleeding Edge Technologies: This is what the Google developers refer to when talking about work on more-cutting-than-cutting-edge technologies (i.e. not well supported!) These include:

  • pageVisibility API: This determines if your web app is actively in view of the user. If not (i.e. user is on another tab), then the non-active pages are paused, hence optimising performance.
  • prerendering API: This predicts where the user will navigate next and pre-loads the page so that it appears to load instantly when it is clicked.
  • Fullscreen API: Fullscreen without the need for flash. All individual DOM elements are styleable which leads to lots more control and customisation.
  • Gamepad API: With powerful multimedia capabilities such as canvas and SVG gaining momentum, games running in the browser are only going to get more sophisticated. This API maps gamepad button events and empowers the user with a more traditional gaming experience.
  • webRTC: This is real time communication through browser alone i.e. plugin-free. It’s based on p2p and a handshake through to the server, and uses the getUserMedia API to talk to the user’s camera/microphone. This has the possibility to create much richer web-browser social video based experiences than was possible before.
  • Speech Input API: With the ability to transcribe voice into text, this will be a game-changer… once the voice recognition technology improves. An extremely exciting prospect for the near future.
  • Video subititles: Using the <track> tag and a linked .vtt file, video subtitles can be implemented. Great for accessibility, with potential for timed metadata and deep-linking into exact points on a video. Perhaps, on-the-fly localised subtitles using multiple .vtt files and the speech input API in the near future?

Conclusion

Technology moves fast. In my time at Google, buggy demos were shown, Chrome extensions were written on the fly and new features were released that the top developers didn’t even know about. The motto of the day seemed to be ‘There are a few bugs, but it works’ and ‘That’s a bug, I’ll fix it’. It’s what happens when you’re at the front of a new technology. But it’s through innovation and fast-paced iterations in which cool things develop…

So it’s a wonder why Microsoft have been so slow in providing HTML5 support for Internet Explorer. With its large market share, mainstream adoption of HTML5 has been massively held back. However, this is all set to change, with the imminent arrival of IE10. Less time will be spent dealing with cross platform and device issues, messing with legacy browser woes and hacking polyfiller shims together. More time will be spent innovating, and building richer, more sophisticated web apps that run through the browser alone. Exciting times to be a web developer! Developers are just scratching the surface of what is possible and as the HTML5 standard matures, as will our web apps. And if we adhere to the 3 S’s and acknowledge that users want things simple, secure and speedy, the future of the web is already in better shape. Exciting times ahead indeed!

Many thanks to Google and the team Sam Dutton, Paul Kinlan, Ido Green and Alex Russell

Useful Links

Testing Unity’s Flash Export on a Large Project – Part 1

Over the last weekend, I’ve been hard at work trying to get an unannounced Mind Candy project (made in Unity) to export to Flash. I thought it would be useful to share some details from the experience since most of the issues I’ve encountered would probably be avoidable if your project is architected in a way that lends itself to Flash export.

During the Christmas holidays, I made a game for Unity’s Flash in a Flash contest. It wasn’t the most exciting game, but it worked. The core mechanic of that game (including new 3.5 features such as nav mesh) exported to Flash well. The reason this game worked is because I had been paying close attention to the Flash export and knew what features wouldn’t work at that time. I avoided anything overly complicated and developed the game with the limitations in mind. Fundamentally, I decided to make a new, simple game rather than trying to port an existing one.

Now, I’m doing the opposite. I’m trying to get an existing game to publish to Flash. This project is a relatively large one. The game has been in development for quite a long time. It contains a lot of complex C# code and most importantly: a lot of features that don’t yet work in the Flash export. Trying to get this game to export to Flash is no easy task. I’ve spent numerous weekends on this since the 3.5 beta was made available and I still haven’t got it to work.

Despite the export (currently) not working for our project, there are a lot of lessons to be learned. Hopefully these will be of use to other people attempting the same task, and will be a good reference point for myself when I inevitably try to export the game again at a later date.

Currently Unsupported Features

Unity have already listed some of the unsupported features in the Flash export as part of the 3.5 preview faq. Some of these features (and the ones that have proved most problematic for me) are:

  • LINQ
  • Terrains
  • Asset Bundles
  • WWW
  • Raknet networking

If you’re using these features, then you’ll encounter a lot of errors as soon as you try to get Unity to build to Flash. Some example errors I’ve seen are:

Networking

  • error CS0246: The type or namespace name `ConnectionTesterStatus’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `NetworkView’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `BitStream’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `WWW’ could not be found. Are you missing a using directive or an assembly reference?

MovieTextures

  • error CS0246: The type or namespace name `MovieTexture’ could not be found. Are you missing a using directive or an assembly reference?

These errors are effectively a checklist of all the classes you’re using that aren’t yet supported and there’s only one thing you can do: remove them from your build. There are numerous ways to do this (depending on what you’re trying to achieve), from brute force deletion to telling Unity to skip these sections in a Flash build. You can do the latter by using platform dependant compilation.

All you need to do is wrap your Flash specific code in a platform check such as:

#if UNITY_FLASH
Debug.Log(&amp;quot;Flash build&amp;quot;);
#endif

In my case, the first thing I had to do was to try and remove these unsupported features. MovieTextures were easy to take out, as they’re not vital to our game. Networking, however was more problematic. And this is my first (and most important) lesson…

Lesson 1 – Separation of Networking Code

Our game currently uses the inbuilt RakNet networking solution. These networking elements are fundamental to our game, and as such, the networking code exists in many different areas of our codebase. When publishing to the web player or a standalone app/exe build this is fine. For Flash export, this suddenly creates a big problem when the networking solution isn’t yet supported.

As an example, if your game uses RPCs across clients to update core data in your game, then you’re going to have problems. I’m sure that there are other solutions which are better suited to Flash export, but this doesn’t fix my immediate problem: we have a game, where our chosen networking solution won’t publish to Flash. Unity suggest that you can use Flash networking instead of RakNet, but since I’m doing this export with tight time constraints (self imposed, by the mere fact it’s a weekend), that solution is not feasible for this test.

This has left me with one option in my mission to get our game working: rip out RakNet. This is not ideal, but luckily, our game copes with it ok.

This raises an interesting point in that the networking code should be as decoupled from the core mechanic of your game as possible. In some cases this can’t be done, but if you can find a way to make your networking layer easily removed/changed, then you’ll be in a much better place than I was regarding Flash export. It will also help you if you ever decide to switch to a different networking solution.

At this point, I’m going to gloss over about 10 other failed builds. It takes a few attempts at building to clear up this first wave of errors. Once you’ve cleared that first wave, you can breathe a sigh of relief and ready yourself for wave two: Attempting an actual build…

Attempting a Build

Once you’ve fixed/removed/hacked-out all the unsupported features, you’ll get to a point where the build process will now try to publish your game to Flash. The type of errors you get now will be more complex than those in wave one. Below is a screenshot of one of my build attempts at this stage:

You’ll note that these errors are more complicated than the “you can’t use ClassX because it’s unsupported” ones. In the case of these errors, it’s up to you to go into each of these classes and try to simplify your code as much as possible.

Some areas where our build failed were where we’d used generics. For example, we had fairly complex code to randomise the order of elements in an array. It wasn’t vital, so it went in the bin. This seems to be a common trend in trying to get this project to build to Flash. I’m slowly, over time, discarding features to the point where it’s a very stripped-down version of the game.

There are a couple of errors regarding our audio library in the above screenshot. This library wouldn’t convert at all (I got multiple waves of errors). My only solution at present has been to remove it.

The last item in that list is log4net. This caused a lot of issues. Rather than spending ages resolving them for this test, I decided it should also be removed. Since we used the logging in a lot of our code, I’ve ended up writing my own logging classes based on the log4net interfaces. This meant that I only had to fix up the imports in the class and our existing logging would still work using Unity’s own Debug.Log etc.

A few more iterations and build attempts occurred before wave 2 was complete. All in all, the first two waves have taken out large chunks of our features, and as a result the game feels somewhat unstable.

Akin to a game of [insert zombie survival FPS game of your choice here], we’ve just about survived the first few waves. We’re battererd, we’re bruised, but most importantly, we’re not defeated! We’re now ready for the next wave. Bring on the boss; the tank; the last major hurdle in the flash export – the conversion of your code to ActionScript.

Converting your code to ActionScript

At this stage, when you try to build, Unity will attempt to convert your source to ActionScript. Having previously spent years as a Flash developer, I find this part of the build rather exciting. The guys at Unity have done a fantastic job of getting this process to the stage it’s at.

That said, this is probably the toughest part of the process. Ripping out features and (to some extent) fixing the errors in the previous stage is easy. Trying to work out why the generated ActionScript doesn’t work is much more difficult. Luckily, when a build fails, you can find all the AS classes in a temp folder in your project (/Temp/StagingArea/Data/ConvertedDotNetCode/global/). This will enable you to look at them (if you wish) and try to understand where it might be going wrong, such that you can adjust your C# or js accordingly.

In my first attempt at this stage, I was left with 87 errors. The following are a small selection of these to give you an idea of the kind of problems I’ve seen:

Conversion Error 1

  • Error: Access of possibly undefined property $Type through a reference with static type Class.

This error seems to be very common and occurs when reflection is used (and probably in other situations). Unfortunately, a lot of our core libraries use reflection, and as such, this is a large problem to try and fix.

Conversion Error 2

  • Error: Call to a possibly undefined method IComparable$1_CompareTo_T through a reference with static type Number.

This has occurred because we’re trying to compare two values whose classes implement IComparable. In our case, this could be worked around relatively easily.

Conversion Error 3

  • Error: Type was not found or was not a compile-time constant: ReadOnlyCollection$1

In some of our classes we’re providing access to ReadOnlyCollections. It seems that we can’t use these at present and we could work round this by simply returning a standard Collection.

Conversion Error 4

  • Error: Call to a possibly undefined method String_Constructor_Char_Int32 through a reference with static type String.

A common style of conversion error that’s quite tricky to work out. I saw a lot of errors similar to this one.

These are just 4 of the 87 errors which need fixing. I expect that if/when all 87 are resolved, I’d have another wave or two to get through before the game would actually build. For now though, it’s Sunday night and I’ve run out of time to work on this test.

Next Steps…

My next challenge in this Flash export test is to go through the aforementioned 87 conversion errors and try to resolve them. I’m hoping that I’ll be able to get the game to build after another solid few days working on the export.

If that task proves too difficult then I will try a different approach of starting from a clean project and adding features one by one. In theory, that should be easier to get working, although that’s not how we’d want to export to Flash in the long run.

If I do get the export to work, I shall write a follow-up post with a walkthrough of some conversion errors. For these, I’ll include (where possible) the raw C#, the converted AS, and examples of how the errors can be avoided/solved.

For now though, I’m going to give up and play some well-earned Killing Floor! :D