ReaderT 101

This blog post is about dependency injection (d.i.) using the Reader monad in Scala. I won’t explain what a monad is nor will I explore any category theory (mostly because I don’t know how to explain any of that). In this post I just want to show the mental model I have when using monadic style with ReaderT.

Note: this post turned out to be quite big. It’s not very dense though! Especially if you’re familiar with Scala you should be able to whisk through most of it.

Dependency injection

Code needs other code. That’s what d.i. is for me. We write separate pieces of code. Often one bit needs to use the other. I’ll use the following example source for this post:

case class Hero(name: String)

// imagine this is a database
case class Ooo(importantSettings: Unit) {
  private[this] val finn = Hero("Finn")
  private[this] val jake = Hero("Jake")

  def findHero(name: String): Hero = {
    // Imagine all kinds of database processing here
    finn
  }

  def friendsRegistry(): Map[String, Hero] = {
    // Moar processing
    Map(finn.name -> jake)
  }

  def evalAdventure(hero1: Hero, hero2: Hero): String = {
    // Jake always saves the day, he's a magic dog!
    if (hero1 == jake || hero2 == jake) "awesome" else "disappointing"
  }
}

// The instance of Ooo we want to inject everywhere
val ooo = Ooo(()) 

// This is a piece of 'business' logic
object AdventureTime {
  def getHero(name: String): Hero = ooo.findHero(name)

  def getBestFriend(hero: Hero): Hero = ooo.friendsRegistry()(hero.name)

  def goOnAdventure(hero1: Hero, hero2: Hero): String = {
    val result = ooo.evalAdventure(hero1, hero2)
    s"Adventure time with ${hero1.name} and ${hero2.name} was $result!"
  }
}
Adventure Time - Land of Ooo

Adventure Time – Land of Ooo

Instead of a stuffy real-world example I’m using Adventure Time. Think of Ooo as a database repository and AdventureTime as some piece of business logic. I assume this code is relatively simple and understandable. The problem is this: how does AdventureTime get a reference to Ooo? In other words, we want to inject Ooo into AdventureTime and possibly other parts of the code.

First, an example of how one could have an adventure:

import AdventureTime._

val hero1 = getHero("Finn")
val hero2 = getBestFriend(hero1)
val result = goOnAdventure(hero1, hero2) 

// result -> "Adventure time with Finn and Jake was awesome!"

A global variable and/or the Singleton

The example above illustrates one of the easiest ways of doing this: use a global variable and refer to that. This works great for small programs but when your program gets a bit larger, or your codebase is a bit older, this becomes very painful. Globals are difficult to maintain, they’re not very flexible, and they make code difficult to unit-test. You can also see in the example that the dependency is kind of hidden.

DI frameworks

Thankfully the industry has moved on from globals (right?) and frameworks like Spring and Guice have been invented to help. I won’t go into details about how they work, but they’re usually similar to constructor injection.

Constructor injection

In OO languages we can use the constructor of an object to provide it with the needed dependency. The AdventureTime object is now a class.

class AdventureTime(ooo: Ooo) {
  def getHero(name: String): Hero = ooo.findHero(name)

  def getBestFriend(hero: Hero): Hero = ooo.friendsRegistry()(hero.name)

  def goOnAdventure(hero1: Hero, hero2: Hero): String = {
    val result = ooo.evalAdventure(hero1, hero2)
    s"Adventure time with ${hero1.name} and ${hero2.name} was $result!"
  }
}

val at = new AdventureTime(ooo)

val hero1 = at.getHero("Finn")
val hero2 = at.getBestFriend(hero1)
val result = at.goOnAdventure(hero1, hero2)

This is a bit better than using global variables. Note that we still need some way to actually get ooo to where we create our at object, but in this post I want to focus on where the dependency is used. You can see that AdventureTime now has an explicit dependency on Ooo.

One caveat of this approach is that your class file should not become too large, otherwise you’re basically back to using a global variable! Constructor injection is not bad, it’s been used to create large systems. It’s fairly flexible, although you usually can’t change the dependency after it’s set. In order to test this you’d need to create a mock implementation or use a mocking library to mock the dependency.

What we actually want

We actually would like to pass the dependency as a parameter to every function that might need it.

object AdventureTime {
  def getHero(ooo: Ooo, name: String): Hero = ooo.findHero(name)

  def getBestFriend(ooo: Ooo, hero: Hero): Hero = {
    ooo.friendsRegistry()(hero.name)
  }

  def goOnAdventure(ooo: Ooo, hero1: Hero, hero2: Hero): Unit = {
    val result = ooo.evalAdventure(hero1, hero2)
    s"Adventure time with ${hero1.name} and ${hero2.name} was $result!"
  }
}
import AdventureTime._
val ooo = Ooo(())

val hero1 = getHero(ooo, "Finn")
val hero2 = getBestFriend(ooo, hero1)
val result = goOnAdventure(ooo, hero1, hero2)

This is a very flexible approach, we could change the dependency with each function call. We don’t need an instance variable to hold the dependency which makes this approach very suitable for, well, functions. We obviously see a pattern in these functions, but we can’t really abstract over it to remove the repetition.

Monads

Let’s see how we can use some functional programming and the Reader monad to improve this. Before we do that though, let’s quickly refresh how monads work. We use an all time favourite, the Option monad. Feel free to skip this explanation if you’re familiar with it.

The example code is actually not very null-safe.

val hero1 = getHero(ooo) // <- hero1 could be null
// which would probably make getBestFriend throw an NPE
val hero2 = getBestFriend(ooo, hero1)
// hero2 can also be null...
val result = goOnAdventure(ooo, hero1, hero2)

One way to handle this would be something like:

val hero1 = getHero(ooo, "Finn")
if (hero1 != null) {
  val hero2 = getBestFriend(ooo, hero1)
  if (hero2 != null) {
    val result = goOnAdventure(ooo, hero1, hero2)
  } else {
    println("No adventure today")
  }
} else {
  println("No adventure today")
}

This kind of clutters up things and distracts from what the code is actually trying to do. The Option monad represents the possibility that something can be null. We can encode this optional behaviour into the types. The monad then let’s us concentrate on the actual happy-path of the code while handling the boiler-plate around null-checking for us.

case class Ooo(importantSettings: Unit) {

  // It's possible the hero can't be found, so it's optional
  def findHero(name: String): Option[Hero] = {
    Some(finn)
  }

  def friendsRegistry(): Map[String, Hero] = {/* same as before */}

  def evalAdventure(hero1: Hero, hero2: Hero): String = {
    /* same as before */
  }
}

object AdventureTime {
  // Another Option here.
  def getHero(ooo: Ooo, name: String): Option[Hero] = ooo.findHero(name)

  // Yet another one. Types tend to ripple through a codebase
  def getBestFriend(ooo: Ooo, hero: Hero): Option[Hero] = {
    ooo.friendsRegistry().get(hero.name)
  }

  def goOnAdventure(ooo: Ooo, hero1: Hero, hero2: Hero): String = {
    /* same as before */
  }
}
import AdventureTime._
val ooo = Ooo(())

val result: Option[String] = for {
  hero1 <- getHero(ooo, "Finn")
  hero2 <- getBestFriend(ooo, hero1)
} yield goOnAdventure(ooo, hero1, hero2)

println(result.getOrElse("There was no adventure :("))

The Option monad does exactly what we want. If there are no nulls, everything works as before. If there is a null somewhere in the process, it kind of ‘sticks’. I.e., no subsequent code is executed and a None is returned. It’s not exactly ‘as before’, we’ve obviously switched to a for comprehension.

We’ve enhanced the return types of our functions to deal with a kind of ‘secondary’ logic so we can focus on the main functionality that we’d like to express. That sounds familiar. What if we could encode our dependency into the return type as well?

Enter the Reader

The Reader monad basically encodes a simple function. It’s type definition is:

type Reader[E, A] = ReaderT[Id, E, A]

Let’s forget the right hand side of that type alias for now. Reader just expresses a function that takes a parameter of type E and returns a value of type A. Think of it as:

def func(e: E): A = {
  // create some A using e
}
// or
val func = (e: E) => {
  new A(e.foo())
}

You see how we could use that to express a dependency. The first type parameter E stands for ‘environment’. In our code E is Ooo and A is whatever our functions return. E.g., an Option[Hero] or a String. The type signature of getHero would become def getHero(name: String): Reader[Ooo, Option[Hero]]. Read: “getHero is a function that returns a function. When the returned function is supplied an Ooo it will return an Option of Hero“.

Let’s add this to our example. Note that all the functions in AdventureTime have the same dependency, so we make a little type alias for it. I’m assuming the reader is familiar with the various ways of creating lambda functions in Scala.

// Warning: this is not the final example, don't write code like this!
type OooReader[X] = Reader[Ooo, X]
object AdventureTime {

  def getHero(name: String): OooReader[Option[Hero]] = Reader{
    (ooo: Ooo) => ooo.findHero(name)
  }

  def getBestFriend(hero: Hero): OooReader[Option[Hero]] = Reader{
    _.friendsRegistry().get(hero.name)
  }

  def goOnAdventure(h1: Hero, h2: Hero): OooReader[String] = Reader{
  (ooo: Ooo) =>
    val resultOfAdventure = ooo.evalAdventure(h1, h2)
    s"Adventure time with ${h1.name} and ${h2.name} was $resultOfAdventure!"
  }
}
import AdventureTime._

val res = for {
  hero1 <- getHero("Finn")
  hero2 <- getBestFriend(hero1.get) // .get !? ick...
  result <- goOnAdventure(hero1.get, hero2.get)
} yield result

This looks similar to before, but we’ve managed to remove all the ooo parameters. Hang on, where are we injecting ooo now? Well, we’re not. This code seems to not do anything. If you inspect the type of res you’ll see it’s scalaz.Kleisli[scalaz.Id.Id,Ooo,String]. 😱

Remember that getHero returns an OooReader, i.e., a function taking an Ooo and returning an Option[Hero]. getBestFriend actually has the same signature. Just like Option, using Reader in a for comprehension sequences the monads into a ‘bigger’ one. For Option this means combining potentially absent values. For Reader it just means: “keep passing the dependency to the next function”. We’ve basically combined all three function calls into one big Reader.

If we want to execute the code we need to supply it with an Ooo using the run function of Reader.

res.run(Ooo(()))
// --> scalaz.Id.Id[String] = Adventure time with Finn and Jake was awesome!
Monad Transformer

Monad Transformer

We’ve run into a problem though. We had to resort to the evil get function for unwrapping our Options. So the Reader basically undid all the Option monad goodness. Ideally the code should handle both monads at once. Fortunately there is a monad transformer for Reader called ReaderT.

What was that weird type signature and what is this Id stuff? Remember the right hand side of the Reader type alias? It was ReaderT[Id, E, A]. It turns out that instead of working with functions of type E => A, we usually work with functions like E => M[A], where M is some kind of monad. ReaderT expresses just that. Reader is actually an alias for ReaderT where M is the Id monad. I see Id as the ‘does nothing’ monad.
ReaderT looks like this:

type ReaderT[F[_], E, A] = Kleisli[F, E, A]

What? Another type alias? Yes, ReaderT is actually equivalent to Kleisli, which is what scalaz uses. Kleisli also adds many convenience functions for combining Kleislis.

Let’s rewrite our example using Kleisli instead:

object AdventureTime {
  // Kleisli[Option, Ooo, Hero] 'represents' Ooo => Option[Hero]
  def getHero(name: String) = kleisli[Option, Ooo, Hero](_.findHero(name))

  def getBestFriend(hero: Hero) = kleisli[Option, Ooo, Hero]{
    _.friendsRegistry().get(hero.name)
  }

  def goOnAdventure(h1: Hero, h2: Hero) = kleisli[Option, Ooo, String]{
  (ooo: Ooo) => 
    val resultOfAdventure = ooo.evalAdventure(h1, h2)
    Some(s"Adventure time with ${h1.name} and ${h2.name} " +
         s"was $resultOfAdventure!")
  }
}
import AdventureTime._

val res = for {
  hero1 <- getHero("Finn")
  hero2 <- getBestFriend(hero1)
  result <- goOnAdventure(hero1, hero2)
} yield result

res.run(Ooo(()))

Before we had Reader just wrapping a function that matches the desired type. There is no such constructor for ReaderT, probably just because kleisli already does exactly the same. In other words, one can create a ReaderT using the kleisli function. The type parameters in order are: the monad of the return value, the environment of the function, and the type of the return value.

The Future

This all looks nice but we might not be convinced yet. Sit tight, I’ll show you a great advantage of using Reader. We’ll have to go even more functional though.

Our for comprehension should belong in some function in the logic layer of our program. We’ve abstracted the dependency on Ooo through the Reader but the sample code still strongly couples to AdventureTime. Let’s remove that by passing the necessary functions as parameters instead!

object SomeFancyLogic {
  def startEpicAdventure(
    getHero: (String) => ReaderT[Option, Ooo, Hero],
    getBestFriend: (Hero) => ReaderT[Option, Ooo, Hero],
    goOnAdventure: (Hero, Hero) => ReaderT[Option, Ooo, String])
   (name: String): ReaderT[Option, Ooo, String] = {
    for {
      hero1 <- getHero(name)
      hero2 <- getBestFriend(hero1)
      result <- goOnAdventure(hero1, hero2)
    } yield result
  }
}

// We usually 'wire up' the parameter group containing the
// functions first
val startEpicAdventureWired = SomeFancyLogic.startEpicAdventure(
                                          AdventureTime.getHero _,
                                          AdventureTime.getBestFriend _,
                                          AdventureTime.goOnAdventure _) _

startEpicAdventureWired("Finn").run(Ooo(()))

Let’s also make our ‘database’ a bit more realistic. In the server world we like to avoid blocking, so APIs for external services usually return Futures.

// The land of Ooo of the future
case class Ooo(importantSettings: Unit) {

  // findHero now returns a Future
  // for simplicity I'm ignoring the Option stuff.
  def findHero(name: String): Future[Hero] = {
    Future.successful(finn) // again, just simulating here..
  }

  def friendsRegistry(): Future[Map[String, Hero]] = {
    Future.successful(Map(finn.name -> jake))
  }
  
  def evalAdventure(hero1: Hero, hero2: Hero): Future[String] = {
    Future.successful{
      if (hero1 == jake || hero2 == jake) "awesome" else "disappointing"
    }
  }
}

// The rest of the code stays almost the same!
// Just change the Monad type parameter from Option to Future

object AdventureTime {
  def getHero(name: String) = kleisli[Future, Ooo, Hero](_.findHero(name))

  def getBestFriend(hero: Hero) = kleisli[Future, Ooo, Hero]{
    _.friendsRegistry().map(_(hero.name))
  }

  def goOnAdventure(h1: Hero, h2: Hero) = kleisli[Future, Ooo, String]{
  (ooo: Ooo) =>
    ooo.evalAdventure(h1, h2).map{result =>
      s"Adventure time with ${h1.name} and ${h2.name} was $result!"
    }
  }
}

object SomeFancyLogic {
  def startEpicAdventure(
    getHero: (String) => ReaderT[Future, Ooo, Hero],
    getBestFriend: (Hero) => ReaderT[Future, Ooo, Hero],
    goOnAdventure: (Hero, Hero) => ReaderT[Future, Ooo, String]
  )(name: String): ReaderT[Future, Ooo, String] = {
    for {
      hero1 <- getHero(name)
      hero2 <- getBestFriend(hero1)
      result <- goOnAdventure(hero1, hero2)
    } yield result
  }
}

/* wiring as before, snipped for brevity o_O */

val future = startEpicAdventureWired("Finn").run(Ooo(()))
Await.result(future, 2.seconds)

A pattern is emerging here! We can actually abstract out the monad! We can also abstract away the dependency on Ooo. It looks like this:

object SomeFancyLogic {
  def startEpicAdventure[M[_]: Monad, E](
    getHero: (String) => ReaderT[M, E, Hero],
    getBestFriend: (Hero) => ReaderT[M, E, Hero],
    goOnAdventure: (Hero, Hero) => ReaderT[M, E, String]
  )(name: String): ReaderT[M, E, String] = {
    for {
      hero1 <- getHero(name)
      hero2 <- getBestFriend(hero1)
      result <- goOnAdventure(hero1, hero2)
    } yield result
  }
}

E is now the generic type for the dependency. M[_] is a type that is actually a type constructor. Look at it as a type with a hole that needs another type to be whole. E.g., Option[String] or Future[Hero]. We also specify that there needs to be an implementation for the Monad type class for M.

The cherry on top

Wildberry is not a cherry but she is pretty.

Wildberry is not a cherry but she is pretty.

Testing this piece of logic now becomes pretty easy. Of course the logic is really simple here.

A unit test should only test the code-under-test. With our new function parameters this means we can easily instruct our test without using any mock libraries. We test Popjam using ScalaCheck to do extensive property based testing. Also note that while the database is using Futures, we don’t actually want to test the asynchronous behaviour of the code, just the logic. Moreover, creating tests with concurrency in them usually leads to brittle time-dependent tests.

Here’s how we could test our logic:

def testEpicAdventure() = {
  // our 'mocked' functions. Usually we would make them return
  // more useful results obviously
  val getHero = (name: String) => kleisli[Id, Unit, Hero]{
    _ => Hero(name)
  }
  val getBestFriend = (h: Hero) => kleisli[Id, Unit, Hero]{
    _ => Hero("Jake")
  }
  val goOnAdventure = (h1: Hero, h2: Hero) => kleisli[Id, Unit, String]{
    _ => "Test adventure"
  }
  
  val wired = startEpicAdventure(getHero, getBestFriend, goOnAdventure) _
  val result = wired("Finn").run(())

  result aka "how did the adventure test go" should equal("Test adventure")
}

We can just use Id for our monad and Unit for the database. I’ve found this way of testing to be a lot more fun than setting up complicated mock, stub, or spy objects.

There are a lot more things we can do with scalaz and ReaderT. Like MonadReader ask for instance. I encourage you to go on that adventure yourself!

Using giter8 to build Scala REST projects with Unfiltered, Netty and Gatling load testing

At Mind Candy we have a number of different Scala REST services to provide common aspects to our games, for example authentication and moderation.

A standard we have developed is to use an Unfiltered Netty standalone server wrapped with the apache-commons daemon library. Also, we want our services to be well tested, easily deployable, and stable under heavy load. For ease of deployment we use Fabric scripts, and for load testing we use Gatling to hammer the services under different stress-tests to check they are up to standard.

We have quite a few services that have a similar setup, and it was getting tedious to create and configure a new project. So, we created a giter8 script to do it all for us!

Giter8

Giter8 is an awesome command line tool which can create a template project structure for you with example files and sbt configuration.

There are quite a few templates contributed by the community already but none did exactly what we want, so we created our own which we’d like to share.

The template will create an example sbt project that:

  • Uses unfiltered and netty to give you a very simple starting endpoint service which will output ‘Hello, <project>’ when hit
  • Gives you some basic test stubs using Specs2
  • Is setup with gatling load testing with an example scenario
  • Is configured with the sbt-idea plugin (IntelliJ IDEA is our preferred IDE)
  • Is configured with the sbt-assemblysbt-release, and sbt-dependency-graph plugins
  • Has a simple fabric deploy file, unix start/stop scripts, and an example init script

The generated project is structured with 3 sub projects:

  • project-example
    • example-core // Business logic
    • example-resources // Containing the Unfiltered Netty request plan
    • example-server // To control the server hosting the plan

Usage:

First make sure you have setup giter8 using the instructions here. Then fire open a terminal and run:

g8 mindcandy/unfiltered-rest-gatling

There are some options to enter, but most can be left as default (See here for more info). The important ones are:

name => The main projects name
project => The name prepended to each of the sub projects
organization => Used as the basis for packages

Once you’ve entered the properties, giter8 will do its magic and whip you up a full sbt project. After it completes, you can cd to the directory, and run sbt.

Test it:

sbt test

Run it:

sbt run

Now open up your favourite browser and go to http://localhost:8889/<project> where <project> is whatever you specified above, and you should get a “hello, project” message back.

Load test it:

Start the server running in one terminal. In another terminal do:

cd yourproject
sbt
project <project>-server
test-load:run

You should get presented with a choice of different classes to run:

  • Engine → Executes the gatling Engine in interactive mode prompting you which scenario to run
  • BatchEngine → Runs all simulations available in a batch with no user prompts
  • Recorder → Starts the pretty cool Gatling recorder which you can use to create scenarios. Basically, you setup your web browser to use the recorder’s proxy, and then just browse as normal on your webpage. Gatling records all of your actions as a scenario that can be replayed and customised.
Hit the option for BatchEngine, then load the results in your browser to see something like:

Results from Gatling are put in the gatling/results directory. In the example scenario in the giter8 skeleton we simulate 10 users hitting the simple endpoint, ramping up over a 3s duration. The results are rendered in a nice html page with graphs showing exactly what happened during the simulation. The full gatling feature set is quite extensive and worth checking out.

Create IntelliJ IDEA project files:

sbt gen-idea

Run from start script:

First setup jsvc. Then :

sbt assembly
cd bin
chmod +x start.sh
./start.sh

This has the same affect as ‘sbt run’, but using the apache-commons daemon wrapper. Try opening the service in your browser again and it should work as before. Don’t forget to stop the server when you’re done with the ../stop.sh script :)

Other pieces:

There is also a sample init script that is configured for the skeleton in the bin folder, which is a good starting point for creating a complete init script.

Last to mention is there is a small fabric file that provides a (very) basic setup for copying over the built assembly onto a test/integration server. You’d need to configure your host and change the paths as appropriate to use it, but it’s useful as a starting point.

Let us know if you find this useful, and thanks to n8han for providing a great tool!

Migrating a Play 1.2 website to Play 2.0

At Mind Candy there are a number of internal websites used for reporting and communication. For example – reading automated build/test status via some REST APIs and turning this into a nice visual status display for large ‘build screens’. These have been authored in Scala using the Play Framework 1.2.x.

Recently, version 2.0 of the Play Framework came out of beta and so I wanted to convert the existing Play 1.2 websites over to it. Amongst other reasons, the ability to build to a .jar file makes for simpler deployment, which I was keen to have.

I couldn’t find a good guide for migrating from 1.x -> 2.0 so I am sharing my experiences of porting a Scala Play 1.2.x application. I expected that it wouldn’t be too hard as I was already using Scala Templates and controllers. That was mostly true, with a couple of issues.

I did find a handy conversion guide on the Play 2 wiki which can help converting some uses (it has some unfortunate formatting though so I’d recommend copying and pasting it)

Starting the project

I’d previously installed Play via brew so an upgrade was as simple as
brew update
brew upgrade play

This installed Play 2 whilst leaving Play 1.2.4 also installed for easy roll-back for developing older stuff if I need it.

For simplicity and because there were various code changes that I knew I needed to make, I created a new site in a new location using play new. However I then used svn copy to copy over source assets and code before modifying. This means in the future I can pull updates down using “svn merge” as work continued on the old site whilst I was porting it over!

All of the public files which aren’t built – javascript, css, images – were simply copied across. In the future I’ll check Play’s support for LESS CSS and the Google Closure JS compiler but for now I just want to get things working.

Initially I copied across all the controllers, models and views and configuration files, though I expected to have to fix those up as the syntax had changed.

After a quick run of play compile I had a whole host of build errors, unsurprisingly. So to cut these down I commented out a lot of the logic – all the routes except the home page /, all the templates and all the controllers except for those needed by the home page.

I fixed up controller in turn and gradually re-enabled the routes, until I hit an issue with Database models that stopped me from migrating the rest of my application (see below)

Migrating dependencies

There were a number of java libraries used by the web app, for example to talk to a Subversion server. (This is one of the reasons I really like Scala – it’s easy to pull in useful Java libraries) In Play 1 that was defined in dependencies.yml, e.g.:

require:
- org.tmatesoft.svnkit -> svnkit 1.3.7
- org.tmatesoft.svnkit -> svnkit-javahl16 1.3.7

repositories:
- tmatesoft:
  type: iBiblio
  root: "http://maven.studio:8081/nexus/content/repositories/tmatesoft"
  contains:
  - org.tmatesoft.svnkit -> *</code>

Play 2 now uses sbt as its build system and thus uses the standard sbt syntax for dependencies, so I edited the the project/Build.scala file to add dependencies:

object ApplicationBuild extends Build {
  // some definitions omitted

  val appDependencies = Seq(
    "org.tmatesoft.svnkit" % "svnkit" % "1.7.4",
    "org.tmatesoft.svnkit" % "svnkit-javahl16" % "1.7.4",
  )

  val main = PlayProject(appName, appVersion, appDependencies, mainLang = SCALA).settings(
    resolvers += "TmateSoft Repository" at "http://maven.tmatesoft.com/content/repositories/releases/",
  )
}

Once the sbt syntax is understood it’s a reasonably simple process to convert dependencies across.

The original play run / test / start / stop command still work, which is helpful. It’s also quite useful to simply run play to get an sbt console and then run ~test which runs test and then re-runs them anytime a source file changes, which is very useful.

For IDE integration, I use IntelliJ IDEA and happily there is an idea command in Play’s SBT which generates an IntelliJ module which has any library dependencies included. However, your code has to build correctly before this works and you probably need to remember to re-run this every time you modify a dependency. If you don’t the IDE won’t show types correctly – typically you will get more ‘false negatives’ of errors showing up in the IDE which will not happen within Play itself.

Migrating the routes file

For some reason the old Routes file had an explicit 404 for the for the favicon. There didn’t seem to be an obvious pure route-based replacement for this and I didn’t need that behaviour so removed the route.

I had to add controllers. to all routes as the full package reference seems to be required now.
e.g. home page changed from
GET / Application.index
to
GET / controllers.Application.index

The syntax for ‘public’ files has changed so I replaced
GET /public/ staticDir:public
with
GET /public/*file controllers.Assets.at(path="/public", file)

I was also serving a single file elsewhere – multiple Assets.at() in the routes file didn’t go well, but I was able to call to add a new method to the Application e.g. ‘serveMyFile’ and then set the routing to be controller.Application.serveMyFile in routes and then have that call the Assets.at() method
e.g.
GET /myfile controllers.Application.serveMyFile
in the Application controller:
def serveMyFile = controllers.Assets.at(path="/path/to/it", file="myfilename.json")

The syntax for parameters in routes has changed and I found that I needed to move the default parameter definitions from the Controller out to the Routes file, using the ?= syntax instead of the Option[T] syntax I had before. That wasn’t hard to do but was slightly irritating in some ways – I’m still not sure if I like having so it in the routes file.

Minor changes to the syntax for parsing out ids from the URL, for example:
GET /versionone/stories/{projectId} versionone.Printer.storiesById
changed to
GET /versionone/stories/:projectId controllers.versionone.Printer.storiesById(projectId: String, timeboxId: String ?= "", template: String ?= "")

Note the additional default parameters being passed. Also in this instance the Id is actually a string, not an Int.

Migrating Controllers

A lot of the package names have changed e.g. play -> play.api so some of the imports needed to be fixed up with the new ‘api’

replaced
import play._
import play.mvc._

with
import play.api._
import play.api.mvc._

With the paths to generated views also changed, I removed lines like import views.Application._

Then with imports fixed up I could then fix up the actual controller code to return an Action and
adding Ok(views.html. … )
e.g. from
def index = {
html.index()
}

to
def index = Action {
Ok(views.html.Application.index())
}

As I mentioned earlier, some changes to controllers were needed because the routes file now supplies default parameters, and I took out some use of the Option[T] type — though as this was generally a String, the empty string works very well as an equivalent to None.

In some controllers I’ve implemented an OAuth callback to read from a Google Calendar – eventually I’d like to play with the built-in OAuth support in Play 2 but in the interest of getting things working quickly I wanted to port over my current code. In order to send a url for Google auth to use as a callback I am calling Router.getFullUrl() but this doesn’t work in Play 2, instead one calls the automatically generated route classes instead. To turn this into a full url you call .absoluteUrl(), but you must remember to add the implicit request to your controller code in order to give this the context needed to generate the url.
e.g. the original code was approximately:

object MyController extends Controller {
  def request = {
    val callbackUrl = Router.getFullUrl(“MyController.callback”)
    // send web request using callbackUrl
  }

  def callback = {
    // handle request
  }
}

the new Play 2 code is:

object MyController extends Controller {
  def request = Action { implicit request =>
    val callbackUrl = routes.MyController.callback().absoluteURL()
    // send web request using callbackUrl
  }

  def callback = Action {
    // handle request
  }
}

This is more robust because that reverse-lookup is now checked at compile time, rather than runtime. However if your code doesn’t build, your IDE will give you errors as the reverse routes don’t exist. I did occasionally have to do a clean build to force regeneration of reverse-lookups, if the code structure had changed significantly.

Migrating Views

I had to replace @asset() with @routes.Assets.at()

Old references to actions can now be replaced with the compile-time checked generated routing objects. But the snag that isn’t documented is that if you are using a sub-project, the routes are generated within the scope of that sub-project.

e.g. replace @action(svn.Sizes.index()) with @controllers.svn.routes.Sizes.index()
This was mentioned on the Play discussion group here.

Migrating Tests

Unfortunately the bundled Play test framework has changed from scalatest to specs2. I could possibly have set up scalatest to work but decided to migrate the tests instead as the new functional tests look useful, as is the ‘property’ support in specs2.

A few imports change – I had to get rid of the scalatest imports and add in:

import org.specs2.mutable._
import play.api.test._
import play.api.test.Helpers._

The test class now simply extends Specification and I had to fiddle around with the syntax, replacing it with a noun and adding some more braces. should be(x) was replaced with mustEqual x

For example:

it should "retrieve svn server prefix" in {
  val svnServers = SvnServer.find().list()
  svnServers.length should be > (0)
}

becomes:

"SvnServer" should {
  "retrieve svn server prefix" in {
    val svnServers = SvnServer.find().list()
    svnServers.length must be_>(0)
  }
}

There is a good list of all the matchers in specs2 at the specs2 site which I found very useful.

I migrated some very basic selenium tests to the new functional test framework – documented here. The advantage of this is that you can separately test the controller, view and the whole thing together with the router.

The old selenium test was

#{selenium 'Home page'}
// Open the home page, and check that no error occured
open('/')
assertNotTitle('Application error')
#{/selenium}

So a simple test that the home page works is:

class ApplicationTest extends Specification {

  "The Application" should {
    "render template correctly " in {

      val html = views.html.Application.index()

      contentType(html) must equalTo("text/html")

      val content = contentAsString(html)
      content must contain ("Welcome to the Tools website")
      content must not contain ("Application error")
    }

    "have a working controller" in {
      val result = controllers.Application.index()(FakeRequest())

      status(result) must equalTo(OK)
      contentType(result) must beSome("text/html")
      charset(result) must beSome("utf-8")

      val content = contentAsString(result)
      content must contain ("Welcome to the Tools website")
      content must not contain ("Application error")
    }

    "run in a server" in {
      running(TestServer(3333)) {

        await(WS.url("http://localhost:3333").get()).status must equalTo(OK)

      }
    }
  }
}

Migrating the Database Models – problems!

Now for the bad news… our database models were making heavy use of the Magic[T] class and that simply isn’t present in Play 2.0 at the moment (Apparently it featured in a beta version but was removed). What Magic[T] did was generate a lot of the code for the CRUD operations from a simple case class. For reference in the old documentation for the play-scala-0.9.1, this is what Magic did.

Whilst I could write all of this code myself, I have about 40 classes to do this for, so this is a non-trivial amount of work.

I also discovered that the DB evolution hashes are different, so play wanted to revert and re-apply all of my DB schema changes, which would have trashed the DB contents. Apparently this is now fixed in the trunk in Play 2 so hopefully that change will be in a release soon.

Because I don’t have the time to rewrite all the DB classes, I decided to split my application into two parts, as there are a number of very independent reports in the site which do not use the DB. The reports will be in a play 2 site and the older code that uses the DB will stay as a Play 1.2 site.

In the future I would like to port everything to Play 2. I’m hoping that someone will reintroduce Magic[T], either into the core of Play 2 or as an additional bit of code / plugin. Alternatively I’ll have to do all the grunt work myself, which would be a shame. It’s the kind of work I like to avoid simply because there is very little value I can add – it will either keep working or (more likely) I will break something! So the best option is to leave this old ‘legacy’ site as play 1.2 until there is an easier/safer upgrade path.

However, I will definitely be using Play 2.0 for new development!

Scala at Mind Candy

Reading some recent negative commentary about Scala with interest I felt like it would be good to share our experiences with Scala.

The Good.

Scala is an expressive language – It often results in a lot less code getting in the way of what you want to do or achieve. The simple example for something like this would be a simple bit of code like this:

case class Person(id: Int, name: String)
def lookupPerson(id: Int): Person = {
  new Person(id, "Test" + id)
} // Token implementation for the example.
val ids = Seq(1,10,100)
ids.filter(id => id < 50).map(lookupPerson)

To write this in Java would require a whole load of boilerplate, which includes the case class generating the stock class methods like toString, multiple lines to create a collection and then transforming a collection.

More powerful ways to write tests – This can fall under the Spiderman grouping of power admittedly, but libraries like specs2 and scalacheck make it easier to create the kind of tests we wanted. Scalacheck is the stand-out example of this where a generator for creating Person objects from above is as easy as this:

object TestGenerators {
  val personGenerator = for {
    id <- arbitrary[Int]
    name <- alphaStr
  } yield new Person(id, name)
}

That’s all it takes and that object can be imported into all of your test code that needs to generate Person objects.

Less magic – A lot of libraries like Spring and Hibernate need to use byte code modification or reflection to achieve what they do, which means that when they fail it can be really hard to diagnose or fix the problems. We’ve even seen some of these do things non-deterministically, which has caused hours of bemusement. Contrary to this, Scala libraries just tend to use the type system to achieve these ends which means that in our case we catch problems at compile-time, not at run-time.

The REPL – The idea scratchpad and general utility will be your friend in times of need. Use this as an education tool to step through an idea with some else. Use it to test some code interactively if you want to confirm something but you’re not quite sure how to code it or what results you’ll get. Use it to solve those gnarly Project Euler problems without having to create a whole new build for each one.

SBT – Controversial one this may be, but it manages to give you the sensible build model and plugin system that Maven has while allowing you to easily create custom tasks. If nothing else being able to run a command, for example the ‘test’ task, on each save is the most useful thing I’ve seen in a while.

POWAH! – There’s an elegance that comes with time when using Scala, in much the same way that it does with a lot of languages, that means code slots together so cleanly and with little friction. For me personally the Option class was the beginning of this change in thinking, where I realised that representing the possible lack of something without using a null made a lot more sense.

The Bad.

SBT – It’s a double edged sword in you’ll need to understand a bit of Scala to be able to do non-trivial configuration in it. Documentation for this has improved massively in recent times, it can still be somewhat impenetrable, especially to someone new to Scala.

Somewhat idiomatic libraries – Databinder Dispatch is a good example of this, writing a custom handler to parse a HTTP response is just unnecessarily puzzling. As with all libraries how easy they are to use and extend should be evaluated, so don’t be blinded by those libraries just because they’re written in Scala. It’s better to pimp a stock Java library that already works well than to use one that is badly written in Scala.

Binary compatibility – This is the stock issue that is often complained about, fortunately SBT does notice when two versions of the same library that relate to two different Scala versions are pulled into the dependencies. The way others have presented this is as a major pain point, it’s generally only so much of an issue as it is with Maven dependencies with a little more granularity. Also if you’re using SBT it’s possible to create dependencies that tie to the Scala version used automatically.

Knowledge – There’s a couple of aspects to this. The first is that Scala is a “new” language and as such there is one learning curve which relates to the language, SBT, the libraries and how to use them all effectively. Beyond this is that some functional programming concepts are foreign to a lot of programmers and this can be a wall that isn’t scalable in a short period of time for a lot of people. Hopefully with time this will become less of an issue but at the moment there aren’t a lot of Scala developers that can hit the ground running.

The Ugly?

As with all new things, there is a learning curve with Scala, which can be problematic, but the benefit of the design is that it’s possible to do something the “wrong way” as the language is very flexible. People with a history in languages like Java can start out writing code that looks not that much different but still get benefits like better collections. Then with time progress onto using more the powerful features in the language like pattern matching and implicits. For the foreseeable future Scala is a tool we intend to keep using, as it’s been of great benefit to us (this week I parsed a 37GB log file with a couple of lines of code in the REPL), maybe you should too…