Google and HTML5 – Simplicity, Security and Speed

Posted on by
Reply

HTML5

HTML5 has been around for a while, but only since recent times however, has it emerged as the major web technology of choice moving forward. Browsers are catching up, increasingly adding HTML5 feature support and there is an emergence of tools offering near-comprehensive compatibility with older browsers and platforms. Not to mention the massive support and investment from tech powerhouses such as Google and Apple, HTML5 really is fast becoming the defacto standard for building powerful, cross-compatible web apps.

This new age of HTML5 powered web apps means we can recreate a seamless native app experience from within the browser. However this time, there’s no flash, no plugins, nor any downloads. Instead, with the help of new HTML5 features such as drag and drop, local storage, multimedia (canvas, svg, video and audio), geolocation, offline and real time communication (webRTC), along with CSS3 and various Javascript API’s, we now have one technology stack that is usable across multiple platforms and devices.

Google

Google have pioneered in this area for a while, having led the way with a plethora of powerful, game-changing, web apps such as Docs and Sites to name but a few. They recently invited a group of geeks/web developers to their London offices in Victoria for a talk about HTML5 and some of the things they’ve been working on. I was part of this group.

The offices are pretty cool. From the famous Google ‘don’t be evil’ posters to the life-size London red bus to the fully catered kitchen and astro-turfed meeting room. It all supported the fun, creative and inspiring vibe floating around the place and this is what Google is about from a cultural point of view. What Google are all about technologically, can be loosely categorised into 3 areas: simplicity, security and speed. These are the primary focuses for the Chrome OS and also for their user-centric designs.

Simplicity

New breed of Web apps: One technology stack that is interoperable between multiple devices and platforms without the need for downloads or cross platform and OS adaptations. Add powerful features such as offline use, canvas and touch event API’s, and the categorisation between a website and web app becomes more converged. FT and Kindle are good examples of this new breed of HTML5 web app.

Chromeframe:  There are a lot of cool new HTML5 features which aren’t compatible with older browsers such as Internet Explorer 6 & 7 and this can be costly to support for developers. One option to reduce this headache is to prompt users to install Google Chromeframe which enables Chrome to work inside Internet Explorer. Instant compatibility.

WebIntents: Web Intents make it easier to share data and web services across apps. E.g. ‘share a link with Twitter’ or ‘edit a photo with image editor before sending’. Instead of integrating each and every one of these web services with your app, the user is empowered with their own choice. The idea is that your service registers an intent to handle an action such as sharing or editing, and the system will learn to find the appropriate services for the user to use based on the user’s preferences. Less integrating, more sharing.

Security

Content Security Policy (CSP): CSP tackles malicious cross site scripting by allowing web pages to limit where external content can be loaded from. Additionally, inline javascript and CSS are not allowed. As well as providing a good security measure, this also helps maintain good semantics.

3Dtin: This is an entirely web based 3D modelling app where anyone can jump in and start creating 3D models. If extra functionality is required, it offers an openID signup using Google Identity Toolkit which adds a layer of security onto the login part of the process. This ‘try now, sign in after’ approach enhances the user experience by allowing users to first engage, then commit after.

Browsers: A good browser makes all the difference. Chrome pushes updates automatically which ensures that the user always has the latest version with all the latest security patches. It also offers comprehensive sandboxing and plugin security for extra layers of protection.

Speed

Offline: HTML5 provides a better way of implementing offline availability and local storage. Previously this was done using cookies and offline emulation (gears). Now, it can be done using application caching and HTML5 storage. Application Cache is like a beefed up implementation of the browser cache, storing references to all required resources in a cache manifest file. HTML5 storage includes API’s such as Web Storage and IndexedDB, which store user state (such as state of play in a game) on the user’s local database for later use.

Frameworks/Templates: The idea of using frameworks and templates isn’t new, nor are they a requisite for building HTML5 web apps, but the ‘do more with less code’ benefits are certainly worth a mention. For Javascript, popular MVC frameworks include Backbone.js and Javascript MVC. SASS and LESS are some of the more well known CSS frameworks. Underscore and Mustache are great for templating. If you want to start building an HTML5 website from a template, then HTML5 Boilerplate is the daddy.

Bleeding Edge Technologies: This is what the Google developers refer to when talking about work on more-cutting-than-cutting-edge technologies (i.e. not well supported!) These include:

  • pageVisibility API: This determines if your web app is actively in view of the user. If not (i.e. user is on another tab), then the non-active pages are paused, hence optimising performance.
  • prerendering API: This predicts where the user will navigate next and pre-loads the page so that it appears to load instantly when it is clicked.
  • Fullscreen API: Fullscreen without the need for flash. All individual DOM elements are styleable which leads to lots more control and customisation.
  • Gamepad API: With powerful multimedia capabilities such as canvas and SVG gaining momentum, games running in the browser are only going to get more sophisticated. This API maps gamepad button events and empowers the user with a more traditional gaming experience.
  • webRTC: This is real time communication through browser alone i.e. plugin-free. It’s based on p2p and a handshake through to the server, and uses the getUserMedia API to talk to the user’s camera/microphone. This has the possibility to create much richer web-browser social video based experiences than was possible before.
  • Speech Input API: With the ability to transcribe voice into text, this will be a game-changer… once the voice recognition technology improves. An extremely exciting prospect for the near future.
  • Video subititles: Using the <track> tag and a linked .vtt file, video subtitles can be implemented. Great for accessibility, with potential for timed metadata and deep-linking into exact points on a video. Perhaps, on-the-fly localised subtitles using multiple .vtt files and the speech input API in the near future?

Conclusion

Technology moves fast. In my time at Google, buggy demos were shown, Chrome extensions were written on the fly and new features were released that the top developers didn’t even know about. The motto of the day seemed to be ‘There are a few bugs, but it works’ and ‘That’s a bug, I’ll fix it’. It’s what happens when you’re at the front of a new technology. But it’s through innovation and fast-paced iterations in which cool things develop…

So it’s a wonder why Microsoft have been so slow in providing HTML5 support for Internet Explorer. With its large market share, mainstream adoption of HTML5 has been massively held back. However, this is all set to change, with the imminent arrival of IE10. Less time will be spent dealing with cross platform and device issues, messing with legacy browser woes and hacking polyfiller shims together. More time will be spent innovating, and building richer, more sophisticated web apps that run through the browser alone. Exciting times to be a web developer! Developers are just scratching the surface of what is possible and as the HTML5 standard matures, as will our web apps. And if we adhere to the 3 S’s and acknowledge that users want things simple, secure and speedy, the future of the web is already in better shape. Exciting times ahead indeed!

Many thanks to Google and the team Sam Dutton, Paul Kinlan, Ido Green and Alex Russell

Useful Links

Building testing tools against REST APIs with node.js and Coffeescript

Posted on by
2

During the dev cycle here at Mind Candy it is useful to have tools to help automate some of the more repetitive tasks, for example, registering a user. The great thing about test tools is that they are a great excuse to try out new technologies too! In this blog post, I’ll be telling you about a tool we wrote to register new users. It uses node.js, the non-blocking I/O server side JavaScript platform that is powered by Google’s V8 VM.

The reasons why I chose node.js for this project are as follows:

  1. It makes working with I/O operations an absolute breeze. Every I/O operation is required to be non-blocking, so its perfectly valid to keep a request open from a client such as a web browser while you make a bunch of (sometimes concurrent) calls off to REST APIs, databases, external systems etc, wait for them to call back to your code and then return a response. All of this happens without occupying a thread per request, unlike your standard java servlet. In fact, node.js is single threaded and uses an event loop, so as long as your expensive I/O operations are non blocking, the event loop just keeps ticking over and your system stays responsive.
  2. While node.js is a comparatively new technology, it has a HUGE and vibrant community behind it. There are so many 3rd party modules that if you need to interface with any other thing, there is most likely a module already written! Node also has an awesome package manager in npm, which makes declaring and downloading modules super easy.
  3. The holy nirvana – the same language running on the server and the client. Since the app is going to be deployed for the web, and therefore going to be powered by javascript, there are no problems with things such as serialising and deserialising objects between client and server (they all natively speak JSON), or different paradigms between languages giving you an impedance mismatch.

The whole project is actually written in a language called Coffescript. For those who haven’t heard of it, Coffeescript is a language that compiles down to javascript. It abstracts away some of the more grizzly parts of javascript, has a nice, clear and concise syntax and has a bunch of useful syntax features built in, such as loop comprehensions, conditional and multiple assignment and a simple class system. It’s like a bunch of best practices for javascript!

So let’s have a look at some of the code. For example, here is how we talk to the Adoption endpoint:

request = require('request')
xmlbuilder = require('xmlbuilder')

class Adoption
    constructor: (@host) ->

    start: (username, password, email, cb) ->
        adoption = xmlbuilder.create()

        adoption.begin('adoption')
            .ele('email').txt(email).up()
            .ele('password').txt(password).up()
            .ele('username').txt(username).up()

        adoptionXml = adoption.toString({pretty: true})

        request({
            method: 'POST',
            uri: "http://#{@host}/my/rest/service",
            body: adoptionXml,
            headers: {
                'Content-type': 'application/xml'
            }
        }, (err, resp, body) ->
            cb(err) if err

            if(body.search(/<error name="username" rule="is_not_unique"\/>/) > 0)
                usernameError = {
                    message: 'username is already taken'
                }

            cb(usernameError)
        )

module.exports = Adoption

First thing to note is indenting and whitespace is important! You MUST use spaces instead of tabs here, otherwise the compiler complains! Here we’re creating a ‘class’ called Adoption. Javascript doesn’t really have classes, but Coffeescript translates that into some javascript that gives you class-like behaviour. At line 5, we declare the class constructor function. In Coffeescript, functions are a very small declaration: just the function arguments in brackets and then an arrow. Anything after the arrow is the function body. The constructor is very simple, all it’s doing is setting the arguments of the function as member variables of that object. Looking at the javascript generated from the Coffeescript compiler illustrates this:

function Adoption(host) {
    this.host = host;
}

The start function (line 7) takes a bunch of parameters of data from the user and a callback function as the last argument. In node.js, if we are doing any async operation such as calling a REST endpoint, we cannot return the response data from that function since that would block the event loop. Instead, we are provided with a callback function which we can then call with the response once the server responds.

On line 10 we build up the xml payload to the Moshi Monsters REST endpoint, using a module called xmlbuilder. It would be a lot simpler if the endpoint accepted JSON! Next (line 17) we send the request to the endpoint itself. Here, we use the excellent request module. If you are familiar with how you perform ajax requests with jQuery, this should look quite familiar to you! Its another example of how node.js makes full stack development a lot less trouble for your developers as so many of the techniques used on the client side can be applied to your server side code.

The request function takes an object with the options for that request, and a callback that gets called upon error or success. The convention in node.js is to always expect the first argument to your callback function as a possible error, since if the async operation fails, you can check that parameter for the exception. On line 25 there is an example of Coffeescript’s postfix form of the if operator.

We then check for some response xml with a regular expression (line 27) and call the callback with the possible error object if the regex matches. Notice we do not have to declare variables before we use them. If we did this in raw javascript, they would end up becoming global variables, but Coffeescript handily declares them up front for us.
The last line is how we expose our class to the rest of the program. Node.js uses the CommonJS module system, so every file loaded is self contained as a module. We can expose our class by assigning it to the module.exports variable. This allows us to instantiate an Adoption object in another file:

Adoption = require('./path/to/adoption.coffee')

I used the brilliant express http server to serve this webapp. It has the concept of ‘middleware’ – effectively a bunch of functions that every request and response pass through. This means it is super easy to add functionality to express like caching, static file serving, and even asset pipelining as we will see later! We can set up a handler for our adoption request like so:

express = require('express')
Adoption = require('./adoption') #note that the file extension is optional!
app = express.createServer()

app.set('view engine', 'jade')

app.use(express.bodyParser())

adoption = new Adoption('www.moshimonsters.com')

app.post('/adopt', (req, res) ->
    username = req.body.username
    password = req.body.password
    email = req.body.email
    adoption.start(username, password, email, (err) ->
        if(err)
            res.json(err.message, 500)
        else
            res.send()
   )
)

app.listen(3000)

We’re using a template language called jade for the client side HTML markup. Jade is a simple, lightweight alternative to html. Here’s an example straight from their website:

doctype 5
html(lang="en")
  head
    title= pageTitle
    script(type='text/javascript')
      if (foo) {
         bar()
      }
  body
    h1 Jade - node template engine
    #container
      if youAreUsingJade
        p You are amazing
      else
        p Get on it!

Jade is nice and lightweight, but you can also do more hardcore stuff like scripting in the templates if you wish. It supports layouts and partials and all kinds of other nice stuff. Check out the website to learn more about it!

The client side javascript for this tool is written in Coffeescript too! But how does the browser understand it? The answer is that it doesn’t – we have to compile it first into javascript. You could do this as part of the build, but we have a better solution available to us.

There is a middleware module for express called connect-assets. This middleware adds asset pipelining to connect, so that you can write your code in Coffeescript and it will compile it on the fly and serve it to the browser, without you having to do anything! It can even minify the resulting javascript. You add it like this:

connectAssets = require('connect-assets')

...

# set build to true to minify the js
app.use(connectAssets({build: false}))

…and then we add a macro into our jade template:

doctype 5
html(lang="en")
  head
    // add macro in the head of your html document
    != js('adoption')

    // rest of your markup below

…passing in the name of your Coffeescript file (minus the .coffee) extension.

Obviously this is not the whole of the source code of the tool, but hopefully it has been a taster of how awesome modern javascript development can be! In the last few years, javascript has gone from being an unliked toy language into something a lot more powerful and expressive. Here at Mind Candy we hope to leverage amazing new tools like node.js and coffeescript in our future work to allow us to become a more happy and productive development team!

Testing Unity’s Flash Export on a Large Project – Part 1

Posted on by
2

Over the last weekend, I’ve been hard at work trying to get an unannounced Mind Candy project (made in Unity) to export to Flash. I thought it would be useful to share some details from the experience since most of the issues I’ve encountered would probably be avoidable if your project is architected in a way that lends itself to Flash export.

During the Christmas holidays, I made a game for Unity’s Flash in a Flash contest. It wasn’t the most exciting game, but it worked. The core mechanic of that game (including new 3.5 features such as nav mesh) exported to Flash well. The reason this game worked is because I had been paying close attention to the Flash export and knew what features wouldn’t work at that time. I avoided anything overly complicated and developed the game with the limitations in mind. Fundamentally, I decided to make a new, simple game rather than trying to port an existing one.

Now, I’m doing the opposite. I’m trying to get an existing game to publish to Flash. This project is a relatively large one. The game has been in development for quite a long time. It contains a lot of complex C# code and most importantly: a lot of features that don’t yet work in the Flash export. Trying to get this game to export to Flash is no easy task. I’ve spent numerous weekends on this since the 3.5 beta was made available and I still haven’t got it to work.

Despite the export (currently) not working for our project, there are a lot of lessons to be learned. Hopefully these will be of use to other people attempting the same task, and will be a good reference point for myself when I inevitably try to export the game again at a later date.

Currently Unsupported Features

Unity have already listed some of the unsupported features in the Flash export as part of the 3.5 preview faq. Some of these features (and the ones that have proved most problematic for me) are:

  • LINQ
  • Terrains
  • Asset Bundles
  • WWW
  • Raknet networking

If you’re using these features, then you’ll encounter a lot of errors as soon as you try to get Unity to build to Flash. Some example errors I’ve seen are:

Networking

  • error CS0246: The type or namespace name `ConnectionTesterStatus’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `NetworkView’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `BitStream’ could not be found. Are you missing a using directive or an assembly reference?
  • error CS0246: The type or namespace name `WWW’ could not be found. Are you missing a using directive or an assembly reference?

MovieTextures

  • error CS0246: The type or namespace name `MovieTexture’ could not be found. Are you missing a using directive or an assembly reference?

These errors are effectively a checklist of all the classes you’re using that aren’t yet supported and there’s only one thing you can do: remove them from your build. There are numerous ways to do this (depending on what you’re trying to achieve), from brute force deletion to telling Unity to skip these sections in a Flash build. You can do the latter by using platform dependant compilation.

All you need to do is wrap your Flash specific code in a platform check such as:

#if UNITY_FLASH
Debug.Log(&amp;quot;Flash build&amp;quot;);
#endif

In my case, the first thing I had to do was to try and remove these unsupported features. MovieTextures were easy to take out, as they’re not vital to our game. Networking, however was more problematic. And this is my first (and most important) lesson…

Lesson 1 – Separation of Networking Code

Our game currently uses the inbuilt RakNet networking solution. These networking elements are fundamental to our game, and as such, the networking code exists in many different areas of our codebase. When publishing to the web player or a standalone app/exe build this is fine. For Flash export, this suddenly creates a big problem when the networking solution isn’t yet supported.

As an example, if your game uses RPCs across clients to update core data in your game, then you’re going to have problems. I’m sure that there are other solutions which are better suited to Flash export, but this doesn’t fix my immediate problem: we have a game, where our chosen networking solution won’t publish to Flash. Unity suggest that you can use Flash networking instead of RakNet, but since I’m doing this export with tight time constraints (self imposed, by the mere fact it’s a weekend), that solution is not feasible for this test.

This has left me with one option in my mission to get our game working: rip out RakNet. This is not ideal, but luckily, our game copes with it ok.

This raises an interesting point in that the networking code should be as decoupled from the core mechanic of your game as possible. In some cases this can’t be done, but if you can find a way to make your networking layer easily removed/changed, then you’ll be in a much better place than I was regarding Flash export. It will also help you if you ever decide to switch to a different networking solution.

At this point, I’m going to gloss over about 10 other failed builds. It takes a few attempts at building to clear up this first wave of errors. Once you’ve cleared that first wave, you can breathe a sigh of relief and ready yourself for wave two: Attempting an actual build…

Attempting a Build

Once you’ve fixed/removed/hacked-out all the unsupported features, you’ll get to a point where the build process will now try to publish your game to Flash. The type of errors you get now will be more complex than those in wave one. Below is a screenshot of one of my build attempts at this stage:

You’ll note that these errors are more complicated than the “you can’t use ClassX because it’s unsupported” ones. In the case of these errors, it’s up to you to go into each of these classes and try to simplify your code as much as possible.

Some areas where our build failed were where we’d used generics. For example, we had fairly complex code to randomise the order of elements in an array. It wasn’t vital, so it went in the bin. This seems to be a common trend in trying to get this project to build to Flash. I’m slowly, over time, discarding features to the point where it’s a very stripped-down version of the game.

There are a couple of errors regarding our audio library in the above screenshot. This library wouldn’t convert at all (I got multiple waves of errors). My only solution at present has been to remove it.

The last item in that list is log4net. This caused a lot of issues. Rather than spending ages resolving them for this test, I decided it should also be removed. Since we used the logging in a lot of our code, I’ve ended up writing my own logging classes based on the log4net interfaces. This meant that I only had to fix up the imports in the class and our existing logging would still work using Unity’s own Debug.Log etc.

A few more iterations and build attempts occurred before wave 2 was complete. All in all, the first two waves have taken out large chunks of our features, and as a result the game feels somewhat unstable.

Akin to a game of [insert zombie survival FPS game of your choice here], we’ve just about survived the first few waves. We’re battererd, we’re bruised, but most importantly, we’re not defeated! We’re now ready for the next wave. Bring on the boss; the tank; the last major hurdle in the flash export – the conversion of your code to ActionScript.

Converting your code to ActionScript

At this stage, when you try to build, Unity will attempt to convert your source to ActionScript. Having previously spent years as a Flash developer, I find this part of the build rather exciting. The guys at Unity have done a fantastic job of getting this process to the stage it’s at.

That said, this is probably the toughest part of the process. Ripping out features and (to some extent) fixing the errors in the previous stage is easy. Trying to work out why the generated ActionScript doesn’t work is much more difficult. Luckily, when a build fails, you can find all the AS classes in a temp folder in your project (/Temp/StagingArea/Data/ConvertedDotNetCode/global/). This will enable you to look at them (if you wish) and try to understand where it might be going wrong, such that you can adjust your C# or js accordingly.

In my first attempt at this stage, I was left with 87 errors. The following are a small selection of these to give you an idea of the kind of problems I’ve seen:

Conversion Error 1

  • Error: Access of possibly undefined property $Type through a reference with static type Class.

This error seems to be very common and occurs when reflection is used (and probably in other situations). Unfortunately, a lot of our core libraries use reflection, and as such, this is a large problem to try and fix.

Conversion Error 2

  • Error: Call to a possibly undefined method IComparable$1_CompareTo_T through a reference with static type Number.

This has occurred because we’re trying to compare two values whose classes implement IComparable. In our case, this could be worked around relatively easily.

Conversion Error 3

  • Error: Type was not found or was not a compile-time constant: ReadOnlyCollection$1

In some of our classes we’re providing access to ReadOnlyCollections. It seems that we can’t use these at present and we could work round this by simply returning a standard Collection.

Conversion Error 4

  • Error: Call to a possibly undefined method String_Constructor_Char_Int32 through a reference with static type String.

A common style of conversion error that’s quite tricky to work out. I saw a lot of errors similar to this one.

These are just 4 of the 87 errors which need fixing. I expect that if/when all 87 are resolved, I’d have another wave or two to get through before the game would actually build. For now though, it’s Sunday night and I’ve run out of time to work on this test.

Next Steps…

My next challenge in this Flash export test is to go through the aforementioned 87 conversion errors and try to resolve them. I’m hoping that I’ll be able to get the game to build after another solid few days working on the export.

If that task proves too difficult then I will try a different approach of starting from a clean project and adding features one by one. In theory, that should be easier to get working, although that’s not how we’d want to export to Flash in the long run.

If I do get the export to work, I shall write a follow-up post with a walkthrough of some conversion errors. For these, I’ll include (where possible) the raw C#, the converted AS, and examples of how the errors can be avoided/solved.

For now though, I’m going to give up and play some well-earned Killing Floor! :D

Scala at Mind Candy

Posted on by
Reply

Reading some recent negative commentary about Scala with interest I felt like it would be good to share our experiences with Scala.

The Good.

Scala is an expressive language – It often results in a lot less code getting in the way of what you want to do or achieve. The simple example for something like this would be a simple bit of code like this:

case class Person(id: Int, name: String)
def lookupPerson(id: Int): Person = {
  new Person(id, "Test" + id)
} // Token implementation for the example.
val ids = Seq(1,10,100)
ids.filter(id => id < 50).map(lookupPerson)

To write this in Java would require a whole load of boilerplate, which includes the case class generating the stock class methods like toString, multiple lines to create a collection and then transforming a collection.

More powerful ways to write tests – This can fall under the Spiderman grouping of power admittedly, but libraries like specs2 and scalacheck make it easier to create the kind of tests we wanted. Scalacheck is the stand-out example of this where a generator for creating Person objects from above is as easy as this:

object TestGenerators {
  val personGenerator = for {
    id <- arbitrary[Int]
    name <- alphaStr
  } yield new Person(id, name)
}

That’s all it takes and that object can be imported into all of your test code that needs to generate Person objects.

Less magic – A lot of libraries like Spring and Hibernate need to use byte code modification or reflection to achieve what they do, which means that when they fail it can be really hard to diagnose or fix the problems. We’ve even seen some of these do things non-deterministically, which has caused hours of bemusement. Contrary to this, Scala libraries just tend to use the type system to achieve these ends which means that in our case we catch problems at compile-time, not at run-time.

The REPL – The idea scratchpad and general utility will be your friend in times of need. Use this as an education tool to step through an idea with some else. Use it to test some code interactively if you want to confirm something but you’re not quite sure how to code it or what results you’ll get. Use it to solve those gnarly Project Euler problems without having to create a whole new build for each one.

SBT – Controversial one this may be, but it manages to give you the sensible build model and plugin system that Maven has while allowing you to easily create custom tasks. If nothing else being able to run a command, for example the ‘test’ task, on each save is the most useful thing I’ve seen in a while.

POWAH! – There’s an elegance that comes with time when using Scala, in much the same way that it does with a lot of languages, that means code slots together so cleanly and with little friction. For me personally the Option class was the beginning of this change in thinking, where I realised that representing the possible lack of something without using a null made a lot more sense.

The Bad.

SBT – It’s a double edged sword in you’ll need to understand a bit of Scala to be able to do non-trivial configuration in it. Documentation for this has improved massively in recent times, it can still be somewhat impenetrable, especially to someone new to Scala.

Somewhat idiomatic libraries – Databinder Dispatch is a good example of this, writing a custom handler to parse a HTTP response is just unnecessarily puzzling. As with all libraries how easy they are to use and extend should be evaluated, so don’t be blinded by those libraries just because they’re written in Scala. It’s better to pimp a stock Java library that already works well than to use one that is badly written in Scala.

Binary compatibility – This is the stock issue that is often complained about, fortunately SBT does notice when two versions of the same library that relate to two different Scala versions are pulled into the dependencies. The way others have presented this is as a major pain point, it’s generally only so much of an issue as it is with Maven dependencies with a little more granularity. Also if you’re using SBT it’s possible to create dependencies that tie to the Scala version used automatically.

Knowledge – There’s a couple of aspects to this. The first is that Scala is a “new” language and as such there is one learning curve which relates to the language, SBT, the libraries and how to use them all effectively. Beyond this is that some functional programming concepts are foreign to a lot of programmers and this can be a wall that isn’t scalable in a short period of time for a lot of people. Hopefully with time this will become less of an issue but at the moment there aren’t a lot of Scala developers that can hit the ground running.

The Ugly?

As with all new things, there is a learning curve with Scala, which can be problematic, but the benefit of the design is that it’s possible to do something the “wrong way” as the language is very flexible. People with a history in languages like Java can start out writing code that looks not that much different but still get benefits like better collections. Then with time progress onto using more the powerful features in the language like pattern matching and implicits. For the foreseeable future Scala is a tool we intend to keep using, as it’s been of great benefit to us (this week I parsed a 37GB log file with a couple of lines of code in the REPL), maybe you should too…

99 Bottles of JMeter on the wall

Posted on by
2

I’ve recently had to do some performance testing on a couple of our new web services. I know of a few handy open source tools available for this. Tsung, Grinder and JMeter spring to mind. I find that I can get up and running in JMeter quicker than I can with the other tools and it tends to be the one I use most. One of the web services I wanted to test required some dynamic content to be generated in the body of the HTTP POST. Normally I’ve been able to get away with using the standard counters and config variables provided by JMeter, but this time I needed a bit more control. Luckily, JMeter allows you to do some scripted preprocessing of HTTP requests and I was able to use this to generate dynamic content within JMeter. I found the JMeter documentation to be a bit lacking in this area, so I created this blog post to give a quick explanation of how I managed to do this.

I’ve created a demo project that you can follow along with to see how it all works. Download the source code here: https://github.com/groodt/99bottles-jmeter Follow the README on GitHub to get everything setup and running. All you need is git, Python and JMeter. Open up the file “Test Plan.jmx” in JMeter to follow along.

The demo
The demo project is a simple web service that parses JSON payloads and prints a modified version of the “99 Bottles of beer on the wall” song onto the console. The JSON payload looks something like this:

{“drink”:”beer”, “bottles”:”99″, “date”:”1321024778956″, “thread”:”4″}

The server then parses these payloads and prints them out to the console:

JMeter then aggregates the response times in the summary report:

The HTTP POST
If you navigate to the “HTTP Request” node in the example you can see the JSON POST body being constructed:

The variables ${drink}, ${bottles}, ${date} and ${thread} are generated dynamically by a script that JMeter executes for each request.

The BSF PreProcessor
The BSF PreProcessor is run before each HTTP request to generate the dynamic content mentioned earlier. The BSF PreProcessor allows you to run Javascript, Python, Tcl and a few other languages inside JMeter. I decided to write my script in Javascript.

If you navigate to the “BSF PreProcessor” node in the example you can see the script that is used:

The Javascript
The simple Javascript basically places 4 variables in scope that are then available for JMeter.

// Calculate number
var count=vars.get("count");
var bottles=99-count;
vars.put("bottles",bottles);

// Calculate drink
var random=new Packages.java.util.Random();
var number = random.nextInt(4);
var drink = vars.get("drink"+number);
vars.put("drink", drink);

// Calculate date
var date=new Packages.java.util.Date().getTime();
vars.put("date",date);

// Calculate thread
var thread=ctx.getThreadNum();
vars.put("thread",thread);
  • In lines 1 to 4, the counter is read from JMeter then 99 is subtracted and the value is placed into scope under the name “bottles”.
  • In lines 6 to 10, a random number from 0 upto 4, it then uses this number as a lookup into the names of drinks (beer, wine, mead, cider) defined in the JMeter General Variables. It then stores this value in a variable named “drink”. It makes uses of java.util.Random to generate the random integer.
  • In lines 12 to 14, java.util.Date is used to generate a timestamp in milliseconds. This value is stored in a variable named “date”.
  • In lines 16 to 18, the JMeter thread number is read from the JMeter context and then stores this value into a variable named “thread”.

Executing Java libraries within the scripts
If you noticed in the scripts above, the Java libraries are exposed in JMeter under Packages.*. This allows you to execute any of the Java standard libraries or Java code in the classpath of JMeter. I think you can also write your own Java code and place it in the JMeter classpath and expose it in the same way.

Putting it all together
Putting all of that together gives you a handy way of doing reasonably complex performance testing in JMeter and I hope you find it useful.

NOSQL Exchange

Posted on by
Reply

This is a quick run through of the NOSQL exchange that Ciaran & I attended on Nov 2 at SkillsMatter, which featured 8 speakers and links to all talks are included.

A lot of people were asking which NoSQL solution to use?

This was the advice given by the speakers…. There is no silver bullet. Is there a need for reading/writing lots of Big data? Think about the shape of the data and how are you going to query your data to help understand which NOSQL solution fits best. Also understand the trade-offs when you choose your solution. Finally at the talks there was a lot of evidence of people using NOSQL solutions when a SQL solution would have sufficed.

1) THE STATE OF NOSQL TODAY by Emil Eifrem
This was the best talk of the day and anyone interested in NOSQL should watch the talk.

NOSQL stands for Not Only SQL.

Main types of NOSQL:

  1. Key-value originated from Amazon’s paper on Dynamo e.g. Riak, Voldemort (used in Linkedin)
  2. Column Family e.g. Cassandra, Hbase, Hyper table
  3. Document databases (most popular) descended from Lotus notes. e.g. CouchDb & MongoDb
  4. Graph Databases (nodes with properties) originated from Euler and Graph theory. e.g. infinitegraph, Neo4J

Documents are superset of Key-values. Graphs are supersets of documents and thus all others. Does this imply you should use Graph NOSQL solutions for all your NOSQL concerns? The graph NOSQL advocates think so.

Trends:

  • Acidity is increasing e.g. MongoDb adding durable logging storage, Cassandra adding stronger consistency
  • More query languages – Cassandra -CQL, CouchDb UnQL, Neo4J – Cyper, Mongo.
  • Potentially more schemas?

NoSql challenges:

  • Mindshare
  • Tool support
  • Middleware support

Oracle now adopting NOSQL with a KeyValue solution despite debunking NOSQL in May this year. NOSQL seems to be following similar historical trends to SQL. SQL which had many vendors to begin with, over time resulted in 4 large vendors. Could NOSQL result in a similar situation in the near future?

2) HANDLING CONFLICTS IN EVENTUALLY CONSISTENT SYSTEMS by Russell Brown
Key quote from this talk: “Large systems are always in some degree of failure”

Problem: According to CAP: Consistency, Availability & Partition tolerance – you can’t have all 3. Have to compromise by picking 2.
PACELC:
In the case of a partition (P), trade availability (A) for consistency (C)
Else (E) trade latency for consistency (C)

Riak inspired by Dynamo. Built in Erlang/OTP. Has features such as MapReduce, links, full text search. Uses vector clocks not timestamps. Statebox for automation of resolving conflicts.
Uses a wheel for storing clustered data.

3) MONGODB + SCALA: CASE CLASSES, DOCUMENTS AND SHARDS FOR A NEW DATA MODEL by Brendan McAdams (creator of Casbah)

MongoDb is not suited for highly transactional applications or ad-hoc intelligence that requires SQL support. MongoDb resolves around memory mapped files. Mongo has an autosharding system.

Things to remember:
The datastore is a servant to the application not vice-versa
Don’t frankenshard

4) REAL LIFE CASSANDRA by Dave Gardner (from Cassandra user group)

  • Elastic – Read/Write throughput increases as scale horizontally.
  • Decentralised no master node.
  • Based on Amazon’s Dynamo paper
  • Rich data set model
  • Tunable
  • High write performance

If your requirements are big data, high availability high number of writes then use Cassandra.
When data modelling, start from your queries and work backwards.
Has expiring columns.
Avoid read before write & locking by safely mutating individual columns
Avoid super columns, instead use composite columns
Use Brisk (uses hadoop) for analysing data directly from Cassandra cluster.

5) DOCTOR WHO AND NEO4J by Ian Robinson
Although it was a fairly slick presentation it seemed to focus too much on modelling Doctor Who and his universe as a working example of graphs & Neo4J. Could this be to hide some shortcomings in Neo4J?

  • Neo4J is a fully ACID replacement for Mysql/Oracle.
  • Neo4j is a NOSQL solution that tries to sell itself as the most enterprise ready solution.
  • Has master/slave nodes.
  • Has 3 licenses: Community/Advanced/Enterprise.

With mentions of 2 phase commits, other than the advantage of modelling relationships such as social networks, there seemed little benefit from moving away from a relational database.
Having spoken to the Neo4J guys afterwards, it seems that the DB loses its ACIDity once you cluster it, and becomes another eventually-consistent store!

6) BUILDING REAL WORLD SOLUTION WITH DOCUMENT STORAGE, SCALA AND LIFT by Aleksa Vukotic

CouchDb:

  • Written in Erlang has Lift support (Scala framework)
  • Exposes REST/JSON endpoints
  • Eventually consistent
  • Versioning appends only updates
  • Mapreduce for querying using views

7) ROBERT REES ON POLYGLOT PERSISTENCE
A muddled presentation talking about mixing graph NOSQL solution with a document based one.

8) THE FUTURE OF NOSQL AND BIG DATA STORAGE by Tom Wilkie
Rather than using the out of the box storage engines for NOSQL solutions, there can be dramatic throughput gains for using alternative storage engines such as Tokutek and Acunu (Castle).

Flash on the Beach 2011

Posted on by
Reply

Last month I went to Flash on the Beach in Brighton. It was the first time I’ve to this particular event and I really enjoyed a lot. For the few of you that don’t know, FOTB is an annual conference full of code, design and inspiration.

The three days in Brighton were full of inspiring stuff, like Carlos Ulloa new WebGL project, Lights, the Seb Lee-Delisle live experiment mixing OpenFrameworks, HTML5 and Phones, as well as lots of new people of the Elevator Pitches. Just check out the impressive main titles video created by Gmunk, creator of some of the FX for Tron Legacy.

Adobe showed some of the new stuff like EDGE, a Flash-like productivity tool to create HTML 5 canvas animations, and talked about Molehill/Stage 3D. They also talked about a new hardware accelerated framework for 2D animation in Flash recently released under the name of Starling. But all that doesn’t mean that that the ”Flash is over” controversy wasn’t in the air, just check out the slide of Adobe’s official position on the question “When must I use Flash and when not?”.

Adobe posture on the Flash-HTML debate

Adobe posture on the Flash-HTML debate

Remy Sharp talked about this question too in “HTML5: Where Flash isn’t needed anymore”. He spoke about the fallback solutions when some features don’t work (usually using Flash), support of different parts of the specification in different browsers/platforms and the pitfalls that we will find on our path embracing this new standard.

In the closing speech John Davey, the creator and main organiser of the event, revealed that this is going to be the last Flash on the Beach! But don’t panic yet! It’s just a change of title and (maybe) format, to allow the conference to not be related with a single technology (i.e. the Flash platform) and be more based on design and creative coding (maybe something like the move of Flash Belt to The Eyeo Festival?)

Of course I’m leaving out a lot of things, like the interesting chat about kids games playability done by Jon Howard, or the demos of the new Away 3D using the new Stage 3D API, so don’t forget to check out the list of speakers and check out their individual sites.

LFPUG invasion!

Posted on by
Reply

Last Thursday Mind Candy had the pleasure of hosting the latest LFPUG meetup!  LFPUG or London Flash Platform User Group is a monthly meet up of Flash enthusiasts who either work in the industry or enjoy experimenting with Flash in their spare time!  Each month the members take the opportunity to talk about exciting projects and new technologies in the field.  The night was a great success around 50 people turned up.  Mind Candy provided beer and snacks and everyone had a chance to mingle and chat before the presentations began.

This month there were two presentations.  First up was Rob McMichael who talked us through the recent breakthroughs with Flex Mobile, he showed off some examples of how easy it is to use the Flex API with mxmlc and the sort of components that are currently available within the framework as well as upcoming features in the upcoming version of Flex. The second presentation was made by Tink who told us all about Flex Navigators and he talked us through some very impressive work he’s been doing on some open source navigators.  He showed off some examples of complex navigators that could nest themselves within each other and produce some beautiful effects such as 3D cover flow. It was a great evening and a great way to meet like minded Flash enthusiasts!

The next meetup is on the 27th October and features a presentation on the version control system Git and the methodology of Designers and Developers working in harmony.

Hope to see you there!

Unite11 – Exporting to Flash from Unity

Posted on by
Reply

Last week, four of the Mind Candy tech team ventured over to San Francisco for the Unite conference. One of the things I was most keen to learn about is the upcoming Flash export.

On the Tuesday, we attended the Flash afternoon where Lucas Meijer and Ralph Hauwert detailed the latest progress on Unity’s export to Flash. On first impressions, the games they’ve been working on look fantastic! One of the reasons I’ve been keen on working with Unity over Flash for 3D games dev is the awesome toolset, and these guys have really nailed the Flash export.

As long as your code isn’t doing anything too “exotic”, Unity will convert your code (c#, js (strict) and boo) to ActionScript. It will then compile this, along with your assets, into a swf using the mxmlc compiler. One thing to note here is that literally everything is compiled into that one swf. There was mention of them using asset bundles at a later date to generate individual swfs for content such as scenes. Since everything is in one swf, that means file sizes could get pretty large really quickly. One of the demo apps generating simple spheres compiled to a ~1.4mb (iirc) swf which is pretty chunky for something so simple.

The generated AS3 code is pretty readable although since there’s no method overloading in Flash, they’ve had to compromise on function names somewhat! That said, it should be pretty easy to debug your exported app.

The features currently supported in Flash are:

  • PhysX
  • Particle system
  • Custom and standard shaders

Things that won’t work in the first version include:

  • Anything needing depth textures
  • Advanced audio such as doppler and reverb
  • Dynamic shadows
  • Mouse lock
  • Unity networking (you’ll have to use a Flash networking solution for now)

Looking at the performance of the games, they seemed to run pretty well. They mentioned trying to push anything you can to the GPU rather than CPU. An example given would be to animate textures in a shader rather than scrolling it in a script, as well as the usual light baking and occlusion culling. Basically, as Lucas put it: “The best way to make your game go faster, is to have it not do stuff”.

So to summarise – from a first glance, the Flash export looks pretty awesome, and the Unity guys have done a great job getting it working. There are still a fair few features missing, but there should be enough there for most simple Unity games to be exported. Unfortunately, both the release date and the price were listed as “TBA”, so I guess you’ll have to wait a little longer before you can try it out.

Arkham City developer Q&A at BAFTA

Posted on by
Reply

I always wondered why I had never noticed BAFTA’s Piccadilly HQ before but armed with the address and an iPhone I spotted the discreet entrance. An anonymous-looking set of doors that opened to reveal a swish foyer and a receptionist who happily had my name “on the list” already. How clandestine! How chic! How very, dare I say it… Hollywood? The swish interior of BAFTA, free bar and free Arkham City comic all were a great introduction for BAFTA’s sneak preview of Batman Arkham City. As I had thoroughly enjoyed, and indeed completed, Rocksteady’s first Batman game (Arkham Asylum) I was eager to get a look at the sequel.

Rocksteady Studios provided both their senior staff and a live demo of the game itself, which they jumped right into to show a level set in Arkham City. The demo was very entertaining with the Producer encouraging the very male audience to whoop whenever a particularly difficult takedown or new combo was performed. A particular crowd-pleaser was the “double silent takedown” wherein the Dark Knight grabs the heads of two unsuspecting goons and mashes them into the floor (in keeping with Batman’s rules of engagement, the goons are knocked unconscious rather than killed). Visually the game held up very well to being projected on a large movie screen, looking admirably like the original Arkham Asylum despite now having to render a much larger open-world play area. However, interior locations are still used extensively to keep the signature Arkham-verse stealth and combat tactics in use.

A nice touch on the audio side was use of intercepted communications and chats from nearby goons as Batman navigates the heights of Arkham City. Swooping effortlessly from rooftop to rooftop Batman hears the state of the city from the nervous exchanges of gossip between thugs. This makes exploration of the city an opportunity for narrative development and engagement.

Although Batman’s core “stealth melee” gameplay remains unchanged, new gadgets and abilities are – as expected – provided in the sequel. Demonstrated was an electronics-futzing gun that allows players to start motors, open doors or activate electromagnets to solve puzzles and progress through the environment. Electrical pulses can be positive or negative, offering a little more depth to the puzzle-solving gameplay.

An intriguing end-of-level cutscene hinted that the big boss in Arkham City may not be the Joker, as he apparently dies of complications following his massive drugs intake during the events of Arkham Asylum. Is this the True Death for the Joker? It seems an unlikely fate for the long-time arch enemy of Batman, but the Rocksteady devs wisely played their cards close to their chests and didn’t say. If I were a betting man I would assume a return at the end of Arkham City to set up Arkham 3 would be likely. Until then, other villains from Batman’s back catalogue provide the opposition, including Two Face and Iceman. No word as to the return of the Sandman and his hallucinogenic reality-bending levels. Personally I would love to see more of the Sandman as his levels provided a welcome break from tromping around Arkham Asylum and some of the more bold design choices in the original game, including one memorable sequence in which an endless corridor melted away to reveal a disturbing dreamscape of Batman’s fears.

An interesting change in Arkham City is the promise of GTA-style open world freedom in choosing which missions to undertake (presumably, only up to a point). Those who found the strict linearity of Arkham stifling should prefer having more choices this time. Additional variety in playable characters was confirmed as the game will start with the player in control of Catwoman, acting in some ways as training before taking the reins of the Dark Knight himself, presumably for the majority of the game.

Many more collectible and hidden items are promised with the cunning idea that many will be inaccessible when first seen. However once photographed they are then recorded on the in-game map to be searched for later, reducing the mental effort needed to track down every single Ridder trophy. Helpfully for occasional players, loading screens give you a synopsis of what you did last, so presumably if you take a break you can pick up pretty much were you left off.

After a pleasingly long live demonstration it was time for questions from the audience, with many ‘heads of department’ from Rocksteady on stage to provide answers. Although most of the questions were somewhat fluffy I felt some interesting tidbits did emerge from the Q&A session.

There was significant technical work on making the licensed Unreal Engine stream in the larger levels in Arkham City without putting up any “loading” screens. This time players will seamlessly traverse from exterior city-navigation into interior bat-stealth and combat sections. Rocksteady noted their preferred conceptual model for streaming was to maintain a ‘bubble’ of loaded high-quality assets around the player, making use of lower level-of-detail models for farther scenery. Fairly standard stuff in other words, although it’s likely there wasn’t time for a more detailed technical explanation.

Although sporting open-world gameplay as popularised by Grand Theft Auto, the developers confirmed that there will be no drivable vehicles in Arkham City (as in Asylum) claiming that “Batman is the best vehicle” for the game environment.

One of the better questions was about Detective mode, which many players of the first game used almost exclusively (to the chagrin of the art team). To give players an incentive to actually turn it off this time, the team differentiated the two modes by making the regular mode for navigation and Detective for stealth. For example, by changing the contrast in Detective mode, making the compass and some game secrets only visible in normal mode.

When asked if original villains be created for the Batman Asylum-verse the team responded that the fun thing about Batman is the villains don’t get killed so they “stick around” and you get to know them. So there are already plenty of canonical villains to choose from. (Which strongly suggests that the Joker is likely to make a reappearance in some form). In terms of working with the Batman IP in general, Rocksteady said that one advantage is that they do not need to tell the whole backstory – everyone knows what Batman’s motivation is. But there can be “too many” stories or characters to choose from.

To give more longevity to the Challenge Rooms (stand-alone “combat puzzles” in the first game) they have introduced Campaigns of several rooms in sequence. The twist is that you are given a number of Modifiers (or perks) for the campaign but have to decide which room to use a modifier on, so giving more variety to each Campaign.

A few short answers from the Q&A:
3D TVs will be supported, as is the current vogue. Rocksteady use focus testing but didn’t do it “too early” instead using it to refine and tweak the gameplay for a smoother experience. The classic Metroid series was revealed as being a large influence on the design of both Arkham games. The studio headcount is at about 100 people now. Their choice of Robin is Tim Drake rather than Dick Greyson.

When asked what was next for Rocksteady Studio they said they wanted to “continue making exceptional quality AAA games”, but they could not say exactly what was next.

From what they said regarding “working very hard to finish the game” it sounded like they had been in crunch for a while to finish the game. The first Arkham Asylum impressed with its polish, focus and originality, with impressive amount of ‘game’ from a relatively small team. Scaling up the scope and the team was an obvious next step in some regards, but is in my opinion a risky one. Doubling down on a large AAA sequel is quite common but if the game doesn’t do well, the risk is equally large. I would have loved to ask them about this and if they would have preferred to diversify and make two smaller games. Also, what are they doing to avoid crunch? Do they even think they need to? Had the crunch been worse on the sequel, what was their staff retention rate and so on. Sadly the questions were, I felt, a little too easy.

I got the impression Rocksteady was a somewhat “old school” developer and I am somewhat of the opinion that they are a dying breed. When every “bet” you make can break the company, you only need to fail once to bring it down. Watching how companies like Rocksteady adapt and survive (or don’t) in a changing commercial environment is going to be fascinating stuff. I wish them success, and can’t wait to get my hands on Arkham City.