London Python Dojo Returns to Mind Candy

Last night saw the return of the London Python Dojo to the Mind Candy office. For those who are unfamiliar, the dojo is a monthly meetup for python enthusiasts that really covers the full range of “What’s Python?” to “I’ve been using Python for 24 years”, where we meet and come up with ideas/scenarios/problems that can be solved with Python in a couple of hours (usually after some refreshments).resized1.resized2

Ideas are written on a whiteboard an then voted on for what to project to take on for the evening. We then break of in to teams and code for about an hour or so and then each team presents what they’ve done and the ideas behind what they were trying to accomplish (even if you don’t have working code).

resized3(Last nights board)

Last night’s task was to implement battleship logic/strategy which then could be played against other teams. You can see the (unofficial) winning team’s code here.

I am really biased as I was in fact a part of Team 1.

Overall a very good evening in good company creatively coding. (as it usually is).

For more information on the London Python Dojo check out and follow @ldnpydojo on twitter.

You can also join the Python UK mailing list here.

A Puppet module for Dynamic DynamoDB

As my colleagues have said in other posts, we make an extensive use of Amazon Web Services at Mind Candy. Recently we decided to use the AWS NoSQL offering DynamoDB for a specific use case in one of our products.

Whilst DynamoDB provides us with a highly distributed NoSQL solution, it works based on telling Amazon what read and write capacity you require via their API. If you find that you go over either of these value you begin to, potentially at least, lose queries if you have not factored in some sort of caching layer using, for example, Amazon SQS.

In the ideal world, Amazon would offer auto scaling features for DynamoDB, however at time of writing they don’t. Instead they advise people to use an independently developed tool called Dynamic DynamoDB written by Sebastian Dahlgren.

Dynamic DynamoDB is a tool written in Python that allows us to effectively auto scale our provisioned reads and writes. It use CloudWatch metrics to establish current usage and then based on the configuration option either scales up or down your provisioned capacity on a per table basis.

As I’ve posted before here, we use Puppet at Mind Candy, so the first point of call whenever a new tool comes along is to see if anyone has written, or started to write, a Puppet module for it. Sadly it didn’t look like anyone had, so we quickly wrote up our own, which is available on Github here.

99 Bottles of JMeter on the wall

I’ve recently had to do some performance testing on a couple of our new web services. I know of a few handy open source tools available for this. Tsung, Grinder and JMeter spring to mind. I find that I can get up and running in JMeter quicker than I can with the other tools and it tends to be the one I use most. One of the web services I wanted to test required some dynamic content to be generated in the body of the HTTP POST. Normally I’ve been able to get away with using the standard counters and config variables provided by JMeter, but this time I needed a bit more control. Luckily, JMeter allows you to do some scripted preprocessing of HTTP requests and I was able to use this to generate dynamic content within JMeter. I found the JMeter documentation to be a bit lacking in this area, so I created this blog post to give a quick explanation of how I managed to do this.

I’ve created a demo project that you can follow along with to see how it all works. Download the source code here: Follow the README on GitHub to get everything setup and running. All you need is git, Python and JMeter. Open up the file “Test Plan.jmx” in JMeter to follow along.

The demo
The demo project is a simple web service that parses JSON payloads and prints a modified version of the “99 Bottles of beer on the wall” song onto the console. The JSON payload looks something like this:

{“drink”:”beer”, “bottles”:”99″, “date”:”1321024778956″, “thread”:”4″}

The server then parses these payloads and prints them out to the console:

JMeter then aggregates the response times in the summary report:

If you navigate to the “HTTP Request” node in the example you can see the JSON POST body being constructed:

The variables ${drink}, ${bottles}, ${date} and ${thread} are generated dynamically by a script that JMeter executes for each request.

The BSF PreProcessor
The BSF PreProcessor is run before each HTTP request to generate the dynamic content mentioned earlier. The BSF PreProcessor allows you to run Javascript, Python, Tcl and a few other languages inside JMeter. I decided to write my script in Javascript.

If you navigate to the “BSF PreProcessor” node in the example you can see the script that is used:

The Javascript
The simple Javascript basically places 4 variables in scope that are then available for JMeter.

// Calculate number
var count=vars.get("count");
var bottles=99-count;

// Calculate drink
var random=new;
var number = random.nextInt(4);
var drink = vars.get("drink"+number);
vars.put("drink", drink);

// Calculate date
var date=new;

// Calculate thread
var thread=ctx.getThreadNum();
  • In lines 1 to 4, the counter is read from JMeter then 99 is subtracted and the value is placed into scope under the name “bottles”.
  • In lines 6 to 10, a random number from 0 upto 4, it then uses this number as a lookup into the names of drinks (beer, wine, mead, cider) defined in the JMeter General Variables. It then stores this value in a variable named “drink”. It makes uses of java.util.Random to generate the random integer.
  • In lines 12 to 14, java.util.Date is used to generate a timestamp in milliseconds. This value is stored in a variable named “date”.
  • In lines 16 to 18, the JMeter thread number is read from the JMeter context and then stores this value into a variable named “thread”.

Executing Java libraries within the scripts
If you noticed in the scripts above, the Java libraries are exposed in JMeter under Packages.*. This allows you to execute any of the Java standard libraries or Java code in the classpath of JMeter. I think you can also write your own Java code and place it in the JMeter classpath and expose it in the same way.

Putting it all together
Putting all of that together gives you a handy way of doing reasonably complex performance testing in JMeter and I hope you find it useful.