One of the more interesting things about starting a RentACoder project is the options they present to you regarding time of delivery:

I’m sure we’ve all been on one of those Option One “Project must be delivered in X days” types. I’ve been lucky enough to be on a few Option Two projects and even a few Option One projects where we’ve negotiated what will be delivered in X days and in what priority. As my project has no deadline and I don’t want my Programmer-For-Hire to rush and reduce quality in order to meet a deadline, I plumped for Option Three.

I quite like the question regarding Managing Coder Estimates - its like your signing up to a contract agreeing that all programmers underestimate by either 2 or 5 times!

The best thing at the moment is IntelliJ, its currently the only thing out there that can navigate across a codebase. With Aptana or Eclipse, if the method exists in another file it cannot navigate to it and you’re forced to resort to searching in files ending with *.js. Second best is Eclipse 3.3 with the JSEclipse 2.0 pre-release installed. (I’m really impressed with Eclipse 3.3 - its improved in several places - but I especially like the Quicksilver style Ctrl-3 shortcut).

How hard can unit-testing JavaScript be?

On the surface of things JavaScript seems to have a lot of the same tools that Java has for unit-testing. JSUnit is the JavaScript equivalent of JUnit and JSMock the equivalent of JMock.

The syntax of JSUnit is fine, its pretty much as you’d expect and conforms to the XUnit conventions. The key bit that’s missing is the fast feedback. When working on a class you want to be able to run the test often so you can see the immediate effects of changes you’ve made and the let the test tell you exactly whats wrong. To run a test with JSUnit, I need to open the JSUnit test runner in a browser, load my test page and then run it. Once I’ve run it, assuming its failed I get a list of tests that failed - but not the reason they failed, I need to double-click - then I get an alert telling me why the test failed. It really interrupts your flow, especially when you’re flipping between multiple classes. If you’re using an IDE with a Browser Preview tab, you can get around this problem with a little JavaScript along the lines of:

(Sorry plus signs don’t seem to show up - hopefully you can get the general idea).

I’ve got this working in Eclipse by installing the EclipseHTMLEditor plugin and including the following common bit of code in all JSUnit tests:

Now a test can be run by just clicking on its Preview tab.

Making a Mockery

JSMock is kind of good - at least someone’s written a mocking framework for JavaScript. As soon as you start to use it - you realize that it is by know means a mature piece of software.

It doesn’t give you any kind of useful messages, for example it will say ‘Unexpected Arguments’ but won’t tell you what the actual arguments were against the expected arguments. Similarly for an unexpected invocation, it won’t tell you what has been called and what was expected to be called. The other missing piece is support for literal objects - JSMock gets it wrong if you pass a literal object as an argument. This is basic stuff - thankfully the code is still quite small and I’ve found myself adding a lot of these features myself just to be able to use it.

The cool thing about JSMock is that because JavaScript is such a flexible language you don’t need to put your method calls in strings like in JMock 1 and you can keep using literal language (unlike in JMock 2):

Another cool thing is that if I have static objects as collaborators I can mock those too:

So I invented (by accident) the Red Green Cup of Tea pattern - write a failing test, make the test go green, and go and make a cup of tea whilst the build runs. I’ve found that whilst making a cup of tea, my pair and I can reflect on what we’ve just done and often find we weren’t 100% happy with what we just did. We then set our cups of tea back down on the desk and Refactor the code.

A quick google for end-to-end test reveals a few examples of how other people view end to end testing:

• automated diagnostic routines tailored specifically for the system under test.

  - taken from a test for an electronic warfare system. In this case they send several low-level signals through the system and make assertions based on the outputs.

• Component tests can be incrementally combined until they provide a validation of the entire process flow.

  - taken from a testing tool - in this case an end to end test is viewed as simply the sum of several component (unit & integration) tests.

• Commands and telemetry are usually transmitted between TKSC and Kibo through the U.S. segment of the ISS, via the Tracking and Data Relay Satellite (TDRS). This time, however, the test was conducted by linking TKSC via JSC to the Kibo PM, which is currently located at KSC. This test closely simulated the actual configuration for the planned Kibo operations.

  - taken from a space station test - in this case the end to end test seems to be a run-through (or rehearsal) of what will actually happen in the real system. However, in this case, they can’t use a real test satellite, so they mock that part out by locating the component at a remote lab.

I see this as three separate ways of doing an end-to-end test:

1. Don’t - each component is well-tested and each interaction is tested with an integration test. (Therefore - the sum of these parts becomes your end-to-end test.) Imagine an end-to-end test that tests end-points A-D. If you have integration tests that cover points A-B, B-C, and C-D, then you already are testing A-D.
2. Automated low-level - you throw stuff in and you get stuff back. But the danger is that this doesn’t prove that the system works, it just proves it behaves in the way you expect it to. Thinking back to the Electronic Warfare System, chucking signals through the components may prove logical correctness, but maybe the only way to really test it is to turn the key and push the big red button.
3. Simulation - you take as much of the real system as possible and start pushing those buttons. Of course the danger here is that you don’t have so many boundary tests and it can be hard to simulate failure - imagine trying to test the satellite falling out of orbit.

Dude where are my end-points?

One common thing amongst a lot of the end-to-end tests is that they seemed to have one single end-point i.e. they put something in and expect something back:

The model used is different to the end-to-end testing infrastructure I have been looking at today. In this model, the infrastructure has several end-points:

In this model, the three end-points cause most of the problems:

1. The tests become white box - the infrastructure knows about the inner-workings of the system. These inner-workings should be irrelevant and should be covered by component-level testing.
2. The tests are surplus - they cover things already covered by unit and integration tests.
3. The tests are brittle - the infrastructure has inner knowledge of the system - it does selects on the database and scrapes webpages - these things change as the code develops therefore the tests break often - not because the tests fail but because the test infrastructure code is too dependent on inner-workings and breaks often.
4. The tests are testing multiple products - if you have multiple end-points this would indicate that your test is actually testing multiple products. Taking the example above - what is being tested? The order viewer, the order taker, or the supplier interface. The answer is they are all being tested, but individually - so you actually no longer have an end-to-end test - the test is a collection of component tests.

So, a possible solution to this would be to break the test in to three end-to-end tests:

1. Test the Order Viewer - order an egg - see that it behaves correctly on the Viewer.
2. Test Ordering An Egg - put stuff in one end and check you get back an egg.
3. Test Ordering An Egg From The Supplier - ask the supplier for some eggs and check you get them.

Simple, right? But it all get’s a bit more complicated when scenarios enter the scene.

Test Scenarios

Test scenarios are essentially dress-rehearsals for things that a customer can do with your product. Look back at the satellite example - they need to run-through what happens when the product is up in orbit and in contact with mission control.

Looking at the Egg System, let’s have a scenario as follows:

Customer already has an egg on order. They want to cancel the egg order. So the request to cancel the order is sent in to the system, the cancel needs to be visible to customer support on the Egg Order Viewer and the cancel needs to be forwarded to the supplier correctly.

Woah, hang on a minute, we’re back to the three end-points, we need to check that order gets sent in and we get a confirmation from the Egg Order Taker, check we can see the cancel on the Egg Order Viewer and check the right stuff gets sent by the Egg Supplier Interface.

Or do we? Let’s take another look.

The scenario can be rewritten as:

1. The customer rings telephone number and says “Cancel my egg!!!”.
2. If the customer’s egg is gone. Customer happy. Test passed.

But wait…

1. The customer rings telephone number and says “I changed my mind!! Don’t cancel my egg!!!”.
2. The operator looks on the order viewer, and stops the cancel. Customer happy. Test passed.

There’s a lot of other scenarios here but essentially we’re only using two interfaces - the Order System in the first case and the Order View in the second case. We don’t care about the supplier in these cases because it doesn’t matter for the product - you need to be able to order and call up and find out what’s going on with your order - that’s it.

So, now we’ve defined these interfaces - it’s easy to re-write the end-to-end testing infrastructure:

Test of much as the real system as possible

Back to the thing we learned early on - end to end tests test as much of the real system as possible. In the particular project I’m working on this is the hard part, the application is deployed across multiple containers, load-balanced, on hard-to-obtain environments etc… etc…

Time is a constraint here, so it’s possible to mock some of these things that don’t really matter and simulate a lot of the others. But now comes an important point - if you can’t test it - should you release it? For example, if it’s not possible to say simulate multiple containers, then is the application really production-ready, should it be deployed, is it essentially untested and therefore not safe? Probably not, but it’s something to definitely think about.

After I wrote the storyboard retrospective, I happened across Joe Walnes blog where I found Building testable AJAX apps (Does my button look big in this?). It was nice to see someone had a similar thoughts to myself as to why you would approach a Javascript project differently than you would any normal project; quoting Joe:
“What’s a bit annoying is that we know all this stuff already. And certainly we do this in any other type of GUI development, but when I came to first doing ajax I thought ‘oh. it’s just javascript, don’t really take it seriously, just wire up a little event in the html’ and it’s almost like we’ve regressed back to the VB days when you built a whole GUI by saying ‘Right, first I’m going to draw a button, then I’m going to put an onclick to the button then I’m going to write 10,000 lines of code on the onclick.’”

There were several other interesting things in Adam and Joes talk. They chose not to use Google Web Toolkit despite working for Google and in fact they did not appear to use any AJAX or Javascript libraries/frameworks apart from JsUnit. The best idea they had was to take a great approach to the application borrowing from ideas that you would use when developing a traditional GUI application (including Michael Feathers’ The Humble Dialog Box). They used a kind of Model View Controller to make the app more testable. In fact they were able to test some code in the controller without requiring a browser using Rhino Engine. The way they tested the view was most interesting: they did it manually - well almost - they had written a small bit of helper code that made all UI functions available to play around with in a sidebar. The sidebar also showed any Javascript events that had been fired (see image).

This seemed like a really logical way to test the UI controls and view of a GUI because often the only way to test usability is to use something. Also, making assertions on pixel widths and things just seems silly and static screen grabs are’nt the same thing as actualling using the app - so why not manually test it - especially nice if you can get some real users.

After speaking to someone in the pub about AJAX development, I was pointed towards two very useful looking tools. Prototype is a JavaScript Framework which provides some useful libraries and also some really useful extensions to the standard DOM. For example, using the extensions you can do cool stuff like this found at Sergio Pereiras site:

The $() operator is just a shortcut for document.getElementById(), and the $A() operator converts an element to an array, you can see how this function works based on the html below:

<select id="lstEmployees" size="10">
    <option value="5">Buchanan, Steven</option>
    <option value="8">Callahan, Laura</option>
    <option value="1">Davolio, Nancy</option>
</select>

The other useful tool is Crosscheck which is a cross-browser testing framework which doesn’t actually require any browsers. This seems like a very useful solution for cross-browser compatibility testing of storyboard.