chris clarke

Trust me, I’m a Framework

From the Spring 2.0 docs:

You can generally trust Spring to do the right thing.

Awesome Clock

GNUs Not U 3

Stop a command from exiting after you logout:

nohup monkeys.sh

Convert html so it can be displayed on a web page:

cat index.html | perl -pe 's/</&lt;/g; s/>/&gt;/g'

See the strings in a binary file:

strings /bin/rm

Show line numbers of a file:

nl monkeys.c

See a particular line in a file:

cat buildOutput.txt | nl | awk '$1==5 {print; }'

Concatenate files horizontally:

paste times dates people

View and save output at the same time:

ls | tee filesInthisDirectory.txt

Say yes to everything:

yes | rm -i *.txt

Send a message to everyone:

echo 'all your base are belong to us' | wall

See a list of processes every 5 seconds:

watch -n5 "ps aux"

Sort by the second column of a file:

cat file.txt | sort -k2

Delete a bunch of files matching a regex:

find . -name "*.svn" -exec rm -rf '{}' \; -print

Reverse a string:

echo "Never odd or even" | rev

What is an end to end test?

In my latest role we’ve been tasked with re-writing an existing end-to-end testing infrastructure. Whilst going through, understanding and improving the existing infrastructure, I started to think about exactly what an end-to-end test is and what it should do.

A quick google for end-to-end test reveals a few examples of how other people view end to end testing:

• automated diagnostic routines tailored specifically for the system under test.

  - taken from a test for an electronic warfare system. In this case they send several low-level signals through the system and make assertions based on the outputs.

• Component tests can be incrementally combined until they provide a validation of the entire process flow.

  - taken from a testing tool - in this case an end to end test is viewed as simply the sum of several component (unit & integration) tests.

• Commands and telemetry are usually transmitted between TKSC and Kibo through the U.S. segment of the ISS, via the Tracking and Data Relay Satellite (TDRS). This time, however, the test was conducted by linking TKSC via JSC to the Kibo PM, which is currently located at KSC. This test closely simulated the actual configuration for the planned Kibo operations.

  - taken from a space station test - in this case the end to end test seems to be a run-through (or rehearsal) of what will actually happen in the real system. However, in this case, they can’t use a real test satellite, so they mock that part out by locating the component at a remote lab.

I see this as three separate ways of doing an end-to-end test:

1. Don’t - each component is well-tested and each interaction is tested with an integration test. (Therefore - the sum of these parts becomes your end-to-end test.) Imagine an end-to-end test that tests end-points A-D. If you have integration tests that cover points A-B, B-C, and C-D, then you already are testing A-D.
2. Automated low-level - you throw stuff in and you get stuff back. But the danger is that this doesn’t prove that the system works, it just proves it behaves in the way you expect it to. Thinking back to the Electronic Warfare System, chucking signals through the components may prove logical correctness, but maybe the only way to really test it is to turn the key and push the big red button.
3. Simulation - you take as much of the real system as possible and start pushing those buttons. Of course the danger here is that you don’t have so many boundary tests and it can be hard to simulate failure - imagine trying to test the satellite falling out of orbit.

Dude where are my end-points?

One common thing amongst a lot of the end-to-end tests is that they seemed to have one single end-point i.e. they put something in and expect something back:

The model used is different to the end-to-end testing infrastructure I have been looking at today. In this model, the infrastructure has several end-points:

In this model, the three end-points cause most of the problems:

1. The tests become white box - the infrastructure knows about the inner-workings of the system. These inner-workings should be irrelevant and should be covered by component-level testing.
2. The tests are surplus - they cover things already covered by unit and integration tests.
3. The tests are brittle - the infrastructure has inner knowledge of the system - it does selects on the database and scrapes webpages - these things change as the code develops therefore the tests break often - not because the tests fail but because the test infrastructure code is too dependent on inner-workings and breaks often.
4. The tests are testing multiple products - if you have multiple end-points this would indicate that your test is actually testing multiple products. Taking the example above - what is being tested? The order viewer, the order taker, or the supplier interface. The answer is they are all being tested, but individually - so you actually no longer have an end-to-end test - the test is a collection of component tests.

So, a possible solution to this would be to break the test in to three end-to-end tests:

1. Test the Order Viewer - order an egg - see that it behaves correctly on the Viewer.
2. Test Ordering An Egg - put stuff in one end and check you get back an egg.
3. Test Ordering An Egg From The Supplier - ask the supplier for some eggs and check you get them.

Simple, right? But it all get’s a bit more complicated when scenarios enter the scene.

Test Scenarios

Test scenarios are essentially dress-rehearsals for things that a customer can do with your product. Look back at the satellite example - they need to run-through what happens when the product is up in orbit and in contact with mission control.

Looking at the Egg System, let’s have a scenario as follows:

Customer already has an egg on order. They want to cancel the egg order. So the request to cancel the order is sent in to the system, the cancel needs to be visible to customer support on the Egg Order Viewer and the cancel needs to be forwarded to the supplier correctly.

Woah, hang on a minute, we’re back to the three end-points, we need to check that order gets sent in and we get a confirmation from the Egg Order Taker, check we can see the cancel on the Egg Order Viewer and check the right stuff gets sent by the Egg Supplier Interface.

Or do we? Let’s take another look.

The scenario can be rewritten as:

1. The customer rings telephone number and says “Cancel my egg!!!”.
2. If the customer’s egg is gone. Customer happy. Test passed.

But wait…

1. The customer rings telephone number and says “I changed my mind!! Don’t cancel my egg!!!”.
2. The operator looks on the order viewer, and stops the cancel. Customer happy. Test passed.

There’s a lot of other scenarios here but essentially we’re only using two interfaces - the Order System in the first case and the Order View in the second case. We don’t care about the supplier in these cases because it doesn’t matter for the product - you need to be able to order and call up and find out what’s going on with your order - that’s it.

So, now we’ve defined these interfaces - it’s easy to re-write the end-to-end testing infrastructure:

Test of much as the real system as possible

Back to the thing we learned early on - end to end tests test as much of the real system as possible. In the particular project I’m working on this is the hard part, the application is deployed across multiple containers, load-balanced, on hard-to-obtain environments etc… etc…

Time is a constraint here, so it’s possible to mock some of these things that don’t really matter and simulate a lot of the others. But now comes an important point - if you can’t test it - should you release it? For example, if it’s not possible to say simulate multiple containers, then is the application really production-ready, should it be deployed, is it essentially untested and therefore not safe? Probably not, but it’s something to definitely think about.