Topic: Functional testing

So, we have a public web application accessible 24x7. Even if we do not process financial transactions or control airport traffic, this poses a kind of stress upon us. To relieve it, functional testing is our true companion.

By functional testing I mean simulating an end user's traversal through the application. This means that, contrary to unit tests, we are testing what the user sees, not what the developer has written. The tests are thus less sensitive to software design changes, internal bug fixes, and even URL refactoring.¹

As a tool, we have chosen Watir, an open-source library for controlling a web browser. It is light-weight and it does what it says on the box: "drives the ... browser the same way people do. It clicks links, fills in forms, presses buttons... also checks results, such as whether expected text appears on the page."

Technically, Watir is a library in Ruby² for controlling Microsoft Internet Explorer on Windows (with support for Linux and Firefox planned in the next major release). The tested web application may be anything (ASP, JSP, PHP, etc.) producing browsable web pages.

Writing tests in Watir is fun in itself. Firstly, you can see the page in the browser and watch the elements being manipulated by an invisible user - buttons, links etc. get highlighted when the test suite accesses them. Secondly, you can even use Ruby console to write the tests interactively so that you command the browser like a starship.

Test scenarios are needed to cover the whole functionality landscape, and to agree with the developers, what the application should do. In theory, this should completely be done at the beginning of the application lifecycle, but in an agile approach (as we had it with Recykl.com), you just define the main use cases at the beginning and fine-tune the details based on end user/customer feedback. Anyway, having the scenarios automated in tests is a good validation of the functional design.

Test data for the functional tests are indispensable for the predictability of the test results. In our case, another important requirement was isolation of the data for the individual tests, as we decided to use just one suite of test data for the whole suite of functional tests. For faster test execution the data is loaded once, not at the beginning of each test in the suite (which is a common Ruby test fixtures practice). To achieve isolation, we successively run the tests under different users, so the risk of interfering is reduced to a minimum.

Now, after some time of using Watir for the testing, there are some lessons learned:

1. the most painful and unpredictable errors result from timing, e.g. in asynchronous AJAX calls - sometimes it's difficult even for a human to tell if we are still waiting for something to happen or that's all
2. use some kind of automated reporting of results (unfortunately Watir currently has a very limited support here)
3. tests are not carved in stone - you will have to update them as your functional design evolves (and it does in a successful application) - so write them flexibly. For example, do not rely on details like exact wording of a link, but instead use a regular expression matching just the main keyword, e.g. /[Ss]ubmit/
4. even though it takes long to run the tests, try to run them after each major changeset - you'll catch bugs and also easier keep up with the intentional changes

Having that in mind, doing functional testing with Watir has significantly increased our confidence in the functioning of our web application and saved us some hot moments in its (sometimes stormy) agile lifecycle.