Topic: The importance of automated testing

Software development today is generally very complex. Large pieces of software can reach millions of lines of code, developed by many developers with varying skill and technique. Quality Assurance, or QA, is a huge job. In the most ideal situation, every possible path of execution is checked. Only in this circumstance is the code fully tested. But this is very difficult to achieve. The number of possible code paths is typically immense. While appropriate for NASA and nuclear powe r plants, the development cycle of a commercial software tool is often too short to aim that high. As a second best alternative, we can aim for complete code coverage, in other words, every line of code is tested.

Obviously, it is impractical to aim for complete code coverage when you are doing QA by hand. The solution is automated testing. Automated testing allows you to retest your application every time you commit a change to version control. This level of automated testing helps to ensure that bugs are never introduced into already working code. Typically the deepest levels of automated testing are used for libraries, where the consequences of bugs are greatly amplified.

Since I started working at my current job last summer I have been working on a single project, written in C#. The project is a compliance testing suite for a proprietary protocol. Eventually (shortly, in fact), a trade association will use the tool we wrote to test vendor’s products. And, if they pass, they get to put a certain logo on their packaging when they sell their product. Many of the tests our tool runs against devices are automated, but some require user intervention. You might think our test tool would use some nice automated testing, particularly for its networking libraries, but unfortunately it does not. The result is less than pretty. During the development cycle we constantly encountered problems with libraries that were part of our tool. Unfortunately, I was on the wrong side of those libraries.. and frequently found myself having to find and help fix bugs in them. Since we had no automated testing, things would often get broken later due to a new feature or code change and the problem wouldn’t be found for weeks.

What is the better way? All of the libraries need to have automatic test suites. This vastly simplifies the QA process which frees up resources to do some real QA and find the more difficult issues. The end result is simply better software. Soon, I will be moving to a different project at work and apparently I’m going to be taking on the role of writing an automated test suite for parts of the core API. This is the way to do it. Unfortunately we are on a tight release schedule and there is no way we’ll even get to test the entire API, let alone shoot for complete code coverage. Ideally the testing process should begin very early on.

Some software projects are even using a test driven development model. Authoring a test for a particular API function has several effects. First, it forces the developer to think in depth about how API function should behave. Particular attention is paid to error handling. Secondly, with the test written first, the function can be tested right from the beginning. There is no QA “catch up” which never works. Instead, the code is always tested. Of course, the automated tests are still no substitute for manual, in depth QA, but they free up resources and catch the stupid bugs sooner.

The most typical type of automated test today is the unit test. In unit testing the focus is on writing tests that test small sections of code. Each unit test tests a particular part of the program. The unit tests also provide a form of documentation as they indicate what the function of the code is. While unit tests ease integration by making sure all the components work, they do not test the integrated product as a whole.

Unit testing has become more and more popular with the rise of the “Extreme Programming” development style and the corresponding increase in test before development. The Ruby on Rails framework actively encourages the authoring of tests before development. For duck typed languages like Python and Ruby runtime testing is more important because compile time type checks are not performed.

I recently read some advice online about honing your programming skills prior to applying for a job by adding unit tests to open source projects. I think this is great advice. Most open source projects would welcome any effort put towards testing as it is often very useful. Wikipedia lists over 40 unit testing frameworks in just about every language. You can choose a small open source project to get your feet wet. Then, you have something to put on your resume. And, it is a specialized skill in demand, too!