Topic: Test Automation Best Practices Part 1

Know your objective

There are many good reasons for doing test automation.

Test automation can save time, make testing easier, and improve the testing coverage. It can also help keep testers motivated; I can list a page of other benefits that can be derived from doing functional test automation. However, it is not likely that your organisation will need to do all these things at the same time. Different groups typically have different hopes and ideas of what they want test automation to offer them. These need to be stated, or else disappointment is likely.

A clear objective to work towards during test automation is highly important. It provides us with a compass heading towards which planning and implementation will be directed. Eventually, the results of the test automation implementation will be compared against the original objective to establish whether we did reach our goal or not.

Mistakes made...

"Test automation is the answer to our testing problems"
Test automation assists in the testing process, and can therefore provide value to a test team. Certain benefits can be derived, but we need to know and understand exactly what they are without having false expectations or hopes. Automation is not a replacement for an inadequate test process or an inefficient test team. Address the testing problems prior to embarking on a costly test automation implementation. Automation's objective is definitely not to solve all testing problems in existence.

"Since we have limited time to test, let's use test automation."
Test automation takes 3 to 128 times longer to implement than manual testing, and therefore it should be properly planned to achieve the expected benefits. If time is of the essence, consider adding more resources to the test process (this also has limited benefits, and might not always be a solution) rather than embarking on the lengthy process of implementing test automation. Functional test automation is definitely not a quick solution to a test project in distress. Multiple other solutions are better suited to address this need.

Test Automation requires a manual test process

"Success in test automation requires careful planning and design work, and it's not a universal solution. ... automated testing should not be considered a replacement for hand testing, but rather as an enhancement." (James Bach [pg. vii] in his foreword: Software Testing with Visual Test 4.0, Thomas Arnold II, IDG Books Worldwide Inc. Foster City, CA.: 1996. ISBN 0-7645-8000-0.)

The words of James Bach and many others are quite clear on this subject. Test automation will not give you a correct testing process or methodology. It will not enforce a methodology or process. Test automation merely supports the test process; it will not replace the manual test process.

Mistakes made ...

"We need to implement testing in our projects. Let's buy a tool and automate the testing."
You can script to your heart's content, if you do not have a proper manual test process in place the benefits of test automation will never be seen. It will only assist you in doing the incorrect process a lot faster. Test automation merely assists and enhances the test process itself. Ensure that a proper test methodology is implemented before attempting to add test automation.

"We can just add the test automation to our current test process and start automating our current test scripts."
Test Automation requires test cases to support the automation process. If my test cases do not incorporate the all-important logon to the SUT (system under test), the automation script will need a lot of manual intervention, and would therefore be more manual than automated. The current test process might have to be changed, or adapted, to be implemented for the tool suite selected. Review and adapt the test process to ensure maximum benefit is realised from the tool suite to be used in your test process.

The earlier the involvement in the Software Development Life Cycle (SDLC) the greater the benefits

We all know how testing has its own phases aligned with the phases of the SDLC. The earlier testing gets involved in the life cycle of the project the better. Many articles have been written on this subject, and it holds true. Since test automation supports the testing process, the test team can get involved very early in the life cycle of a project. If the test team uses a test automation tool suite, they will most likely have a requirements management tool or test management tool included in the suite. These tools can assist the test team in various ways from an early stage in the development life cycle, even before a line of code is developed.

The more mature techniques and methods of test automation are closely tied to the testing methodology being used. An early start to the test life cycle, at the start of the development project life cycle, is vital to achieving the best results, and ensuring maximum return on investment.

Mistakes made…

"The test automators cannot get involved in the project before the developers release the first build."
A software development project does not start with code development. Since functional test automation is also a development project, as we will discuss in Best Practice 9, it follows the same phases as a software development project and does not start at the coding phase. Proper planning and design are essential to the success of the functional test automation implementation, and need to be conducted hand-in-hand with the test life cycle.

"Testing is only a single phase in the SDLC and therefore testers and test automation only need to be part of the project for the testing phase."
This 'prehistoric' misconception in the IT industry has been proven one of the major reasons for bad software quality.

Test Automation is a fulltime effort

When people are allowed to work on test automation on their own time, or as a back burner project whenever the test schedule allows, test automation becomes a sideline effort. Test Automation does not get the time and focus it needs.

Getting into automated testing requires preparing not only yourself, but also your company and environment. This requires focus and dedicated resources. {mosgoogle}

Mistakes made…

"We have a test automation tool. Can the test team see if they can use it?"
It is clear that the expectation and an understanding of test automation is lacking. I can guarantee that the tool implementation will not be successful, and would result in almost no benefit to the test team. The test team is usually confronted with manual test work for development projects that does not meet the code delivery date for testing. The test automation implementation would probably be of the capture-record-and-playback form, and therefore the benefit would be low.

Test Automation will require training for the staff involved with it. Time is required to design and implement the architecture. Referring to the development nature of functional test automation implementation, the team will have to learn a new skill set. Test automation implementation requires dedication and focus to ensure success. In this instance, it is better to not implement test automation than to allocate ad hoc time for experimentation.

Select the correct product or suite of products

Buying the wrong tool is listed as one of the major challenges in test automation, because - no matter what kind of process, methodology, or organisation you have - if the tool is not a good technical and business fit, it will not be used.

We know that a good process and organisation are also essential for test automation. However, if the tool will not function at a basic level, the people who should use the tool, and thereby gain the benefit from the tool, will not use it.

Unfortunately, too few people do adequate research before buying a test tool. Adequate research includes defining a set of tool requirements based on what the intended users of the tool need to accomplish, developing a set of evaluation criteria by which candidate tools will be judged, and taking the experience of other people who have used the tools into consideration.

Select a tool that will allow you to implement automated testing in a way that conforms to your long-term testing strategy and test methodology.

Mistakes made…

"A vendor demonstrated a tool to us and it seems as if the tool will add value. I think we should buy it."
Take time to define the tool requirements in terms of technology, process, applications, people skills, and organisation. Then look at multiple tool vendors and select the right fit for your circumstances. A good idea would be to do a proof of concept project with the tool that best fits your criteria.

"We purchased the whole suite of tools from the vendor, but are only using the functional test automation tool."
Test Automation tools are expensive, and to justify the Return On Investment (ROI) it is normally not effective and efficient to use only the functional test automation tool in the product suite. Other benefits can be derived from using the full suite of products; for example, proper defect management, metric reporting on test status, etc. Purchase what you need, or use what you have fully.

"The yearly license fees are so expensive we cannot afford to use the tool suite any more."
Too often test automation tools end up as 'shelf-ware' because of the cost of the licenses compared to the benefit derived. Refer to mistake two above for one of the reasons why the ROI is not being realised. Before signing the purchase order, be aware of the impact of annual licensing fees.

Get executive management commitment

Test automation can yield very substantial results, but requires a substantial commitment to be successful. Do not start without commitments in all areas, the most important being executive management.

'Thomas R. Arnold II, VP at ST Labs, Inc., sums it up fairly well:
"I would like to say up front ... that no test automation tool is a silver bullet. Automation takes time, effort, and commitment from all involved, including an understanding from management about the realities of what automation can and cannot do." ' 1
Management support is needed in designing and deploying test processes that will support the effective use of test tools, reinforce the role and use of automated test tools in the organisation, and allow time for tools to be integrated in the testing process.

Without management support, the entire test automation effort is at risk. If management does not - clearly and consistently - show their support for test automation, people will be less inclined to show interest in using the tools. This is a major concern, especially considering that overcoming the learning curve of some tools requires dedication.

Perhaps the greatest challenge seen in management support is balancing the high expectations of tool benefits against the time, effort, and discipline it takes to implement the tool. Management may become impatient about the lack of tool progress and shift their support to other initiatives.

Mistakes made…

"We thought these tools were going to solve your testing problems."
When making the case to management for acquiring test tools, present the challenges as well as the benefits. Make sure that management understand their influence on how others will accept automated test tools.

"We purchased a tool for you 3 months ago, and yet we cannot see any benefits in the project."
Communicate to management from the start that it takes time and planning to build a firm foundation of people, processes, and the right tools. Keep management informed of tool progress, and any issues that arise