Topic: Automation Tools - When to use them and how best t

When to use Automation tools like winrunner, rational robot etc.?
Automating test cases and maintaining them requires significant effort, so before investing both money and effort in it, do some cost-benefit analysis.Ask yourself following questions. To answer them, you may need to do some small pilot project on automating some test cases and running them using the preferred tools. 

After automating the test case, how many times you will use/run the automation script for testing, before dumping it forever or modifying/upgrading it for future Releases?
It takes roughly 3 times the effort to automate a test case than to manually execute it once.The more times you run/use the automation script without modifying it, the more will be the ROI(Return on Investment).Atleast you should be able to use it 5 times.So don't go for automation tools if the product is a short term product which doesn't involve many rounds of testing.

Is 'Time spent running the automation script' + 'Analysing the test results' less than 'Time spent on manual execution'?
Usually a automated script runs faster than manual execution.There are exceptions to this assumption. Also we first run the automation script completely and then analyze the test results, which can take considerable time, but when we manually execute, we don't need separate time to analyze the results,as we know what actions caused a check point to fail. Where as in the former case, we need to do some investigation as to what actions resulted to failing a check point.
So the 'Time spent on running the automation script' + 'analysing the test results' must be less than the 'Time spent on manual execution'
Ofcourse, if the automation scripts are well developed, don't require constant monitoring and run without manual intervention, then you can kick start them at night, come back and check the results in the morning. This way you can cut the time on running the scripts

Is it resuable for future releases of the product?If Yes how much effort will be required to upgrade the automation script?
The ROI will be more, if with little change, you can completely reuse the automation script.

Is the Product stable or does it keep changing quite often because of which the automation script needs to modified?
Never develop automation scripts on products that are already not stable. An unstable product keeps changing quite often resulting in the change of the automation script.

Are the test cases fit to be automated?
Certain test cases are not fit to be automated. Identiy them and don't automate them.The criteria on how to identify such cases change from product to product.I will share an example from my own experience. We automated a complex test case, which took long time to run. Also the product is an highly integrated application, so the test case often jumps from one application to another.The product's performance was not consistent.Adding to this, if somebody changed some settings of one application, then sometimes the test case came to a sudden stop.If the automation script stopped running because of any of the above mentioned issues, then because of the nature of the test case and the way automation script
was developed, we have to start running again right from the beginning. So the end result was, the time taken to run the automation script was way more than the time taken to execute it manually.
Fit test cases when automated run from end to end without any manual intervention and verify all the check points they intend to. It partly depends on how the automation script is developed.

    How best to use automation tools?
     Additional tips to make best use of automation tools

Use automation tools for performing tedious and repetitive tasks, so that the testers can concentrate on more important tasks.One example is for performing Sanity testing

Use automation tools to test the breadth of entire product/application being tested. Leave the job of doing in-depth testing to test engineers and manual execution

Don't automate highly complex test cases.Its better to execute them manually.

Automation scripts whose execution time is less than 2 hours often run smoothly.

Don't completely forgo manual testing.Automation scripts do exactly what they are programmed do, they don't deviate.That is their advantage and disadvantage. After first few rounds of testing, if you run the same set of test cases, you won't be catching many bugs, to catch more, we need to deviate from the flows and that we humans can do while doing manual testing.

Edited by Mithi25 - 08Sep2009 at 11:29pm