Select the appropriate technique/s for the project

As maturity levels improve in the functional test automation field, we see a decrease in cost for test automation implementations, and an increase in benefits derived for the project or organisation.

The increase in maturity levels has provided us with more mature techniques for implementing test automation. These techniques vary from the simplest record-and-playback to the most mature, viz., keyword or action based. These techniques each have their own benefits and drawbacks, appropriate for certain specific circumstances.

During a small data conversion project, a few lookup data tables' contents in the old system need to be compared to the new lookup data tables' contents after the batch conversion. If this can be done via the enquiry functions of both systems, the fastest method might be record-and-playback. This is a once off test of the conversion, and writing specific scripts to do this might take longer than the record-and-playback method. In terms of cost and timing, the record-and-playback technique would be the most appropriate for this project. This is one of the few instances where I would suggest using the record-and-playback technique; most of the time the benefits of this technique are few compared to the more mature techniques like data-driven and keyword test automation.

It is important to know, and be able to use, multiple techniques appropriately. If we do not use the various techniques when appropriate, our test automation will not provide us with the benefits we are aiming for.

Mistakes made

"Record-and-playback is sufficient for our needs."
The benefits of this simplistic technique are few, and its application is widely misused. More mature techniques are available with exponentially more benefits in terms of ease of use, cost saving, and time saving if implemented correctly.

"We have created our test cases for the project and are approaching the release of the first build. Can we implement keyword test automation before the end of the project."
Keyword test automation is a very advanced test automation technique, and is closely tied to the testing methodology being used for the project. The preparation work for implementing keyword test automation requires more time than some other techniques. The framework needs to be established, and test design needs to support the method. I would not suggest implementing the method from scratch at this late stage of the project.

Do not attempt to automate all tests

Test Automation does not work well for all situations or tests. Some tests are better left for manual execution. If a test will not be repeated more than three times, it is probably not a good candidate to automate. It takes a lot longer to automate a test than to execute it manually. The benefit of the automated tests is linked to how many times it would be repeated. For example, multiple builds, or testing on multiple platforms, or executing the same test for multiple data sets.

Software that is tested manually will be tested with a randomness that helps find bugs in more varied situations. Since a software program usually will not vary, automated testing may not find some bugs that manual testing will. Automated software testing is never a complete substitute for manual testing.

Proper analysis needs to be done to identify the correct test cases to automate to ensure ROI for the effort. How to select tests for automation is a subject on its own, but the following suggestions can be helpful:

Perform a proof-of-concept (POC) on a cross section of test cases for the system to determine technical applicability and return on investment (ROI) metrics.
Perform an ROI analysis for each testing regimen to select automation candidates.
Perform a risk analysis on the automation candidates to prioritise what should be automated first.
If the tests selected for functional test automation are identified and prioritised, the test cases with the best potential to provide benefit will be automated first. If time constraints affect the test automation delivery, at least we know the largest possible benefit would be derived.

Mistakes made

"It was so much fun that we automated all the tests."
It might have been fun, but the ROI was definitely not optimal. Time spent on automation of certain tests would never be recovered. A general rule of thumb would be never to automate more than 60% of your test cases to ensure maximum benefit with the lowest investment cost.

Manage the automation as a development project

Test Automation development is a software development process, although it is seldom treated that way. Following software development practices can make the difference between the success and failure of an automated testing project.

We need to run test automation projects just as we do our other software development projects. The following pertains to test automation projects, and is true for development projects:

Projects need developers dedicated to developing test automation.
Test automation automates a task in which the programmer is probably not an expert. Therefore, expert testers should be consulted and provide the requirements.
Test automation benefits can be seen if we design our approach before we start coding.
Test automation code needs to be tracked and safeguarded. Therefore, we need to use source code management.
Test automation will have bugs. Therefore, we need to plan to track them, and test for them.
Users will need to know how to use it. Therefore, we need user documentation.
The type of work done during test automation is software code development, and since these are programs, they must be managed in the same way that application code is managed.
{mosgoogle}
Mistakes made

"When the code is complete, give it to the testers so they can implement test automation."
Why would we follow a phased approach to software development, but not with test automation script development? We should enforce the same stringent processes and phases on test automation scripters as on developersand the scripts need to be tested.

"You don't need to follow all the phases of the software development life cycle with test automation."
The fact that we are writing software to test software, and not to perform some business scenario, does not mean the method is different. What is important for the one is important for the other; both are software development projects.

Use the correct resources

One area consistently missed by organisations desiring to automate testing is the staffing issue.

Automated testing is different from other types of testing:

Most tools use procedural languages to create their scripts.
These tools are very similar to IDEs (Integrated Development Environments).
Debugging scripts is the same as debugging traditional software.
…and therefore, resources are going to need:

academic-level knowledge of procedural coding languages,
basic knowledge of coding practices and procedures, and
in-depth knowledge and practical experience with the tool being used for the test automation.
Because of this and because 'record-and-playback' (which generally does not require learning the back-end scripting) is not a valid way to automate most tests, the use of these tools becomes like that of a development effort. Given that such is the case, there must be support within the organisation to realise this and hire staff accordingly, as well as giving the necessary time for the development.

To accomplish this, a Test Automation Tool Specialist, or similar, position must be created, and staffed with at least one senior-level programmer. It does not really matter in what languages the programmer is proficient; what is important is that this person must be capable of designing, developing, testing, debugging, and documenting code.

Test Automation is a combination of testing and development. To mentor and teach your team and resources, use consultants as appropriate to bootstrap your effort. Learn as much as possible from theses consultants to enable you to avoid the stumbling blocks, and proceed successfully with your automation effort after they depart.

Mistakes made

"We sent our testers on a test automation course, and they still can't implement proper test automation scripts."
Learning the test tool Integrated Development Environment (IDE) does not make you a good scripter. A 3-day course will not teach you programming principles. Mentoring by a developer can accelerate the learning process. Add a person to the test team who is a test scripter. This person should be comfortable working with code, and be able to take the basic test designed by a test analyst and convert it into an automated script.

"The testers did attend the vendor training."
The vendor training will train your staff in the product. Teaching someone Microsoft's development studio will not make them a code writer or programmer. The person will merely know how to use the IDE to write code. It is going to require more than just vendor training to turn your staff into test automators. I suggest hiring a test automation consultant to assist with getting the test team out of the starting block. The consultant can train and mentor your staff, and teach them how to avoid the most common pitfalls. Start the team with simple basic scripting concepts, and add complexity later.

Develop for maintenance

Without proper automation architecture planning and design the test team will soon find itself with hundreds or thousands of test scripts, thousands of separate run result files/logs, and the combined work of maintaining the existing scripts for various versions of the system, as well as creation of new scripts for new enhancements. Soon they will find themselves with a maintenance nightmare.

To avoid the maintenance problems that so many organisations succumb to, the following pointers can help:

Create a reusable architecture.
Develop generic, cross application initialisation and configuration parameters.
Create reusable, cross application functions.
Design test scripts by assembling components.
Avoid the creation of disposable automation through proper planning and design of your test automation implementation.

Mistakes made

"We did not design with maintenance in mind."
If you did not start your test automation with a design for maintenance, you have probably already become a victim of the maintenance nightmare. Review your current scripts and determine what amount of script/code snippets can be reused if you implement a different approach (maintenance focused). If the amount of rework is large, it might be better to restart your effort with the correct approach in mind.

"No standards were followed in script creation."
This will also result in difficulty during the maintenance phase; or if the resource that created the scripts leaves for greener pastures. Coding and scripting standards allow for ease of maintenance for the whole team. Different people can understand each other's scripts with ease, and when someone leaves the team it is not the end of the world. When test automators return to scripts that they have not worked on for a while it is also easier to understand what they did the first time round. This leads to a reduction in script maintenance time.

Start to implement standards and good coding practices for the test automators or scripters in your project/organisation. It is never too late to start with this practice.

Review and improve implementation process after each project