• Test cases must have a clear and differentiated scope – It is important that test cases have a clearly differentiated scope and that they not deviate from that scope.
• The tests must be written at the right level of abstraction – Tests must be written at the right level of abstraction such as the higher business level, lower user interface level, or both. It is also important test tools provide this level of flexibility.

The implementation of this methodology is framework dependent. This framework requires the development of data tables and keywords, independent of the test automation tool used to execute them and the test script code that "drives" the application-under-test and the data.

If we were to map out the actions we perform with the mouse when we test our Win calculator functions by hand, we could create the following table. The "Window" column contains the name of the application window where we're performing the mouse action (in this case, they all happen to be in the Calculator window). The "Control" column names the type of control the mouse is clicking. The "Action" column lists the action taken with the mouse (or by the tester). And the "Arguments" column names a specific control (1, 2, 3, 5, +, -, and so on).

Methodology

The keyword-driven testing methodology divides test creation into two stages:

·         Planning Stage

Analyzing the application and determining which objects and operations are used by the set of business processes that need to be tested.

Determining which operations require customized keyword to provide additional functionality, to achieve business-level clarity, and/or to maximize efficiency and maintainability.

·         Implementation Stage

Building a collection of references that uniquely identify objects, sometimes known as an "object repository", and ensuring that all such references have clear names that follow any predetermined naming conventions. (This is primarily for test automation.)

Developing and documenting business-level keywords in function libraries. Creating function libraries involves developing customized functions for the application that needs to be tested.

Although this methodology requires more planning and a longer initial time-investment than going directly to the test creation stage and recording your steps, this methodology makes the test creation and test maintenance stages more efficient and keeps the structure of individual tests more readable and easier to modify.

While a keyword-driven methodology is mostly useful for automated testing, some manual testing systems like Mercury Interactive's TestDirector for Quality Center allow a similar procedure for manual testing as well.

Vision for Automation

It is also important to have a clear vision for automation. Such a "vision" should include things such as:

• Having a good methodology – It is important to have a good integrated methodology for testing and automation that places testers in the driver’s seat. It is also important to employ the best technology that supports the methodology, maximizes flexibility, minimizes technical efforts, and maximizes maintainability.

• Have the right tool – Any tool that is employed should be specifically designed for keyword based testing. It should be flexible enough to allow for the right mix of high and low level testing. It should allow the testers to quickly build keyword tests, without difficulty. It should also not be overly complicated for automation engineers to use when implementing the automation.
• Succeed in the three "success factors for automation" – There are three critical success factors for automation that the vision should account for. They are:
1. test design
2. automation solution
3. organization

Success Factors for Automation

Test Design

Test design is more important than the automation technology. Design is the most underestimated part of testing. It is my belief that test design, not automation or a tool, is the single most important factor for automation success. To understand more about test design see these previous articles:

• Key Principles of Test Design
• The First Holy Grail of Test Design
• The Second Holy Grail of Test Design
• The Third Holy Grail of Test Design

Comprehensive Automation Architecture

• Scope, assumptions, risks
• Methods, best practices
• Tools, technologies, architecture
• Stake holders, including roles and processes for input and approvals
• …and more…

The "right" team must also be assembled. This team should include:

• Test management who is responsible for managing the test process.
• Test development who is responsible for production of tests. Test development should include test leads, test developers, end users, subject matter experts, and business analysts.
• Automation engineering who is responsible for creating the automation scheme for automatic execution. Members of this team include a lead engineer as well as one or more automation support engineers.
• Support functions, providing methods, techniques, know how, training, tools, and environments.

For the team there should be a clear division of tasks and responsibilities as well as well defined processes for decision making and communication.

How to Measure Success

With any major undertaking, it is important to define and measure "success". There are two important areas of measurement for success – progress and quality.

Progress

You should measure test development against the test development plan. If goals are not reached, act quickly to find the problems. Is the subject matter clear? Are stake holders providing enough input? Is it clear what to test (overall, per module)? Is the team right (enough, right skill set mix)?

You should measure automation and look at things such as implemented keywords (actions) and interface definitions (defined interface dialogs, pages, etc).

You should measure test execution looking at things such as how many modules are executed and how many executed correctly (without errors)?

Quality

Some of the key quality metrics include:

• Coverage of system and requirements
• Assessments by peers, test leads, and by stake holders (recommended)
• Effectiveness
• Are you finding bugs?
• Are you missing bugs?
• Can you find known bugs (or seeded bugs)?
• After the system is released, what bugs still come up? You should consider calculating the "Defect Detection Percentage" (Dorothy Graham, Mark Fester)
• Mine your bug base for additional insights

Conclusion

It is important to understand that keywords are not magic, but they can serve you well. What is more important is to take the effort seriously and "do it right". Doing it right means that test design is essential, both global test design and the design of individual test cases.

Automation should be done but it should not dominate the process. Automation should flow from the overall strategy, methodology, and architecture. It is also very important to pay attention to organization – the process, team, and project environment.