Topic: ROI in Test Automation Part2

This is the part2 of the ROI of test automation article. Please read the part1 first. 

Modularize the automation scripts.  Utilize an effective test automation coding strategy (with automation tools) of wrapping redundant navigation, data input and baseline verifications into function libraries, which modularizes scripts, prepares the scripts for changes to the application and minimizes the maintenance of scripts of future builds.

 

Train engineers in testing and test automation discipline.  Develop a hiring, training, and support model that includes engaging the appropriate resources, developing a rigorous training program and a building a successful mentoring/support model. It typically requires 6 9 months of targeted lessons, workshops, labs and assignments to fully train a functional test engineer and 9 12 months to fully train a performance test engineer.

 

Maintain your test suite.   Why spend all that time (and money) developing a robust suite of automation tests and then not support and maintain it? Test suites need to be maintained with each new build and release of an application. Plan for a script maintenance program by either creating a centralized test team that performs the work (see figure 9) or contracting out a firm who provides this service.   Maintenance of robust scripts typically requires 20% of the time of originally creating the automation scripts (assuming that major overhauls are not being done to the system-under-test).   By utilizing a centralized team or utilizing an outsourced service for maintenance, resources can be allocated more efficiently and your investment can be protected.

if Case Studies:

ROI - Case Study functional testing

Industry:  Insurance

Business Purpose:  Verify the functionality and validation of the base system for entering life insurance policies.

Number of Automation Scripts: 3,900

Time to run automation scripts: 12 business days

Initial cost: 557 people /days (2 engineers working concurrently built this suite)

Time to manually execute: 139 people/days (28 per day)

# of runs to break even:  4.01 runs (to date there have been > 15 runs)

ROI: Savings of time (127 business days on each run), improved customer satisfaction (126 critical defects found and addressed).

Details: This Fortune 500 client found automation so beneficial, they created a centralized testing group to leverage automation throughout their organization.  They implemented load, volume, stress and performance testing, along with rigorous functionality testing.  Testing has become an integral component to delivering to their internal customers high-quality products.

 

ROI - Case Study scalability/performance

Industry:  Brokerage

Business Purpose:  Scale on-line brokerage application to 20,000 concurrent users

Number of Automation Scripts:  22

Time to perform 1 set of tests:  1 hour per run  (1-user - 100 -  500 - 1,000  and then incrementing by 1k  until 15,000 users)

ROI: Found critical system errors in architecture components, database issues.  Multiple iterations allowed for a much more scalable architecture and application, enhancing the user experience.

Details:  This customers success also led them to form a centralized function for load and performance testing.  Due to the large volume of transactions that they rolled out to their very large client base, their test team was responsible for pinpointing bottlenecks and problematic areas of scalability before these applications went into production.  These tests were a pre-emptive solution for providing quality software that scales to meet the performance requirements of a customer base that has many available competitive choices, should the online systems not meet the customers needs.

 

Example of Calculation

Lets assume you are designing software to be run on your clients desktop.  The architecture is web on the client side performing complex transactions.  To appeal to the greatest market, you have decided to support Internet Explorer 5.0 and 6.0, Windows NT (sp6), 2000, and XP.  Lets assume you have identified 500 scenarios (transactions) that test functionality and verify data throughout the system.  Your choice is manual testing versus automation. (We will assume that the choice ‘not to formally test is illogical because it would only push large quantities of defects to production- i.e. your clients, and eventually force them to go to another vendor, effectively putting you out of business). 

 

 

Item	Manual            Tester	Manual Cost	Automated Tester	Automated Cost
Cost	$75 per hour		$75 per hour
Time to author 1 test case	1 hour per or 7 per day		1 hour per or 7 per day
Time to author 500 test cases	71.4 days	$42,840	71.4 days	$42,840
Time to automate	0 days	0	100 days	$60,000
Time to execute and analyze EACH test case and results	3 per hour or 24 per day		20 per hour or 240 per day (depending upon the speed of the processor and the speed of the application)
Time to execute and analyze ALL test cases and results	20.8 days	$12,480	2.1 days	$1,260
Total run through the code release for 1 release		$55,320		$104,100
Time to test for each subsequent release	20.8 days1	$12,480	Rework on scripts at 5% or 5 days1	$3,000
Assume new code releases 1x per month or 12 per year	249.6 days	$149,760	Rework on scripts at 5% or 60 days	$36,000
Total for 1 year		$205,080		$140,100
Cost of testing on all pertinent combinations (IE 5.0 & IE 6.0 with Win NT, 2000, XP)	All tests run on 61 configurations for each code change.	$953,880		$320,100

1 Assumes no additional test cases are authored or automated.

 

The savings in this example would be a tremendous time savings, which would equate to $633,700 .
 

Evaluate your application as a candidate for automation….

IF you answer YES to any of these questions, your application is a good candidate for automation based upon a return on investment.

Ø Does your application need to be tested on multiple hardware or software configurations?

Ø Will you have more than 5 new builds/patches/fixes of the application?

Ø Is a vendor developing the application for you and needs to meet Service Level Agreements?

Ø Do you have more than 5 concurrent users on your application?

 

if Ø Do you have repetitive tasks that are performed to maintain an application (i.e. data loading, configuration, etc.)?

Conclusion: 

The return on investment for test automation is quite obvious. If performed correctly, the automation suite will prove to be much more efficient than manual testing in finding defects on the functional side and the only way of finding scalability/load/performance issues on the multi-user side.  It can be run at night, on weekends and holidays and can be left unattended.  The tools never get bored or tired and never assume the application/architecture works.  And it can emulate as many users as can be anticipated accessing the application, performing any mix of transactions needed.

if Therefore, the ROI, (tangible + intangible benefits/initial cost) of automated testing provides a tremendous return, as long as the 6 reasons for failure are overcome.