Active TopicsActive Topics  Display List of Forum MembersMemberlist  CalendarCalendar  Search The ForumSearch  HelpHelp
  RegisterRegister  LoginLogin


 One Stop Testing ForumTypes Of Software Testing @ OneStopTestingAutomated Testing @ OneStopTesting

Message Icon Topic: ROI in Test Automation Part2

Post Reply Post New Topic
Author Message
Mithi25
Senior Member
Senior Member
Avatar

Joined: 23Jun2009
Online Status: Offline
Posts: 288
Quote Mithi25 Replybullet Topic: ROI in Test Automation Part2
    Posted: 30Oct2009 at 11:45pm

This is the part2 of the ROI of test automation article. Please read the part1 first. 

Modularize the automation scripts.  Utilize an effective test automation coding strategy (with automation tools) of wrapping redundant navigation, data input and baseline verifications into function libraries, which modularizes scripts, prepares the scripts for changes to the application and minimizes the maintenance of scripts of future builds.

 

Train engineers in testing and test automation discipline.  Develop a hiring, training, and support model that includes engaging the appropriate resources, developing a rigorous training program and a building a successful mentoring/support model. It typically requires 6 9 months of targeted lessons, workshops, labs and assignments to fully train a functional test engineer and 9 12 months to fully train a performance test engineer.

 

Maintain your test suite.   Why spend all that time (and money) developing a robust suite of automation tests and then not support and maintain it? Test suites need to be maintained with each new build and release of an application. Plan for a script maintenance program by either creating a centralized test team that performs the work (see figure 9) or contracting out a firm who provides this service.   Maintenance of robust scripts typically requires 20% of the time of originally creating the automation scripts (assuming that major overhauls are not being done to the system-under-test).   By utilizing a centralized team or utilizing an outsourced service for maintenance, resources can be allocated more efficiently and your investment can be protected.

if Case Studies:

ROI - Case Study functional testing

Industry:  Insurance

Business Purpose:  Verify the functionality and validation of the base system for entering life insurance policies.

Number of Automation Scripts: 3,900

Time to run automation scripts: 12 business days

Initial cost: 557 people /days (2 engineers working concurrently built this suite)

Time to manually execute: 139 people/days (28 per day)

# of runs to break even:  4.01 runs (to date there have been > 15 runs)

ROI: Savings of time (127 business days on each run), improved customer satisfaction (126 critical defects found and addressed).

Details: This Fortune 500 client found automation so beneficial, they created a centralized testing group to leverage automation throughout their organization.  They implemented load, volume, stress and performance testing, along with rigorous functionality testing.  Testing has become an integral component to delivering to their internal customers high-quality products.

 

ROI - Case Study scalability/performance

Industry:  Brokerage

Business Purpose:  Scale on-line brokerage application to 20,000 concurrent users

Number of Automation Scripts:  22

Time to perform 1 set of tests:  1 hour per run  (1-user - 100 -  500 - 1,000  and then incrementing by 1k  until 15,000 users)

ROI: Found critical system errors in architecture components, database issues.  Multiple iterations allowed for a much more scalable architecture and application, enhancing the user experience.

Details:  This customers success also led them to form a centralized function for load and performance testing.  Due to the large volume of transactions that they rolled out to their very large client base, their test team was responsible for pinpointing bottlenecks and problematic areas of scalability before these applications went into production.  These tests were a pre-emptive solution for providing quality software that scales to meet the performance requirements of a customer base that has many available competitive choices, should the online systems not meet the customers needs.

 

Example of Calculation

Lets assume you are designing software to be run on your clients desktop.  The architecture is web on the client side performing complex transactions.  To appeal to the greatest market, you have decided to support Internet Explorer 5.0 and 6.0, Windows NT (sp6), 2000, and XP.  Lets assume you have identified 500 scenarios (transactions) that test functionality and verify data throughout the system.  Your choice is manual testing versus automation. (We will assume that the choice ‘not to formally test is illogical because it would only push large quantities of defects to production- i.e. your clients, and eventually force them to go to another vendor, effectively putting you out of business). 

 

 

Item

Manual            Tester 

           

Manual Cost

Automated Tester

Automated Cost

Cost

$75 per hour

 

$75 per hour

 

Time to author 1 test case

1 hour per or 7 per day

 

1 hour per or 7 per day

 

Time to author 500 test cases

71.4 days

$42,840

71.4 days

$42,840

Time to automate

0 days

0

100 days

$60,000

Time to execute and analyze EACH test case and results

3 per hour or 24 per day

 

20 per hour or 240 per day (depending upon the speed of the processor and the speed of the application)

 

Time to execute and analyze ALL test cases and results

20.8 days

$12,480

2.1 days

$1,260

Total run through the code release for 1 release

 

$55,320

 

$104,100

Time to test for each subsequent release

20.8 days1

$12,480

Rework on scripts at 5% or 5 days1

$3,000

Assume new code releases 1x per month or 12 per year

249.6 days

$149,760

Rework on scripts at 5% or 60 days

$36,000

Total for 1 year

 

$205,080

 

$140,100

Cost of testing on all pertinent combinations (IE 5.0 & IE 6.0 with Win NT, 2000, XP)

All tests run on 61 configurations for each code change.

 

 

 

$953,880

 

 

 

 

$320,100

1 Assumes no additional test cases are authored or automated.

 

The savings in this example would be a tremendous time savings, which would equate to $633,700 .
 

Evaluate your application as a candidate for automation….

IF you answer YES to any of these questions, your application is a good candidate for automation based upon a return on investment.

Ø Does your application need to be tested on multiple hardware or software configurations?

Ø Will you have more than 5 new builds/patches/fixes of the application?

Ø Is a vendor developing the application for you and needs to meet Service Level Agreements?

Ø Do you have more than 5 concurrent users on your application?

 

if Ø Do you have repetitive tasks that are performed to maintain an application (i.e. data loading, configuration, etc.)?

Conclusion: 

The return on investment for test automation is quite obvious. If performed correctly, the automation suite will prove to be much more efficient than manual testing in finding defects on the functional side and the only way of finding scalability/load/performance issues on the multi-user side.  It can be run at night, on weekends and holidays and can be left unattended.  The tools never get bored or tired and never assume the application/architecture works.  And it can emulate as many users as can be anticipated accessing the application, performing any mix of transactions needed.

if Therefore, the ROI, (tangible + intangible benefits/initial cost) of automated testing provides a tremendous return, as long as the 6 reasons for failure are overcome.




Post Resume: Click here to Upload your Resume & Apply for Jobs

IP IP Logged
Post Reply Post New Topic
Printable version Printable version

Forum Jump
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot delete your posts in this forum
You cannot edit your posts in this forum
You cannot create polls in this forum
You cannot vote in polls in this forum



This page was generated in 0.109 seconds.
Vyom is an ISO 9001:2000 Certified Organization

© Vyom Technosoft Pvt. Ltd. All Rights Reserved.

Privacy Policy | Terms and Conditions
Job Interview Questions | Placement Papers | Free SMS | Freshers Jobs | MBA Forum | Learn SAP | Web Hosting