Topic: Automated Testing - FAQs

Message

   Automated Testing - FAQs

This forum is for Automated Testing related general discussions, i.e.
how to get started using automated testing tools; automated testing
tool evaluations; which tool to use; what to automate; what not to
automate; when to automate; managing automated testing expectations;
automated testing strategies; when to use a vendor provided tool,
when to develop a tool in-house or look at open-source; and other experienced-based practices
and key concepts.

Before posting any questions, please review the following frequently
asked questions (FAQs). If you don't find your question here or
the answer is not to your satisfaction, please feel free to post a new
question.

Content of the Automated Testing FAQs, to date:

FAQ1: I am new to automated testing, where do I start?
FAQ2: Where can I find articles about Automated Testing?
FAQ3: Where can I find books about Automated Testing?
FAQ4: I have a question related to a specific tool, e.g. Winrunner, Robot, QARun, etc
FAQ5: Which is the best testing tool on the market?
FAQ6: Where can I find testing tool comparisons and evaluations?
FAQ7: What is a Test Harness? (also referred to as Test Oracles)
FAQ8: How can I manage false expectations that come with Automation?
FAQ9: What is the Return on Investment (ROI) for Automated Testing?
FAQ10: What is a Keyword or data driven framework?
FAQ11: Why automate?
FAQ12: When to automate?
FAQ13: What to automate?
FAQ14: How to get tool evaluation copies?
FAQ15: Testing tool suggestions for various development languages, i.e. Java, C++, DHTML?
FAQ16: Testing tool suggestions for SAP, PeopleSoft, Siebel, etc?
FAQ17: Testing tools for different Operating Systems and other environments
FAQ18: Where can I get tool training on-line?
FAQ19: Where can I get free testing tools?
FAQ20: Is 100 percent automation possible?
FAQ21: Links to popular scripting languages for testing
FAQ22:Open Source Test Automation

==================================================================
FAQ1: I am new to automated testing, where do I start?

If you are new to test automation you might want to start by reading the automation
articles or books mentioned in FAQ2 and FAQ3. For starters, you might want to read
the following posts and query for similar ones:

Automation will it fail me?
I'm new and I need your opinions about tools
I am new, please help
Steps for automated testing
Automators vs Manual Tester Ratios
Programming languages and testing

FAQ2: Where can I find articles about Automated Testing?

There are numerous test automation articles available. You might want to use a search
engine such as Google or start by reviewing the following posts:

Automated testing vs Human testing
Implementation of Automated Tools
Test Automation articles

FAQ3: Where can I find books about Automated Testing?

QALinks Books
The Best Books
Automated Testing reference books
Books about Automated Testing tools

FAQ4: I have a question related to a specific tool, e.g. Winrunner, Robot,
QARun, etc

Specific questions related to specific tools are to be posted under the relevant tool forum.
Scroll down on the main qaforums page and you will find forums for most all the major tools on the market.
Please post specific tool questions in the relevant tool forum, as this is a general automated testing discussion forum.

FAQ5: Which is the best testing tool on the market?

This question is asked often and the responses will vary, depending on the user's experience and other criteria.
See for example this post: Automated Testing Tools

It is important to evaluate the tool based on your system-engineering environment, available budget, schedules, skills available, and other
testing needs. Some tools work better in specific environments, while in another environment a tool can cause compatibility problems if for example one of the application's major grids/controls is not compatible with the specific tool. Therefore, if a vendor wants to demonstrate their tool, please ask them to demonstrate the tool using your application under test. Other criteria is listed in the Test Tool Comparison/Evaluation in FAQ6, next.

Sometimes you might have to develop a tool in-house, because none of the vendor provided tools are compatible with your system-engineering environment or for other reasons a test harness will have to be developed. See the related discussion below on Test Harnesses.

Other times you might want to take advantage of the many available open source testing tools, for example take a look at all such tools listed here

A related poll is here 
Best Automated Test Tool - 2005 

To be most effective, sometimes it will require a mix of vendor provided, homegrown, and possibly open-source testing tools.

FAQ6: Where can I find testing tool comparisons and evaluations?

see

Comparisons for the simple minded
How long to evaluate GUI-automation test tools
Document comparing automated test tools
Mercury or Rational?
Tool comparison
Which of the automated test tool has the best service
Automation Tool Criteria
Test Tool evaluation
Tool Evaluation

FAQ7: What is a Test Harness? (also referred to as Test Oracles)

Automated testing does not only involve using vendor provided testing tools, i.e. capture/playback tools, etc. or a modification of the tool's script generation/language - whether data driven or keyword driven - instead, it often requires the development of a test harness. For additional info, check this post: Test Harnesses

Also, see
Test Harness
Can Perl script be used for testing automation?
Integration Test with Test Drivers
Components of an ideal test harness
A non-GUI linux test harness
Test oracles

FAQ8: How can I manage false expectations that come with Automation?

Along with the idea of automated testing come high expectations. Although automated testing can produce a return on investment if implemented effectively, an immediate pay-back is not always achieved.
It is important to manage the high expectations that come with automated testing. Remember automated testing is just one tool of the many available in a tester's tool box. See the following post for related info:
Misconceptions of Mgmt

FAQ9: What is the Return on Investment (ROI) for Automated Testing?

Automated testing can provide several benefits when it is implemented correctly. The test engineer must evaluate whether the potential benefits fit the required improvement criteria and whether the pursuit of automated testing on the project is still a logical fit, given the organizational environment, i.e. tool budgets, schedules, skills available, etc. At times putting a dollar value to automated testing ROI can be difficult, since there are many variables to consider, i.e. upfront investment, training, developing the automated test scripts, how often are the automated testing scripts used, etc. Also review
the following posts:

Justify the use or purchase of the testing tools
Automated Testing Vs Manual - Time Saving?

FAQ10: What is a Keyword or data driven framework?

see
"Test-Plan" driven Method (Keith Zambelich)
Test engine
Key-Word Driven Methodology
Expected results for Data Driven tests
How Many Of You Are Using Table/Keyword-Driven Methodology?
Table/Keyword Driven (3rd Generation) Automation Questions

FAQ11: Why automate?

see
Why automate?
Is automated testing the answer?

FAQ12: When to automate?

see
When should I start using an Automated tool for Regression testing
Automated or Not automated that is the question
Manual Testing Vs. Automated Testing

FAQ13: What to automate?

see
How much automated testing?
Criteria for Automated vs. Manual Testing

FAQ14: How to get tool evaluation copies?

see
Evaluation copy...........
More info to follow

FAQ15: Testing tool suggestions for various development languages, i.e. Java, C++, DHTML?

see
What is the best automation tool for Lotus Notes/Domino, VB and Java
Best ActiveX Control recognition in Automated Tool!!
Testing VB.Net Applications?
Unit testing tools for VB.Net
Best tool for Java
Automated Testing - FoxPro
Automation Tools to run against Apps with Stingray Controls

FAQ16: Testing tool suggestions for SAP, Siebel, etc?
Mercury is the leader in this market..but Rational/IBM tools now also are SAP compatible, etc.
placeholder

FAQ17: Testing tools for different Operating Systems and other environments

There are specific forums for Macintosh and Unix - scroll down on the main page.
There is also a separate forum for PDAs and embedded systems - scroll down on the main page.
I've found that few tools are available for the Linux environment and of the few tools that are available, i.e. for example Rational Functional Tester, the capture portion of it's capture/playback features won't be usable. In my experience when it comes to Linux functional test automation (note: not performance testing) it is best to use open source, or scripting languages such as Perl or Python, etc., not any of the current crop of vendor provided tools (unless perhaps your application runs Java on Linux)

Also review the following
Test automation tools for PDAs
Best tool to test mainframe
Test tool best for AS400
Linux Automation Tool

FAQ18: Where can I get tool training on-line?

see

Qatraining.net provides on-line training
Also see this related discussion
Tutorials on WinRunner

FAQ19: Where can I get free testing tools?

see

Free GUI test tools
Open Source Test Suites

For evaluations of free testing tools see Danny Faught's Open Testware Reviews

FAQ20: Is 100 percent automation possible?

Actually since 100 percent of testing is generally not possible given the time and budget constraints, so is 100 percent automation not possible. Mostly it's not feasible and most likely not cost-effective see

Is 100 percent automation possible?

FAQ21: Links to popular scripting languages for testing

Ruby
Python

FAQ22: Open Source Test Automation
Many open source test automation tools are available, see Opensourcetesting - I have spent a lot of time evaluating the STAF framework - see The Software Test Automation Framework and from what I have seen, i.e. the capabilities it provides, etc. it looks very impressive. I will keep automation forum members posted on actual implementation and additional findings.

Author	Message
tanushree Senior Member Joined: 04Apr2007 Online Status: Offline Posts: 2160	Topic: Automated Testing - FAQs Posted: 03Oct2007 at 12:30am
	Automated Testing - FAQs This forum is for Automated Testing related general discussions, i.e. how to get started using automated testing tools; automated testing tool evaluations; which tool to use; what to automate; what not to automate; when to automate; managing automated testing expectations; automated testing strategies; when to use a vendor provided tool, when to develop a tool in-house or look at open-source; and other experienced-based practices and key concepts. Before posting any questions, please review the following frequently asked questions (FAQs). If you don't find your question here or the answer is not to your satisfaction, please feel free to post a new question. Content of the Automated Testing FAQs, to date: FAQ1: I am new to automated testing, where do I start? FAQ2: Where can I find articles about Automated Testing? FAQ3: Where can I find books about Automated Testing? FAQ4: I have a question related to a specific tool, e.g. Winrunner, Robot, QARun, etc FAQ5: Which is the best testing tool on the market? FAQ6: Where can I find testing tool comparisons and evaluations? FAQ7: What is a Test Harness? (also referred to as Test Oracles) FAQ8: How can I manage false expectations that come with Automation? FAQ9: What is the Return on Investment (ROI) for Automated Testing? FAQ10: What is a Keyword or data driven framework? FAQ11: Why automate? FAQ12: When to automate? FAQ13: What to automate? FAQ14: How to get tool evaluation copies? FAQ15: Testing tool suggestions for various development languages, i.e. Java, C++, DHTML? FAQ16: Testing tool suggestions for SAP, PeopleSoft, Siebel, etc? FAQ17: Testing tools for different Operating Systems and other environments FAQ18: Where can I get tool training on-line? FAQ19: Where can I get free testing tools? FAQ20: Is 100 percent automation possible? FAQ21: Links to popular scripting languages for testing FAQ22:Open Source Test Automation ================================================================== FAQ1: I am new to automated testing, where do I start? If you are new to test automation you might want to start by reading the automation articles or books mentioned in FAQ2 and FAQ3. For starters, you might want to read the following posts and query for similar ones: Automation will it fail me? I'm new and I need your opinions about tools I am new, please help Steps for automated testing Automators vs Manual Tester Ratios Programming languages and testing FAQ2: Where can I find articles about Automated Testing? There are numerous test automation articles available. You might want to use a search engine such as Google or start by reviewing the following posts: Automated testing vs Human testing Implementation of Automated Tools Test Automation articles FAQ3: Where can I find books about Automated Testing? QALinks Books The Best Books Automated Testing reference books Books about Automated Testing tools FAQ4: I have a question related to a specific tool, e.g. Winrunner, Robot, QARun, etc Specific questions related to specific tools are to be posted under the relevant tool forum. Scroll down on the main qaforums page and you will find forums for most all the major tools on the market. Please post specific tool questions in the relevant tool forum, as this is a general automated testing discussion forum. FAQ5: Which is the best testing tool on the market? This question is asked often and the responses will vary, depending on the user's experience and other criteria. See for example this post: Automated Testing Tools It is important to evaluate the tool based on your system-engineering environment, available budget, schedules, skills available, and other testing needs. Some tools work better in specific environments, while in another environment a tool can cause compatibility problems if for example one of the application's major grids/controls is not compatible with the specific tool. Therefore, if a vendor wants to demonstrate their tool, please ask them to demonstrate the tool using your application under test. Other criteria is listed in the Test Tool Comparison/Evaluation in FAQ6, next. Sometimes you might have to develop a tool in-house, because none of the vendor provided tools are compatible with your system-engineering environment or for other reasons a test harness will have to be developed. See the related discussion below on Test Harnesses. Other times you might want to take advantage of the many available open source testing tools, for example take a look at all such tools listed here A related poll is here Best Automated Test Tool - 2005 To be most effective, sometimes it will require a mix of vendor provided, homegrown, and possibly open-source testing tools. FAQ6: Where can I find testing tool comparisons and evaluations? see Comparisons for the simple minded How long to evaluate GUI-automation test tools Document comparing automated test tools Mercury or Rational? Tool comparison Which of the automated test tool has the best service Automation Tool Criteria Test Tool evaluation Tool Evaluation FAQ7: What is a Test Harness? (also referred to as Test Oracles) Automated testing does not only involve using vendor provided testing tools, i.e. capture/playback tools, etc. or a modification of the tool's script generation/language - whether data driven or keyword driven - instead, it often requires the development of a test harness. For additional info, check this post: Test Harnesses Also, see Test Harness Can Perl script be used for testing automation? Integration Test with Test Drivers Components of an ideal test harness A non-GUI linux test harness Test oracles FAQ8: How can I manage false expectations that come with Automation? Along with the idea of automated testing come high expectations. Although automated testing can produce a return on investment if implemented effectively, an immediate pay-back is not always achieved. It is important to manage the high expectations that come with automated testing. Remember automated testing is just one tool of the many available in a tester's tool box. See the following post for related info: Misconceptions of Mgmt FAQ9: What is the Return on Investment (ROI) for Automated Testing? Automated testing can provide several benefits when it is implemented correctly. The test engineer must evaluate whether the potential benefits fit the required improvement criteria and whether the pursuit of automated testing on the project is still a logical fit, given the organizational environment, i.e. tool budgets, schedules, skills available, etc. At times putting a dollar value to automated testing ROI can be difficult, since there are many variables to consider, i.e. upfront investment, training, developing the automated test scripts, how often are the automated testing scripts used, etc. Also review the following posts: Justify the use or purchase of the testing tools Automated Testing Vs Manual - Time Saving? FAQ10: What is a Keyword or data driven framework? see "Test-Plan" driven Method (Keith Zambelich) Test engine Key-Word Driven Methodology Expected results for Data Driven tests How Many Of You Are Using Table/Keyword-Driven Methodology? Table/Keyword Driven (3rd Generation) Automation Questions FAQ11: Why automate? see Why automate? Is automated testing the answer? FAQ12: When to automate? see When should I start using an Automated tool for Regression testing Automated or Not automated that is the question Manual Testing Vs. Automated Testing FAQ13: What to automate? see How much automated testing? Criteria for Automated vs. Manual Testing FAQ14: How to get tool evaluation copies? see Evaluation copy........... More info to follow FAQ15: Testing tool suggestions for various development languages, i.e. Java, C++, DHTML? see What is the best automation tool for Lotus Notes/Domino, VB and Java Best ActiveX Control recognition in Automated Tool!! Testing VB.Net Applications? Unit testing tools for VB.Net Best tool for Java Automated Testing - FoxPro Automation Tools to run against Apps with Stingray Controls FAQ16: Testing tool suggestions for SAP, Siebel, etc? Mercury is the leader in this market..but Rational/IBM tools now also are SAP compatible, etc. placeholder FAQ17: Testing tools for different Operating Systems and other environments There are specific forums for Macintosh and Unix - scroll down on the main page. There is also a separate forum for PDAs and embedded systems - scroll down on the main page. I've found that few tools are available for the Linux environment and of the few tools that are available, i.e. for example Rational Functional Tester, the capture portion of it's capture/playback features won't be usable. In my experience when it comes to Linux functional test automation (note: not performance testing) it is best to use open source, or scripting languages such as Perl or Python, etc., not any of the current crop of vendor provided tools (unless perhaps your application runs Java on Linux) Also review the following Test automation tools for PDAs Best tool to test mainframe Test tool best for AS400 Linux Automation Tool FAQ18: Where can I get tool training on-line? see Qatraining.net provides on-line training Also see this related discussion Tutorials on WinRunner FAQ19: Where can I get free testing tools? see Free GUI test tools Open Source Test Suites For evaluations of free testing tools see Danny Faught's Open Testware Reviews FAQ20: Is 100 percent automation possible? Actually since 100 percent of testing is generally not possible given the time and budget constraints, so is 100 percent automation not possible. Mostly it's not feasible and most likely not cost-effective see Is 100 percent automation possible? FAQ21: Links to popular scripting languages for testing Ruby Python FAQ22: Open Source Test Automation Many open source test automation tools are available, see Opensourcetesting - I have spent a lot of time evaluating the STAF framework - see The Software Test Automation Framework and from what I have seen, i.e. the capabilities it provides, etc. it looks very impressive. I will keep automation forum members posted on actual implementation and additional findings. Post Resume: Click here to Upload your Resume & Apply for Jobs

	IP Logged


One Stop Testing Forum : Types Of Software Testing @ OneStopTesting : Automated Testing @ OneStopTesting