Flexibility and Power is Often Sacrificed for Ease of Scripting.
Many of the available tools have as a selling point that they make
writing automated tests quick and easy. To do this they rely on
simplified scripting languages, which don't allow access to external
resources or publicly developed and supported libraries. The languages
also often limit the extent of abstraction and reuse, or at the least
do not encourage good coding practices for maintainability over the
long run. To make scripting easier, the tools lose the complexity of a
full language, so organizations can only feasibly test the easier
problems, missing most of the unique value of automation. Because of
the lack of flexibility, it is common to see organizations needing
several different tools for web testing, desktop GUI testing,
client/server testing, load testing, etc. Now testers have to learn
many simplified scripting languages, so in the long run it may not be
so simple to learn.
For both automated and manual testing, test cases which document the
steps required to pass the business rules, are the base of any test plan. Because
automated scripts are usually managed separate from the test cases, the automated
scripts and the test cases have a high probability of getting out of sync. Soon,
no one knows whether the test case or the test script represent the current
state of the application.
In many cases, the data being tested cannot be easily separated from the actual test
script. This requires creating several test scripts or requiring the data be included
in the code itselft in order to test the many small rules of a single business rule. This not
only requires extra work to test a that single feature, but it also makes it a nightmare to
maintain the test scripts that only differ in the data being used to test the application.
Many of the automated testing frameworks available don't take the lifecycle
of a product into consideration. A product must go through development, alpha
testing, beta testing, and performance testing before it reaches the masses.
Because of the infeasibility of editing large numbers of scripts, many organizations only
run their automated tests against one environment, thus missing opportunities for
validation and regression testing as the application advances in the product lifecycle.
Most testing tools require learning a propriety language or technology. This
makes it harder to find experienced people, which restricts a company's options in
selecting testing tools. The narrow applicability of proprietary technologies may also
deter current employees from acquiring a more in-depth knowledge of the tool.