|
The basic purpose of this document is to give a high level introduction to Software Load and Performance testing methodologies
and strategies. This document is intended to facilitate Software test
Managers, Project Managers, Software Engineers, Test Leads, Test
engineers, and QA leads — anyone who is responsible for planning and/or
implementation of a successful and cost effective performance testing
program.
Scope
The scope of all the conceptualization mentioned in this document is only in the Test Execution in Automation Context.
In this context the attributes of Load & Performance testing covered are as follows,
Load / Performance Test Planning
Load / Performance Tool Evaluation / Selection
Load Test Process / Methodology and Test Strategy
Load / Performance Test Start/Stop Criteria
Test Environmental Setup and Pre-Requisites
Test Scenarios Definition including Load Scenario, Data Volume, Virtual Users Ramp Rates and Scripting Guidelines etc.
Pass / Fail / Exit Criteria
Analysis and Report Generation
{mosgoogle left}
2 Load / Performance Test Planning
To make any operation/mission successive planning plays the most
vital role and according to the 80-20 theory, 80% of the time should be
spent in planning and 20% in real time plan execution/implementation.
In the similar fashion, Software performance test planning is very
crucial as well.
Any Software Performance Test Plan should have the minimal contents such as,
Performance Test Strategy and Scope Definitions.
-Test Process and Methodologies to follow in different test phases.
-Test Tool Details (Tool Evaluation, Selection, Configuration, Addins, Third Party Utilities Integration, Os Configurations etc)
-Test Cases Details including Scripting, Library Development and
Script Maintenance Mechanisms to be used in Every Performance Test
Phases.
-Resource Allocations and Responsibilities for Test Participants.
-Test Life Cycle Tasks Management and Communication Media.
-Risk Management Definitions.
-Test Start /Stop Criteria along with Pass/Fail Criteria Definitions.
-Test Environment Setup Requirements. (Hardware Setup, Software Setup, Os Configurations, Third Party Utilities Setup etc.)
-Multi-Platform Performance Test Strategies.
-Application Deployment Setup Requirements for Load Test.
-Virtual Users, Load (Iterations Vs Users), Volume Load Definitions for Different Load/Performance Test Phases.
-Results Analysis Algorithms and Reporting Format Definitions.
3 Load / Performance Tool Evaluation & Selection
Tool Evaluation is yet another important task in Performance Test
Automation wherein there are several things to be considered. In cases
wherein the tool evaluation and selection is completely done by the
Client then readers may skip this Topic and proceed to next.
While selecting any tool for Load or/and Performance testing the following things should be analyzed such as,
-Test Budget Vs Available Load/Performance tools in the market mapping to the Budget.
-Protocols, Development & Deployment Technologies, Platforms,
Middle-Ware tools / Servers, Third-Party Integrations of the
Application Under test Vs Support for these factors in the available
tools with prioritization of the availability in the tool for the Scope
of expected test.
-Complexity of the Tool Usage Vs Availability of the tool experts
along with the timeline requirements for the tool scripting / load
scenario creation / tool configuration with respect to Man-hours and
Other Resource Requirements.
-Tools Limitations and Work-Around factor mapping with the current scope of testing.
-ToolÙs Integration / Portability Factors with Other Tools used to Monitoring, Analyzing and Test Management.
-On Evaluation and Selection of Base tool, third party monitors /
tools to be used in Integration with Main Load Testing Tool should be
defined.
(Third Party Monitors / Tools like ‘Optimize ITÙ, ‘WeblogicÙ,
‘Oracle ToolsÙ, ‘Spotlight On OracleÙ, ‘Fog-LightÙ and Test Management
Tools like Mercury-Test Director, Segue-Silk Plan Pro, Compuare-QA
Center etc.)
4 Test Process / Methodology and Test Strategy
Test Process, Methodology and Strategy for any Load / Performance
testing will definitely vary for every project based on the Client
Requirements / Company Process Implementations but still here we are
going to put some light on the common generic performance test process,
methodologies and strategies followed in the industry.
By using a methodology a project can make sure that resources are
applied to the problem effectively and that the people involved
approach the work in a structured way. In Performance testing Process
there are couple of classifications as mentioned below,
Benchmark Testing
This task tests the newly coded performance features to assert that
they do actually improve the performance of the application. There is
always a danger that a new performance feature decreases the
performance of the application instead of increasing it. Tests should
be constructed so that they executed a standard benchmark style test
with the feature switched on and with the feature switched off.
The first time this particular task is performed the task
becomes developing the benchmark tests. When this task is performed in
later iterations it becomes re-running the benchmark tests with the new
version of the code in order to track whether or not the code is
improving.
Analysis Testing
Benchmark tests are good for a basic understanding of what is
happening and are very useful for tracking improvements over time but
they are not so good at isolating the reasons of the next major
performance problem. Analysis Testing refers to designing tests that
attempt to isolate the next major performance problem. These kinds of
tests may do something completely different to the benchmark tests in
order to explore what is happening in the target application. The tests
are designed to explore theories as to where the next performance
problem may be.
This is also where supplementary tools are used most often.
Additional tools such as method-level profilers and operating system
performance monitors can be useful in working out where a problem might
be occurring.
Long Running Tests
It is important to include some tests that run for long periods of
time. There are a number of problems that may occur only after the
target application has been running for more than a day or even after a
week. Memory leaks in particular may take many hours before they become
measurable. The length of these tests should be related to how often
you plan to run the target application before restarting. If you plan
to restart the target application every day then the test length should
simulate one dayÙs worth of work but if you plan to restart the target
application only once a week then the test length should simulate a
weekÙs worth of work.
This type of testing involves long periods of time and this causes problems such as,
1. Since the tests take so long you are not typically able to perform as many of them.
2. These tests take so long that they tie up resources for long periods of time.
One thing that you can do to is compact the work
performed over one day into a short period time by making use of how
normally a server machine is not run at one hundred percent of CPU for
the whole day. By creating a sustained workload for a shorter period of
time you can usually simulate on days worth of work in a shorter
period. For problems like memory leaks it doesnÙt normally matter how
long the test takes what is important is how much work the test does. A
quick half-hour test may not exercise the application for long enough
but a sustained high-workload test that simulates one days worth of
work in six hours should hopefully identify any problems.
For example a typical interactive application that services
internal users might look like the following chart with peaks at around
11am in the morning and 2pm in the afternoon.
By making the CPU do more work in less time you can compact the
overall length of the test into a shorter period and so perform long
running tests more easily.
Due to the hassle and restrictions involved in this kind of testing
it should be done after the application is behaving reasonably well
with shorter testing periods. But also it shouldnÙt be left too late
either since it takes so long to run these tests it is a good idea to
get started with them before the end of the time allocated to
performance testing.
Distributed Testing
Another issue that often comes up is where the clients are located
to do their testing. If the clients are located near where the server
is running then the test may not adequately simulate the eventual
production configuration. If the eventual production configuration is
to have the client processes distributed nation-wide over a companyÙs
private network then some testing should be done to simulate this
otherwise if there is a problem with the network then testing with the
client processes near to the server may not uncover this.
There are generally practical problems with doing this kind
of testing. There may be many different sites from where the clients
may be executed and managing the load testing clients out on all of
these sites can be a hassle. Due to these sorts of problems not all of
the performance testing has to be done in a distributed way but some of
it should be. Also like the long running tests this sort of testing
should be done after the application is behaving reasonably well in a
non-distributed way. This way the methodologies can be implemented
based on the scope of testing requirements at a particular phase of
performance testing.
If we talk about the Test Process in general practice, once
the Benchmark attributes like Load Scenario with load (Iterations /
Scenario / Script), Virtual users with ramp-up rates, Business
Transactions Definitions, Finalization of Tools for testing, Hardware
Environment, SW-Environment like OS configuration, DB, Data
Dependencies, Tool configurations, Application under test deployment
details are finalized and documented the initial base benchmark test
can be conducted in a normal circumstances to make sure that the
application is in a stable state to go through the heavy load test. If
any issues found during this test, then release cycles may be repeated
till application reaches enough stability to start heavy Load /
Performance test.
The next task at the end of Baseline Performance test can be
to start the Load Test. Load testing is an important task accomplished
through Stress and Data Volume testing. This methodology of testing is
used to establish an initial sound structure upon which the product is
developed. A load test is meant to exercise the design of the
application and reveal weaknesses in the targeted product before
“General Availability”. By revealing problems during the development we
do not only prepare for a successful deployment, but also a long
prosperous product life cycle.
Load testing comprises of Individual Module testing run on
Iterative basis. Load tests are conducted against the Application Under
Test using a varying number of virtual users whose results are used for
developers/architects to validate/tune the code / design
/configurations for optimising the product performance. The metrics
established from the load test will be compared against the baseline to
identify performance discrepancies.
The cycles of such load tests and tuning can be repeated till
the product stability reaches enough to go to production OR stability
as defined by the client.
Along with load testing, Stress testing is implemented as a
sub-set of load testing. The stress is nothing but applying the load in
the ramp-up of virtual users in the groups at the pre-defined rates.
(I.e. Group1 – Users will ramp at rate of 5 users/scenario –
5,10,15,20). There are cases when Client expects to benchmark the
bottlenecks for specific application functionalities commonly
used/accessed by all users the most and that too at a time. For such
cases, we can put rendezvous points at appropriate transaction actions
wherein all the users will ramp up at the pre-defined rate but when it
comes the time to execute the action of rendezvous action, users will
wait till all Virtual Users reach till that point and perform that
action at a time. So in such a scenario bottlenecks can be easily
located. Basically Stress test are intended to be modular in design so
each functional script can be run independently.
In the performance testing process, there are cases in which
some part of application is data volume dependant like Search Screens
wherein the Queries / Stored Procedures Performance based on amount of
data volume existing in the database, then for such cases before
starting the test, the dump of some huge amount of data should be
inserted in the database through some SQL Scripts or Automation Scripts
etc. so that the results of such screens can help in database tuning /
optimisation.
At the end of all such tests, the results from the load
testing tool, monitoring tools give a bright idea about the tuning
requirements in the system like CPU, Memory, OS, HW, SW, Code,
Application Configuration, Middle-tier servers etc. and such Load Test
and Results Analysis Vs tuning cycles can be repeated till the
application reaches stability as expected by the client.
5 Load / Performance Test Start/Stop Criteria
In case of any performance testing there should be a defined /
documented Start and End Criteria to avoid the adhoc process resulting
in weird outputs. This start / end criteria should be well defined in
the Performance Test Plan along with all the risk management factors.
The start of any Performance or load testing should be done for any
application or any module of the application only when the Design,
Database Structure and Application Development platform is finalized
and there will be no major changes at any layers of the application,
which affect Directly or Indirectly to the Performance Factor. It
really again depends on the scope of testing such that the test is
going to be conducted for a specific Module or the whole System.
Once the Load or Performance Test is conducted, results are
collected and analyzed the cycles of tuning will carry on but yes
definitely the end criteria should be well defined to Stop this
Performance Testing and Tuning Cycles based on various factors like
product GA release timelines (Project Deadlines) and Client
Requirements in terms of System Stability, Scalability and reliability
under load and Performance Tolerance Criterions.
In some projects the end criteria is defined based on the Client
Performance Measures Requirements defined for each section of the
application so if product reaches to that expected level then that
might be considered as the end criteria for Performance Testing.
6 Test Environmental Setup and Pre-Requisites
In the performance testing, by the term test environment here, itÙs
composed of hardware environment (Machine, Memory, Virtual Memory,
Number of CPUÙs etc), Software environment like Os, Os configuration,
Files System Settings, Free Space for logs, Application Configuration
along with Middle Tier Server and associated Configurations, Database,
Network etc.
Among all of the above environmental variables defined few or all
might be used for setup based on the Deployment Setup Client / Customer
expects to have in production.
This setup also involves the Automation Tools installation,
configurations, Agent Installations and setup in Load Clients, Third
Party Monitors on Clients / Servers etc.
There may or may not be any pre-requisites if all of the
above variables setup is all set before starting the test but some
pre-requisites like Reference Data, Test Input Data in the database
Insertion etc.
Also System Settings at Os Level, Tier-Level, Server Level etc.
should be listed in a document as Checklists and should be verified
before starting any Performance Test.
7 Test Scenarios
Test Scenarios Definition includes various tasks like Business
Scenarios Definitions, Writing Automation Scripts for Test Cases
Corresponding to defined Business / Functional Scenarios, expected Data
Volume for Test mapping to Virtual Users Ramp Rates and Scripting
Guidelines etc.
In the test scenario development, the scripts written can be
made data driven to facilitate Iterative testing in Load Test
Scenarios. Once these details are well defined in Test Automation Plan,
then Virtual User Test Scripts can be written to simulate User Client
Actions / Client - Server Request-Response Hits etc. On completion of
baseline scripting, based on data-drive requirements the test scripts
can be parameterized to make them compatible to execute the same
functionality / User Actions for different set of Test Data on
Iterative basis.
In some typical cases in few scenarios wherein the performance of
some sections of the application is based on the volume of the data
residing in the database then Data Volume Based Performance Test is a
must. In such cases, the data insertion SQL Scripts can be written and
executed before the execution of Load Test and Data-Volume based Test
Scripts should be mapped to this data.
Also along with these factors definitely the number of Virtual
Users, amount of load (i.e. Preset Number of Iterations/script) should
be predefined from the Customer Production load Volume in peak and
non-peak hours. Also in the Stress test which is a part of load test
itself, the Virtual Users need to be set in Ramp Up by the pre-defined
rates like 5,10,15,20,25 Users etc. Along with the Stress Ramping Rate
setting of the Virtual Users in the load scenario, there are cases
wherein if the requirement is to perform a particular Action/Operation
by all the virtual users at a time for finding out the bottle neck
points, Test Scripts should have rendezvous points for such actions so
that whenever all the virtual users are performing different operations
for different test cases, they will wait till all Virtual Users get to
the rendezvous points and execute that action at a time and at the end
of that action then again the Ramp-up of Virtual Users is re-started.
8 Analysis and Report Generation
Once the results are ready, analysis is the most important job in
the whole performance test if the requirements of load or performance
test are not defined.
In the analysis, the main criteria for analysis should be to find
out the bottle neck points wherein we should be able to track out the
transactions / users / hits creating the highest level of performance
reduction. The main factors to be consideration are,
Request / Response Timings to-fro from Client to Server
-Transaction Timings Vs Users
-Transaction Timings Vs Load
-Transaction Timings Vs CPU Utilization
-Server Memory Utilization
-Database Server Performance using DB Server Monitors
-Web / Application Server Performance
-Paging Rate Ratios
-Open Cursor Factors in Database
-Load Balancer Factors
-Network Delays
-Bean Performance Factors using Monitors like Optimize IT for Java Based Applications
-Other Monitors like Application Server (e.g. Web-logic) Monitors etc.
Thus now after consolidating all the results for the above points,
the statistical analysis should be done and the Performance Test Report
should be created such that the Development, Design-Architecture,
Database / IT, PMG group can get a clear idea about the Application
Performance and all the factors affecting it.
After Analysis then based on the scope the corresponding tuning
like Os tuning, Database Tuning, Memory Up-gradations, CPU counts in
Multi-CPU machine, Code-Tuning, Application Server Cache tuning or any
other tuning as per the scope can be done and same tests executed
earlier should be repeated in the tuned environment to make out the
differences in the results before and after tuning. These tuning cycles
can be repeated until the pre-defined required Performance is achieved.
In the next version of this document we will be placing the examples
for each type of Performance Analysis with the corresponding graph with
all the algorithms used for analysis.
9 Pass / Fail / Exit Criteria
At the end of the Load / Performance Test execution, we can find
the number of failed users, failed transactions or failed scripts but
the Pass / Fail Criteria for a particular test should be well defined
before execution of the test as basically this criteria entirely
depends on the scope of testing and the environment. For e.g. in some
cases few transaction steps failed which has some workarounds or the
reasons for failing the transactions or virtual users are due to
unusual attributes behavior which is not related to the application
then the results, or few times the tests / scripts / virtual users /
transactions failure might happen due to Load Tool Configuration or
Load Tool Defects also so its very important to analyze and make a
precise differentiation of the Defect occurred in the test and then
finalize whether the test was pass or fail.
Also based on the failed Items due to any of the performance test
parameters the Failed Item Details should be included in the
Applications Limitations List. By doing so, the limitations list can
give a clear picture about the Applications Scalability / Load or
Performance Limitations with respect to corresponding Load Parameters
which in turn will give inputs for the tuning / enhancements for the
Application.
Based on the Test Results tuning cycles will be driven and test
phases will as well be repeated but the exit criteria should be
pre-defined to avoid hazardous tuning life cycle.
-------------
http://www.quick2sms.com - Send Unlimited FREE SMS to Any Mobile Anywhere in INDIA,
Click Here
|