Topic: Evaluation Criteria for Black Box Testing........

Message

                                    Evaluation Criteria

        The following is a list of evaluation criteria that may be considered when selecting a black box security testing tool. Many of the criteria listed here are from Appendix B of [Dustin 01]. Readers are encouraged to consult this original document as well, since it gives an expanded list of evaluation criteria and also provides evaluation results for several major test tool suites (albeit not security-specific test tools). Not all of the criteria listed below may be relevant to all test organizations or all test projects. In addition to the criteria listed here, organizations may also want to consider support for the specific black box security testing technologies described previously in this document.

1.Ease of Use

2.Intuitive and easy to use for users new to automated testing tools

3.Easy to install; tool may not be used if difficult to install

4.Tasks can be accomplished quickly, assuming basic user proficiency

5.Easy to maintain automated tests,
with a central repository that enables users to separate GUI object
definitions from the script

6. Can vary how designs and documents are viewed (zooming, multipage diagrams easily supported, multiple concurrent views); basic windowing

1.Tool Customization

2.Fully customizable toolbars to reflect any commonly used tool capabilities

3.Tool customizable: fields added, deleted

4. Fully customized editor with formats and colors for better readability

5.Tool support for required test procedure naming convention

1.Can be used with non-Microsoft platforms (UNIX, Linux, FreeBSD, Mac)

2.Tests for common website vulnerabilities

3. Evaluates the test environment as well as the software

4.Supports standard web protocols for fuzzing and domain testing.

1. Test Coverage and Completeness

            Coverage refers to the ability of the tools to test for all (known) categories of vulnerabilities relevant to the product that has been developed. It is important to obtain a sense of the percentage and nature of potential vulnerabilities the tools tests for. For example, if evaluating a web-based system, the organization will want to determine whether the test tool identifies issues that may result from improper input validation, SQL insertion attacks, cross-site scripting attacks, or improper session management.

5. Accuracy/False-Positive Rate

1. Is there a large number of false positives? False positives will result in more analysis work for the tester, who will be required to manually evaluate the results of the test tool.

2. Is there a large number of unidentified vulnerabilities?

6. Test Language Features

1.  Allows add-ins and extensions compatible with third-party controls

2.Does not involve additional cost for add-ins and extensions

3. Has a test editor/debugger feature

4. Test scripting language flexible yet robust; allows for modular script development

5. Scripting language not too complex

6. Scripting language allows for variable declaration and use and for parameter to be passed between functions

7.  A test script compiler or an interpreter used?

8.Allows for interfacing and testing of external .dll and .exe files

9.Published APIs: Language Interface Capabilities

10.  Tool is not intrusive: source code of application does not need to be expanded by inserting additional statements or dlls for the application to be compatible with the tool

11.Allows for data-driven testing

12.Allows for automatic data generation

13.  Allows for adding timers for timing transaction start and end

14.Allows for adding comments during recording

15.Allows for automatic or specified synchronization between client and server

16.Allows for object data extraction and verification

17.Allows for database verification

18.Allows for text (alphanumeric) verification

19.Allows for wrappers (shells) whereby multiple procedures can be linked and called from one procedure

20.Allows for automatic data retrieval from any data source—RDBMS, legacy system, spreadsheet—for data-driven testing

21. Allows for use of common spreadsheet for data-driven testing

22.Ease of maintaining scripts when application changes

7.Test Management
         1.

            Supports test execution management
         2.

            Support for industry standards in testing processes (e.g., SEI/CMM, ATLM, ISO)
         3.

            Interoperability with tools being used to automate traditional testing
         4.

            Application requirements management support integrated with the test management tool
         5.

            Requirements management capability supports the trace of requirements to test plans to provide requirement coverage metrics
         6.

            Test plans can be imported automatically into test management repository from standard text files
         7.

            Can be customized to organization’s test process
         8.

            Supports planning, managing, and analyzing testing efforts; can reference test plans, matrices, product specifications, in order to create traceability
         9.

            Supports manual testing
        10.

            Supports the migration from manual to automated scripts
        11.

            Can track the traceability of tests to test requirements
        12.

            Has built-in test requirements modules
        13.

            Can check for duplicate defects before logging newly found defects
        14.

            Allows for measuring test progress
        15.

            Allows for various reporting activities
        16.

            Allows for tracking of manual and automated test cases
        17.

            Has interface to software architecture/modeling tool
        18.

            Is integrated with unit testing tools
        19.

            Has interface to test management tool
        20.

            Has interface to requirements management tool
        21.

            Has interface to defect tracking tool
        22.

            Has interface to configuration management tool
        23.

            Provides summary-level reporting
        24.

            Includes error filtering and review features
        25.

            Enables metric collection and metric analysis visualization
   8.

      Interoperability
         1.

            Major test automation suites provide functionality that is useful in any large-scale testing process. For smaller, more specialized tools, interoperability with other test tool suites may be considered as an evaluation criterion.
   9.

      Load and Stress Test Features
         1.

            All users can be queued to execute a specified action at the same time
         2.

            Automatic generation of summary load testing analysis reports
         3.

            Ability to change recording of different protocols in the middle of load-recording session
         4.

            Actions in a script can be iterated any specified number of times without programming or rerecording of the script
         5.

            Different modem connection speeds and browser types can be applied to a script without any rerecording
         6.

            Load runs and groups of users within load runs can be scheduled to execute at different times
         7.

            Automatic load scenario generation based on load testing goals: hits/second, number of concurrent users before specified performance degradation, and so on
         8.

            Cookies and session IDs automatically correlated during recording and playback for dynamically changing web environments
         9.

            Allows for variable access methods and ability to mix access methods in a single scenario: modem simulation or various line speed simulation
        10.

            Ability to have data-driven scripts that can use a stored pool of data
        11.

            Allows for throttle control for dynamic load generation
        12.

            Allows for automatic service-level violation (boundary value) checks
        13.

            Allows for variable recording levels (network, web, API, and so on)
        14.

            Allows for transaction breakdown/drill-down capabilities for integrity verification at the per client, per session, and per instance level for virtual users
        15.

            Allows for web application server integration
        16.

            Supports workload, resource, and/or performance modeling
        17.

            Can run tests on various hardware and software configurations
        18.

            Support headless virtual user testing feature
        19.

            Requires low overhead for virtual user feature (web, database, other?)
        20.

            Scales to how many virtual users?
        21.

            Simulated IP addresses for virtual users
        22.

            Thread-based virtual user simulation
        23.

            Process-based virtual user simulation
        24.

            Centralized load test controller
        25.

            Allows for reusing scripts from functional test suite
        26.

            Support for WAP protocol testing against WAP Gateway or web server
        27.

            Compatible with SSL recording
        28.

            Compatible with which network interaction technologies? (e.g., streaming media, COM, EJB, RMI, CORBA, Siebel, Oracle, SAP)
        29.

            Compatible with which platforms? (e.g., Linux, UNIX, NT, XWindows, Windows CE, Win3.1, Win95, Win98, Win2000, WinME)
  10.

      Monitor Test Features
         1.

            Monitors various tiers: web server, database server, and app server separately
         2.

            Supports monitoring for which server frameworks? (e.g., ColdFusion, Broadvision, BEA WebLogic, Silverstream, ATG Dynamo, Apache, IBM Websphere, Oracle RDBMS, MS SQL Server, Real Media Server, IIS, Netscape Web Server
         3.

            Supports monitoring of which platforms? (e.g., Linux, NT, UNIX, XWindows, Windows CE, Win3.1, Win95/98, Win2000)
         4.

            Monitors network segments
         5.

            Supports resource monitoring
         6.

            Synchronization ability in order to determine locking, deadlock conditions, and concurrency control problems
         7.

            Ability to detect when events have completed in a reliable fashion
         8.

            Ability to provide client-to-server response times
         9.

            Ability to provide graphical results and export them to common formats
  11.

      Consulting Requirements
         1.

            Maturity of vendor
         2.

            Market share of vendor
  12.

      Vendor Qualifications
         1.

            Financial stability of vendor
         2.

            Length of time in business
         3.

            Technological maturity
  13.

      Vendor Support
         1.

            Software patches provided, if deemed necessary
         2.

            Upgrades provided on a regular basis
         3.

            Upgrades backward compatible: scripts from previous version can be reused with later version
         4.

            Training available
         5.

            Help feature available; tool well documented
         6.

            Tech support reputation throughout industry
         7.

            No consulting needed?
         8.

            Availability of and access to tool user groups
  14.

      Product Pricing
         1.

            Price consistent within estimated price range
         2.

            Price consistent with comparable vendor products
         3.

            ROI compared to current in-house technology
         4.

            ROI compared to in-house development of needed technology

Author	Message
Harini Newbie Joined: 15Feb2007 Online Status: Offline Posts: 1	Topic: Evaluation Criteria for Black Box Testing........ Posted: 15Feb2007 at 6:15pm
	Evaluation Criteria The following is a list of evaluation criteria that may be considered when selecting a black box security testing tool. Many of the criteria listed here are from Appendix B of [Dustin 01]. Readers are encouraged to consult this original document as well, since it gives an expanded list of evaluation criteria and also provides evaluation results for several major test tool suites (albeit not security-specific test tools). Not all of the criteria listed below may be relevant to all test organizations or all test projects. In addition to the criteria listed here, organizations may also want to consider support for the specific black box security testing technologies described previously in this document. 1.Ease of Use 2.Intuitive and easy to use for users new to automated testing tools 3.Easy to install; tool may not be used if difficult to install 4.Tasks can be accomplished quickly, assuming basic user proficiency 5.Easy to maintain automated tests, with a central repository that enables users to separate GUI object definitions from the script 6. Can vary how designs and documents are viewed (zooming, multipage diagrams easily supported, multiple concurrent views); basic windowing 1.Tool Customization 2.Fully customizable toolbars to reflect any commonly used tool capabilities 3.Tool customizable: fields added, deleted 4. Fully customized editor with formats and colors for better readability 5.Tool support for required test procedure naming convention 1.Can be used with non-Microsoft platforms (UNIX, Linux, FreeBSD, Mac) 2.Tests for common website vulnerabilities 3. Evaluates the test environment as well as the software 4.Supports standard web protocols for fuzzing and domain testing. 1. Test Coverage and Completeness Coverage refers to the ability of the tools to test for all (known) categories of vulnerabilities relevant to the product that has been developed. It is important to obtain a sense of the percentage and nature of potential vulnerabilities the tools tests for. For example, if evaluating a web-based system, the organization will want to determine whether the test tool identifies issues that may result from improper input validation, SQL insertion attacks, cross-site scripting attacks, or improper session management. 5. Accuracy/False-Positive Rate 1. Is there a large number of false positives? False positives will result in more analysis work for the tester, who will be required to manually evaluate the results of the test tool. 2. Is there a large number of unidentified vulnerabilities? 6. Test Language Features 1. Allows add-ins and extensions compatible with third-party controls 2.Does not involve additional cost for add-ins and extensions 3. Has a test editor/debugger feature 4. Test scripting language flexible yet robust; allows for modular script development 5. Scripting language not too complex 6. Scripting language allows for variable declaration and use and for parameter to be passed between functions 7. A test script compiler or an interpreter used? 8.Allows for interfacing and testing of external .dll and .exe files 9.Published APIs: Language Interface Capabilities 10. Tool is not intrusive: source code of application does not need to be expanded by inserting additional statements or dlls for the application to be compatible with the tool 11.Allows for data-driven testing 12.Allows for automatic data generation 13. Allows for adding timers for timing transaction start and end 14.Allows for adding comments during recording 15.Allows for automatic or specified synchronization between client and server 16.Allows for object data extraction and verification 17.Allows for database verification 18.Allows for text (alphanumeric) verification 19.Allows for wrappers (shells) whereby multiple procedures can be linked and called from one procedure 20.Allows for automatic data retrieval from any data source—RDBMS, legacy system, spreadsheet—for data-driven testing 21. Allows for use of common spreadsheet for data-driven testing 22.Ease of maintaining scripts when application changes 7.Test Management 1. Supports test execution management 2. Support for industry standards in testing processes (e.g., SEI/CMM, ATLM, ISO) 3. Interoperability with tools being used to automate traditional testing 4. Application requirements management support integrated with the test management tool 5. Requirements management capability supports the trace of requirements to test plans to provide requirement coverage metrics 6. Test plans can be imported automatically into test management repository from standard text files 7. Can be customized to organization’s test process 8. Supports planning, managing, and analyzing testing efforts; can reference test plans, matrices, product specifications, in order to create traceability 9. Supports manual testing 10. Supports the migration from manual to automated scripts 11. Can track the traceability of tests to test requirements 12. Has built-in test requirements modules 13. Can check for duplicate defects before logging newly found defects 14. Allows for measuring test progress 15. Allows for various reporting activities 16. Allows for tracking of manual and automated test cases 17. Has interface to software architecture/modeling tool 18. Is integrated with unit testing tools 19. Has interface to test management tool 20. Has interface to requirements management tool 21. Has interface to defect tracking tool 22. Has interface to configuration management tool 23. Provides summary-level reporting 24. Includes error filtering and review features 25. Enables metric collection and metric analysis visualization 8. Interoperability 1. Major test automation suites provide functionality that is useful in any large-scale testing process. For smaller, more specialized tools, interoperability with other test tool suites may be considered as an evaluation criterion. 9. Load and Stress Test Features 1. All users can be queued to execute a specified action at the same time 2. Automatic generation of summary load testing analysis reports 3. Ability to change recording of different protocols in the middle of load-recording session 4. Actions in a script can be iterated any specified number of times without programming or rerecording of the script 5. Different modem connection speeds and browser types can be applied to a script without any rerecording 6. Load runs and groups of users within load runs can be scheduled to execute at different times 7. Automatic load scenario generation based on load testing goals: hits/second, number of concurrent users before specified performance degradation, and so on 8. Cookies and session IDs automatically correlated during recording and playback for dynamically changing web environments 9. Allows for variable access methods and ability to mix access methods in a single scenario: modem simulation or various line speed simulation 10. Ability to have data-driven scripts that can use a stored pool of data 11. Allows for throttle control for dynamic load generation 12. Allows for automatic service-level violation (boundary value) checks 13. Allows for variable recording levels (network, web, API, and so on) 14. Allows for transaction breakdown/drill-down capabilities for integrity verification at the per client, per session, and per instance level for virtual users 15. Allows for web application server integration 16. Supports workload, resource, and/or performance modeling 17. Can run tests on various hardware and software configurations 18. Support headless virtual user testing feature 19. Requires low overhead for virtual user feature (web, database, other?) 20. Scales to how many virtual users? 21. Simulated IP addresses for virtual users 22. Thread-based virtual user simulation 23. Process-based virtual user simulation 24. Centralized load test controller 25. Allows for reusing scripts from functional test suite 26. Support for WAP protocol testing against WAP Gateway or web server 27. Compatible with SSL recording 28. Compatible with which network interaction technologies? (e.g., streaming media, COM, EJB, RMI, CORBA, Siebel, Oracle, SAP) 29. Compatible with which platforms? (e.g., Linux, UNIX, NT, XWindows, Windows CE, Win3.1, Win95, Win98, Win2000, WinME) 10. Monitor Test Features 1. Monitors various tiers: web server, database server, and app server separately 2. Supports monitoring for which server frameworks? (e.g., ColdFusion, Broadvision, BEA WebLogic, Silverstream, ATG Dynamo, Apache, IBM Websphere, Oracle RDBMS, MS SQL Server, Real Media Server, IIS, Netscape Web Server 3. Supports monitoring of which platforms? (e.g., Linux, NT, UNIX, XWindows, Windows CE, Win3.1, Win95/98, Win2000) 4. Monitors network segments 5. Supports resource monitoring 6. Synchronization ability in order to determine locking, deadlock conditions, and concurrency control problems 7. Ability to detect when events have completed in a reliable fashion 8. Ability to provide client-to-server response times 9. Ability to provide graphical results and export them to common formats 11. Consulting Requirements 1. Maturity of vendor 2. Market share of vendor 12. Vendor Qualifications 1. Financial stability of vendor 2. Length of time in business 3. Technological maturity 13. Vendor Support 1. Software patches provided, if deemed necessary 2. Upgrades provided on a regular basis 3. Upgrades backward compatible: scripts from previous version can be reused with later version 4. Training available 5. Help feature available; tool well documented 6. Tech support reputation throughout industry 7. No consulting needed? 8. Availability of and access to tool user groups 14. Product Pricing 1. Price consistent within estimated price range 2. Price consistent with comparable vendor products 3. ROI compared to current in-house technology 4. ROI compared to in-house development of needed technology Post Resume: Click here to Upload your Resume & Apply for Jobs

	IP Logged


One Stop Testing Forum : Types Of Software Testing @ OneStopTesting : Manual Testing @ OneStopTesting