Benefits and Limitations of Black Box Testing. As previously discussed, black box tests are generally conducted when the tester has limited knowledge of the system under test or when access to source code is not available. On its own, black box testing is not a suitable alternative for security activities throughout the software development life cycle. These activities include the development of security-based requirements, risk assessments, security-based architectures, white box security tests, and code reviews. However, when used to complement these activities or to test third-party applications or security-specific subsystems, black box test activities can provide a development staff crucial and significant insight regarding the system’s design and implementation.

Black box tests can help development and security personnel

• identify implementation errors that were not discovered during code reviews, unit tests, or security white box tests

• discover potential security issues resulting from boundary conditions that were difficult to identify and understand during the design and implementation phases

• uncover security issues resulting from incorrect product builds (e.g., old or missing modules/files)

• detect security issues that arise as a result of interaction with underlying environment (e.g., improper configuration files, unhardened OS and applications)

        Accordingly, black box security test efforts complement the critical security activities throughout the SDLC. The tools help developers and security personnel verify that the system security components are operating properly and also identify potential security vulnerabilities resulting from implementation errors. Additionally, black box security tests can help security practitioners test third-party components that may be considered for integration into the overall system and for which source code is not available. These tests may help the development staff uncover potential security vulnerabilities and make intelligent decisions about the use of certain products within their overall system.

        Although these tests should not be considered a substitute for techniques that help developers build security into the product during the design and implementation stages, without these tests, developers may overlook implementation issues not discovered in earlier phases. Despite the best efforts of the development staff, mistakes do occur—coding errors, incorrect components in the latest software build, unexpected interaction with the deployed environment, and boundary conditions, to name a few. Black box security tests provide a method to validate the security of the system before it is deployed.

Black box testing tools provide various types of automated support for testers. They help testers work more efficiently by automating whatever tasks can be automated, and they also help testers avoid making mistakes in a number of tasks where careful bookkeeping is needed. Their main roles include

• test automation: providing automated support for the actual process of executing tests, especially tests that have already been run in the past but are being repeated

• test scaffolding: providing the infrastructure needed in order to test efficiently

• test management: various measurements and scheduling and tracking activities that are needed for efficient testing even though they are not directly involved in the execution of test cases