SSDL Touchpoints: Security Testing (ST)
The overall goal of the Security Testing practice is quality control. Those performing security testing must ensure the detection and correction of security bugs. The SSG must enforce adherence to standards and the reuse of approved security features.
ST Level 1: Enhance QA beyond functional perspective. The SSG must share its security knowledge and testing results with QA. QA must progress to include functional edge and boundary condition testing in its test suites.
ST1.1
Ensure QA supports edge/boundary value condition testing. The QA team goes beyond functional testing to perform basic adversarial tests. They probe simple edge cases and boundary conditions. No attacker skills required. When QA understands the value of pushing past standard functional testing using acceptable input, they begin to move slowly toward “thinking like a bad guy.” A discussion of boundary value testing leads naturally to the notion of an attacker probing the edges on purpose. What happens when you enter the wrong password over and over?
ST1.2
Share security results with QA. The SSG shares results from security reviews with the QA department. Over time, Quality Assurance Engineers learn the security mindset. Tying results to particular testing patterns (though not necessary for simple exposure to security results), can be a powerful mechanism leading to better security testing.
ST1.3
Allow declarative security/security features to drive tests. Testers target declarative security mechanisms and security features in general. For example, a tester could try to access administrative functionality as an unprivileged user or verify that a user account becomes locked after some number of failed authentication attempts. For the most part, security features can be tested in a similar fashion to other software features as can declarative security mechanisms such as account lockout, transaction limitations, entitlements, and so on. Of course, software security is not security software, but getting started with features is easy.
ST Level 2: Integrate the attacker perspective into test plans. QA must integrate black-box security testing tools into its process. QA must build test suites for functional security features and progress to building adversarial tests that simulate the attacker’s perspective.
ST2.1
Integrate black box security tools into the QA process (including protocol fuzzing). The organization uses one or more black box security testing tools as part of the quality assurance process. The tools are valuable because they encapsulate an attacker’s perspective, albeit in a generic fashion. Tools such as Rational AppScan or HP WebInspect are relevant for Web applications and fuzzing frameworks such as PROTOS and Codenomicon are applicable for most network protocols. In some situations, the other groups might collaborate with the SSG to apply the tools. For example, a testing team could run the tool, but come to the SSG for help interpreting the results. In other cases, the SSG may run the tools at the proper stage of the SSDL.
ST2.3
Begin to build/apply adversarial security tests (abuse cases). Testing begins to incorporate test cases based on abuse cases provided by the SSG. Testers move beyond verifying functionality and take on the attacker’s perspective. For example, testers might systematically attempt to replicate incidents from the organization’s history. Abuse and misuse cases taking into account the attacker’s perspective can also be driven from security policies, attack intelligence, and guidelines. This turns the corner from testing features to attempting to break the software under test.
ST Level 3: Deliver risk-based security testing. QA must include security testing in automated regression suites. The SSG must ensure this security testing and its depth is guided by knowledge about the codebase and its associated risks.
ST3.1
Include security tests in QA automation. Security tests run alongside functional tests as part of automated regression testing; the same automation framework houses both. Security testing is part of the routine. Security tests can be driven from abuse cases identified earlier in the lifecycle or tests derived from creative tweaks of functional tests.
ST3.2
Perform fuzz testing customized to application APIs. Test automation engineers customize a fuzzing framework to the organization’s APIs. They could begin from scratch or use an existing fuzzing toolkit, but customization goes beyond creating custom protocol descriptions or file format templates. The fuzzing framework has a built-in understanding of the interfaces it calls into. Test harnesses developed explicitly for particular applications can make good places to integrate fuzz testing.
ST3.3
Drive tests with risk analysis results. Testers use architecture analysis results to direct their work. For example, if the architecture analysis concludes “the security of the system hinges on the transactions being atomic and not being interrupted partway through,” then torn transactions will be become a primary target in adversarial testing. Adversarial tests can be developed according to risk profile—high risk flaws first.
ST3.4
Leverage coverage analysis. Testers measure the code coverage of their security tests in order to identify code that isn’t being exercised. Code coverage drives increased security testing depth. Standard issue black box testing tools achieve exceptionally low coverage, leaving a majority of the software under test unexplored. Don’t let this happen to your tests. By the way, standard measurements for coverage are fine.