- csrc home
- news & events
Combinatorial methods have been demonstrated to significantly reduce the cost and increase the quality of software testing As part of NIST's mission to advance the science of testing and measurement, we have conducted extensive analyses of software failure in real-world systems. Combinatorial testing takes advantage of the interaction rule, which is based on our analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. Two primary tools can be used: ACTS, which generates tests, and CCM, to measure combinatorial aspects of test quality.
Tools provided freely on this site are used by hundreds of organizations around the world for software and hardware testing of reliability, safety, and security.
Combinatorial testing poster summarizes the methodology.
|Tool releases, May 2013: Version
2.8 of ACTS
testing tool and Version 0.8
CCM coverage measurement tool now available.
Talks for 2013: Carnegie Mellon University, Feb 12 (Kuhn), Loyola University, Feb 26 (Kuhn) ; IEEE ECS seminar, Singapore, July 16 (Kacker)
Combinatorial Coverage Measurement - NIST IR 7878 (released Sept. 2012)
2nd International Workshop on Combinatorial Testing association with 5th IEEE International Conference on Software Testing, Verification and Validation (ICST 2013, March 18-22) Luxembourg.
Some of our accomplishments to date include:
2009 Excellence in
Software on this site is free
of charge and will remain free in the future. No
license is required and there are no restrictions on use.
an agency of the United
We have over 1,300 users as of April 2013, in nearly all major industries. Here is a breakdown of our user base.
1) Combinatorial testing tool, for generating test suites. Advanced Combinatorial Testing System (ACTS) can compute tests for 2-way through 6-way interactions. An easy-to-use GUI is included. More here. A comparison of ACTS with similar tools shows that ACTS produces smaller test sets (with the same degree of coverage) and is faster than others. To request a copy, send email to Rick Kuhn - firstname.lastname@example.org. Please include your first and last name, and company or university name (this helps us with management support for the project!)
2) Combinatorial coverage measurement tool, for evaluating quality of test suites. Useful for gaining the advantages of combinatorial testing without disrupting existing test practice. The CCM measurement tool can analyze existing tests for 2-way through 6-way interactions they already have. An easy-to-use GUI is included. To request a copy, send email to Rick Kuhn - email@example.com. Please include your first and last name, and company or university name (this helps us with management support for the project!)
3) Testing tutorial, Practical Combinatorial Testing, NIST SP800-142 (pdf 81 pages, 841Kb). This publication provides a self-contained tutorial on using combinatorial testing for real-world software. It introduces the key concepts and methods, explains use of software tools for generating combinatorial tests, and discusses advanced topics. The material is accessible to an undergraduate student of computer science or engineering, and includes an extensive set of references to papers that provide more depth on each topic. Oct. 2010. Public domain, distribution unlimited; 81 pages. Over 22,000 downloads!
4) Security policy testing tool. The Access Control Policy Test (ACPT) tool allows policy authors to conveniently specify access control models (such as RBAC and Multi-Level models) and rules as well as access control properties. From the specified models and rules, the ACPT tool automatically synthesizes deployable policies in XACML and generates combinatorial tests to verify security policy implementations. Complete test cases are generated, consisting of test inputs and expected output for each set of inputs. ACPT uses ACTS to provide 2-way to 4-way combinatorial testing of policies. To request a copy, send email to Vincent Hu - firstname.lastname@example.org.
5) Web app testing tool. CPUT (Combinatorial-based Prioritization of User-session-based Testsuites) applies combinatorial methods to testing web applications. Test prioritization is used to make web app testing much more manageable. The tool allows testers to easily collect, prioritize, and reduce user-session-based test cases. CPUT provides (1) guidance to users on how to configure their web server to log important usage information, (2) automated parsing of web logs into XML formatted test cases that can be used by test replay tools, (3) automated prioritization of test cases by length-based and combinatorial-based criteria, and (4) automated reduction of testcases by combinatorial coverage.