- csrc home
- news & events
Combinatorial methods have been demonstrated to significantly reduce the cost and increase the quality of software and system testing As part of NIST's mission to advance the science of testing and measurement, we have conducted extensive analyses of software failure in real-world systems. Combinatorial testing takes advantage of the interaction rule, which is based on our analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. Two primary tools can be used: ACTS, to generate tests, and CCM, to measure combinatorial aspects of test quality.
This article explains how the approach is related to statistical Design of Experiments.
Tools provided freely on this site are used by hundreds of organizations around the world for software and hardware testing of reliability, safety, and security. Our research collaborations include the U. of Texas Arlington, US Air Force, Carnegie Mellon, U. of North Texas, Johns Hopkins University Applied Physics Lab, U. of Maryland Baltimore County, the Centro Nacional de Metrologia of Mexico, NASA, and the U. of Maryland University College.
Combinatorial testing poster summarizes the methodology.
|Quick Start, for those new to combinatorial testing. It's easy to learn the basics of this approach.
1. 3-page summary of the method.
2. An extended how-to tutorial that provides more depth (81 pgs.).
3. Briefing slides (approx. 60 min).
Workshop in 2014 The 3rd International Workshop on Combinatorial Testing will be held in Cleveland, Ohio, USA, in association with the IEEE International Conference on Software Testing, Verification and Validation (ICST) 2014.
New book We have published the first textbook on combinatorial methods in software testing, Introduction to Combinatorial Testing, Rick Kuhn, Raghu Kacker, Yu "Jeff" Lei, with chapter contributions from Renee Bryce, Eduardo Miranda, Sreedevi Sampath, and George Sherwood. (CRC Press, ISBN 1466552298, June 2013; 319 pgs.). The book provides software testers, developers, and students with a self-contained tutorial on how to use these methods for real-world software. "I thoroughly recommend it to anyone involved in the practice of software testing." - ACM Computing Reviews
Talks for 2014: ICST keynote, Apr. 2 (Kuhn), ASTQB keynote, Mar. 26 (Kuhn), US Census Bureau, Mar 13 (Kuhn), Carnegie Mellon University, Feb 12 (Kuhn)
Tool releases, May 2013: Version 2.8 of ACTS testing tool and Version 0.8 of CCM combinatorial coverage measurement tool now available.
1st Asian Workshop on Combinatorial Testing (PPT in zip file)
Combinatorial Coverage Measurement - NIST IR 7878 (released Sept. 2012)
2nd International Workshop on Combinatorial Testing association with 5th IEEE International Conference on Software Testing, Verification and Validation (ICST 2013, March 18-22) Luxembourg.
Some of our accomplishments to date include:
2009 Excellence in
Software on this site is free
of charge and will remain free in the future. No
license is required and there are no restrictions on use.
an agency of the United
We have over 1,400 users as of June 2013, in nearly all major industries. Here is a breakdown of our user base.
1) Combinatorial testing tool, for generating test suites. Advanced Combinatorial Testing System (ACTS) can compute tests for 2-way through 6-way interactions. An easy-to-use GUI is included. More here. A comparison of ACTS with similar tools shows that ACTS produces smaller test sets (with the same degree of coverage) and is faster than others. ACTS was developed by NIST and the University of Texas at Arlington. To request a copy, send email to Rick Kuhn - firstname.lastname@example.org. Please include your first and last name, and company or university name (this helps us with management support for the project!)
2) Combinatorial coverage measurement tool, for evaluating quality of test suites. Useful for gaining the advantages of combinatorial testing without disrupting existing test practice. The CCM measurement tool can analyze existing tests for 2-way through 6-way interactions they already have. An easy-to-use GUI is included. CCM was developed by NIST and the Centro Nacional de Metrologia of Mexico. To request a copy, send email to Rick Kuhn - email@example.com. Please include your first and last name, and company or university name (this helps us with management support for the project!)
3) Testing tutorial, Practical Combinatorial Testing, NIST SP800-142 (pdf 81 pages, 841Kb). This publication provides a self-contained tutorial on using combinatorial testing for real-world software. It introduces the key concepts and methods, explains use of software tools for generating combinatorial tests, and discusses advanced topics. The material is accessible to an undergraduate student of computer science or engineering, and includes an extensive set of references to papers that provide more depth on each topic. Oct. 2010. Public domain, distribution unlimited; 81 pages. Over 24,000 downloads!
4) Security policy testing tool. The Access Control Policy Test (ACPT) tool allows policy authors to conveniently specify access control models (such as RBAC and Multi-Level models) and rules as well as access control properties. From the specified models and rules, the ACPT tool automatically synthesizes deployable policies in XACML and generates combinatorial tests to verify security policy implementations. Complete test cases are generated, consisting of test inputs and expected output for each set of inputs. ACPT uses ACTS to provide 2-way to 4-way combinatorial testing of policies. To request a copy, send email to Vincent Hu - firstname.lastname@example.org.
5) Web app testing tool. CPUT (Combinatorial-based Prioritization of User-session-based Testsuites) applies combinatorial methods to testing web applications. Test prioritization is used to make web app testing much more manageable. The tool allows testers to easily collect, prioritize, and reduce user-session-based test cases. CPUT provides (1) guidance to users on how to configure their web server to log important usage information, (2) automated parsing of web logs into XML formatted test cases that can be used by test replay tools, (3) automated prioritization of test cases by length-based and combinatorial-based criteria, and (4) automated reduction of testcases by combinatorial coverage.