- csrc home
- news & events
Combinatorial methods have been shown to significantly reduce cost and increase quality for software and system testing. The basis for combinatorial testing is the interaction rule, which is based on analysis of thousands of software failures. The rule states that most failures are induced by single factor faults or by the joint combinatorial effect (interaction) of two factors, with progressively fewer failures induced by interactions between three or more factors. Therefore if all faults in a system can be induced by a combination of t or fewer parameters, then testing all t-way combinations of parameter values is pseudo-exhaustive and provides a high rate of fault detection. Two primary tools can be used: ACTS, to generate tests, and CCM, to measure combinatorial aspects of test quality.
Talks for 2014 (Kuhn): Cloud Tech & Government IT, Defense Strategies Inst., Sept. 23-24, Alexandria; NASA IV&V Workshop, Sept 9-11, Fairmont, WV; Cyber Intelligence USA, June 20, Arlington; Cloud for Defense Symposium, Apr. 25, Alexandria; NITRD/Natl Science Foundation, Apr. 24; IEEE Intl Conf on Software Testing keynote, Apr. 2, Cleveland, American Software Testing Qualifications Board keynote, Mar. 26, San Francisco; US Census Bureau, Mar 13, Carnegie Mellon University, Feb 12
Video of 2013 NASA IV&V conference briefing on application of combinatorial coverage measurement. (Charley Price and Rick Kuhn)
Tool releases, May 2014: Version 2.9 of ACTS testing tool and Version 0.8 of CCM combinatorial coverage measurement tool now available.
Combinatorial Coverage Measurement - NIST IR 7878 (released Sept. 2012)
Tools provided freely on this site are used by hundreds of organizations around the world for software and hardware testing of reliability, safety, and security. Our research collaborations include the U. of Texas Arlington, US Air Force, Carnegie Mellon, U. of North Texas, Johns Hopkins University Applied Physics Lab, U. of Maryland Baltimore County, Centro Nacional de Metrologia of Mexico, NASA, and the U. of Maryland University College.
This article explains how the approach is related to statistical Design of Experiments.
Combinatorial testing poster summarizes the methodology.
Some of our accomplishments to date include:
2009 Excellence in
Software on this site is free
of charge and will remain free in the future. No
license is required and there are no restrictions on use.
an agency of the United
We have over 1,700 users as of June 2014, in nearly all major industries. Here is a breakdown of our user base.
1) Combinatorial testing tool, for generating test suites. Advanced Combinatorial Testing System (ACTS) can compute tests for 2-way through 6-way interactions. An easy-to-use GUI is included. A comparison of ACTS with similar tools shows that ACTS produces smaller test sets (with the same degree of coverage) and is faster than others. ACTS was developed by NIST and the University of Texas at Arlington. To request a copy, send email to Rick Kuhn - firstname.lastname@example.org. Please include your first and last name, and company or university name (this helps us with management support for the project!)
2) Combinatorial coverage measurement tool, for evaluating quality of test suites. Useful for gaining the advantages of combinatorial testing without disrupting existing test practice. The CCM measurement tool can analyze existing tests for 2-way through 6-way interactions they already have. An easy-to-use GUI is included. CCM was developed by NIST and the Centro Nacional de Metrologia of Mexico. To request a copy, send email to Rick Kuhn - email@example.com. Please include your first and last name, and company or university name (this helps us with management support for the project!)
3) Testing tutorial, Practical Combinatorial Testing, NIST SP800-142 (pdf 81 pages, 841Kb). This publication provides a self-contained tutorial on using combinatorial testing for real-world software. It introduces the key concepts and methods, explains use of software tools for generating combinatorial tests, and discusses advanced topics. The material is accessible to an undergraduate student of computer science or engineering, and includes an extensive set of references to papers that provide more depth on each topic. Oct. 2010. Public domain, distribution unlimited; 81 pages.
4) Security policy testing tool. The Access Control Policy Test (ACPT) tool allows policy authors to conveniently specify access control models (such as RBAC and Multi-Level models) and rules as well as access control properties. From the specified models and rules, the ACPT tool automatically synthesizes deployable policies in XACML and generates combinatorial tests to verify security policy implementations. Complete test cases are generated, consisting of test inputs and expected output for each set of inputs. ACPT uses ACTS to provide 2-way to 4-way combinatorial testing of policies. To request a copy, send email to Vincent Hu - firstname.lastname@example.org.
5) Web app testing tool. CPUT (Combinatorial-based Prioritization of User-session-based Testsuites) applies combinatorial methods to testing web applications. Test prioritization is used to make web app testing much more manageable. The tool allows testers to easily collect, prioritize, and reduce user-session-based test cases. CPUT provides (1) guidance to users on how to configure their web server to log important usage information, (2) automated parsing of web logs into XML formatted test cases that can be used by test replay tools, (3) automated prioritization of test cases by length-based and combinatorial-based criteria, and (4) automated reduction of testcases by combinatorial coverage.