Combinatorial Methods for Trust and Assurance

Downloadable Tools

Research tools to support combinatorial testing. No license is required and there are no restrictions on distribution or use. All software is provided free of charge and will remain free in the future. NIST is an agency of the US Government, so this software is public domain. You are free to include it and redistribute it in commercial products if desired. ACTS is in Java, and will run on any platform with the latest Java release (Java free download is at java.com).  There are many users running current Windows, Mac OS, and Linux systems. 

ACTS Basic version download:  Java .jar file. No license is required and there are no restrictions on distribution or use. All software is provided free of charge and will remain free in the future.

Combination Frequency Difference tool download: Python .py files. Please send a request to M S Raunak -- raunak@nist.gov with your name and the name of your organization. No other information is required, but we like to have a list of organizations to show our management where the software is being used. We will send you a download link.   No license is required and there are no restrictions on distribution or use. All software is provided free of charge and will remain free in the future.

To obtain the advanced ACTS tool, which supports constraints, please send a request to Rick Kuhn - kuhn@nist.gov with your name and the name of your organization. No other information is required, but we like to have a list of organizations to show our management where the software is being used. We will send you a download link.   No license is required and there are no restrictions on distribution or use. All software is provided free of charge and will remain free in the future.

To obtain the CAGen tool, please see the SBA Research site:  CAGen.  The site includes a web interface to try the award winning tool online. 

Other tools are available on Github:  https://github.com/usnistgov/combinatorial-testing-tools.  These provide additional capabilities that can be used with ACTS or independently. See the Software descriptions below for more information on these. 

Tutorials and Guides

• Tutorial on combinatorial methods for software testing (pdf in English or Español)
• ACTS User Guide - how to use the ACTS test generation tool
• Combinatorial Coverage Measurement - explains various coverage measurements and how to use the tool for computing these.  There is also a manual for the command line version of the CCM tool. 
• Fault ID user manual - for tool that helps identify likely fault-triggering combinations in failing tests
• PEV tool user manual - testing rule-based expert systems or business rule engine/workflow systems.
• Combination Frequency Difference (CFD) tool - The CFD is a Python-based tool designed for analyzing and detecting statistically significant value combinations in two sets of data identified as a class and a non-class datasets. This prototype calculates and identifies notable value interactions using user-defined parameters and provides options to visualize the results.

Software

• Advanced Combinatorial Testing System (ACTS) - generates test sets that ensure t-way coverage of input parameter values; includes support for constraints and variable-strength tests.  Both a GUI and command-line version are included. (runs on Java platforms)
• CAgen - a high performance t-way test generation tool, compatible with other combinatorial testing tools and is available as a command line tool and Web GUI. CAgen aims to make combinatorial testing more efficient and more accessible and user-friendly.      |CAgen tool web interface and download|
• Combinatorial Coverage Measurement - computes a number of coverage measures of an existing test set.  Both at GUI and command-line version are included. (runs on Java platforms)
• Web app Testing Tool - CPUT (Combinatorial-based Prioritization of User-session-based Testsuites) applies combinatorial methods to testing web applications. Test prioritization is used to make web app testing much more manageable. (runs on Java platforms)
• Combinatorial Sequence Test Generator - generates sequence covering arrays, useful for event driven systems including GUIs, protocols, hardware testing. (C source, compile for target platform)
• Fault Identification tool - helps to narrow down the set of possible fault-triggering combinations in failing tests
• Access Control Policy Test (ACPT) - uses combinatorial testing with model checking to produce tests for access control policies (contact Vincent Hu - vincent.hu@nist.gov)
• PEV tool for testing rule-based expert systems or business rule engine/workflow systems.  See also the PEV user manual for more on this tool. 

• A version of the NASA Automated Requirements Measurement tool developed by P Laplante can be found here ARM.   It takes as input a conventionally structured software or systems requirements specification (SRS) and analyzes it for various text-based properties.

To obtain the tools, please send a request to Rick Kuhn - kuhn@nist.gov  including your name and the name of your organization. No other information is required, but we like to have a list of organizations so that we can show our management where the software is being used. We will send you a download link.

User Feedback

Users have been very positive, and are applying ACTS to a wide variety of software.

• "The tool is pretty easy to use, and has already reduced our planned number of tests by 20% or so.
• "At present I am working with a development team to enable automated testing of a complex rating algorithm under development within a property/casualty insurance system. The system component under test accepts several hundred independent input variables which are used to produce rating factors as output.  I have had some opportunity to use [NIST-ACTS] as a means to calculate test coverage and identify missing test cases in our current population of test data. I'm pleased to report that the combinatorial coverage data has proven useful to us in identifying gaps. Eventually, I would like to use this data to extend our test coverage as well -- at coverage strength 3 we're producing a volume of test cases that should be manageable when paired with automated test data generation. I can also foresee other uses which would pair automated testing tools with [NIST-ACTS] output to improve our test coverage for other systems."
• "Our feedback on [NIST-ACTS] is extremely positive. The tool implement in a very efficient way the algorithm of N-Wise reduction we were looking at. In fact, we would be interested in including your technology in our tool [ ]."
• "[NIST-ACTS] is a good application. It was easy to figure out and use. ... I would use this application again."
• "I've fund it pretty intuitive and greatly helped to reduce the test size. I'll continue to use this tool at next opportunity."
• "Not only does [NIST-ACTS] handle phenomenal combinatorial complexity without breaking a sweat, it is also very easy and straightforward to use. The user interface is very well thought out, intuitive, and delightfully uncluttered.