Computer Security Resource Center

Computer Security Resource Center

Computer Security
Resource Center

Automated Combinatorial Testing for Software

Combinatorial Testing Tools


This page provides downloads for 

  1. ACTS covering array generator
  2. .NET test configuration generator
  3. Sequence covering array generator

ACTS and CCM test tools - beta release

Updated 14 August 15. The latest release is acts_3.0.jar
(Notice: ACTS was previously named FireEye. Some earlier publications on our site refer to it as FireEye. )


1. Overview

Combinatorial testing can be approached in two ways: generating a test suite of all t-way combinations (for a specified level of t) using ACTS, or measuring the combinations covered by an existing test suite using CCM. Unless a test suite was developed using ACTS or a similar covering array tool, it is unlikely to provide 100% t-way coverage, but t-way coverage may still be sufficiently strong for testing. CCM can be used to determine t-way coverage of an existing test suite, and ACTS can be used to generate new test suites.

(1) ACTS is a combinatorial testing tool that can be used to generate t-way test sets. Major features of ACTS include uniform and mixed strength test set generation, high-level constraints support, and an easy-to-use GUI. ACTS is developed by the US National Institute of Standards and Technology (NIST) and the University of Texas at Arlington.

(2) CCM is a coverage analyzer tool that can be applied to existing tests to determine if they already provide sufficient t-way coverage. It can measure coverage for mixed level arrays, and provides high-level constraints support and an easy-to-use GUI. CCM is developed by US National Institute of Standards and Technology (NIST) and the Centro Nacional de Metrologia of Mexico.

 The software provided on this web site is provided by NIST as a public service.  Please see additional details of disclaimer notice regarding usage at the bottom of this page. 


2. Downloading and installing ACTS

ACTS is available in two forms:  a GUI-based tool for easy definition and generation of test configurations, and a command line version that can be used in scripts or other automated tools.  
The latest release is acts_3.0.jar.

Download:
ACTS is now distributed as a zip file, which can be downloaded here. The zip archive contains the following files: (1) an executable jar file for the GUI version; (2) an executable jar file for the command line version; (3) a release document; and (4) a user guide.  

ACTS  has been primarily developed and tested on Windows PCs. However, the portability feature of Java  allows the tool to run on other platforms such as Unix and Linux.

Platform:  ACTS is written in Java, and requires J2SE 6.0 or above. Both versions are distributed as executable jar files. 

You probably already have Java; if not, it can be downloaded here:  http://java.com    
Just click on "Free Java Download" and install according to the instructions that they provide.

ACTS can be easily installed by putting the jar file into a working directory. To start ACTS, do:

(for GUI version):  double-click the jar file

(for command line version):  java -jar jar_file 

Sanity testing: An XML file that contains a sample configuration file can be downloaded here. (After downloading, change name of file from "tcas_xml.txt" to "tcas.xml".)  After the GUI is opened, load the sample configuration file, and then build the system. A simple user guide for ACTS is contained in the zip file. 


 3. Downloading and installing CCM

The CCM combinatorial coverage analyzer is developed by the US National Institute of Standards and Technology (NIST) and the Centro Nacional de Metrologia (CENAM) of Mexico.  No guarantee of any kind is implied by the distribution of the code.  This tool is available in both .Net and Java.   The current tool is GUI only, but we are developing a command line version for use in test scripts or other offline batch processing. 

User manual

0.8 version released 27 June 13:

Download:
CCM is  distributed as a zip file, which can be downloaded here. The zip archive contains the Windows setup file and binaries for installation.   Double click on the setup file to install.  If you have a previous release of CCM, you may need to uninstall it first.  There is also a Java version  here.  

ACTS  has been primarily developed and tested on Windows PCs. However, the portability feature of Java  allows the tool to run on other platforms such as Unix and Linux.

Platform:  Both .Net and Java  versions of CCM are available:  C# version requires .Net components and a Windows environment.   Java version requires  requires J2SE 6.0 or above.


.Net Test Configuration Generator - beta release

This tool uses the covering array output from ACTS to generate a collection of .Net configuration files covering all t-way combinations of config values.  See help option of the tool for full documentation.  Configuration Generator Tool is developed by the US National Institute of Standards and Technology (NIST).  No guarantee of any kind is implied by the distribution of the code.

zip file of executable


Sequence Covering Array Generator - beta release

This is an alpha release of an experimental tool.  Please report any problems encountered and we'll try to get them resolved quickly.  The tool uses .Net components and requires a Windows environment.   generates sequence covering arrays, useful for event driven systems including GUIs, protocols, hardware testing.   See chapter 5 of the Practical Combinatorial Testing tutorial, here, for an explanation of sequence covering arrays.  

zip file of executable


The research software provided on this web site (“software”) is provided by NIST as a public service. You may use, copy and distribute copies of the software in any medium, provided that you keep intact this entire notice. You may improve, modify and create derivative works of the software or any portion of the software, and you may copy and distribute such modifications or works.  Modified works should carry a notice stating that you changed the software and should note the date and nature of any such change.  Please explicitly acknowledge the National Institute of Standards and Technology as the source of the software.

The software is expressly provided “AS IS.”  NIST MAKES NO WARRANTY OF ANY KIND, EXPRESS, IMPLIED, IN FACT OR ARISING BY OPERATION OF LAW, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, NON-INFRINGEMENT AND DATA ACCURACY.  NIST NEITHER REPRESENTS NOR WARRANTS THAT THE OPERATION OF THE SOFTWARE WILL BE UNINTERRUPTED OR ERROR-FREE, OR THAT ANY DEFECTS WILL BE CORRECTED.  NIST DOES NOT WARRANT OR MAKE ANY REPRESENTATIONS REGARDING THE USE OF THE SOFTWARE OR THE RESULTS THEREOF, INCLUDING BUT NOT LIMITED TO THE CORRECTNESS, ACCURACY, RELIABILITY, OR USEFULNESS OF THE SOFTWARE. 

                You are solely responsible for determining the appropriateness of using and distributing the software and you assume all risks associated with its use, including but not limited to the risks and costs of program errors, compliance with applicable laws, damage to or loss of data, programs or equipment, and the unavailability or interruption of operation.  This software is not intended to be used in any situation where a failure could cause risk of injury or damage to property.  The software was developed by NIST employees  and Employees of University of Texas.  NIST employee contributions are not subject to copyright protection within the United States. 

 

Created May 24, 2016, Updated January 17, 2018