Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

NIST CSWP 22 (Initial Public Draft)

Combination Frequency Differencing

Date Published: December 6, 2021
Comments Due: February 7, 2022 (public comment period is CLOSED)
Email Questions to:


Richard Kuhn (NIST), M S Raunak (NIST), Raghu Kacker (NIST)


Combinatorial coverage measures have been defined and applied to a wide range of problems, including fault location and evaluating the adequacy of test inputs and input space models. More recently, methods applying coverage measures have been used in applications of artificial intelligence and machine learning for explainability and analyzing aspects of transfer learning. These methods have been developed using measures that depend on the inclusion or absence of t-tuples of values in inputs and test cases.

This paper introduces a new method related to combinatorial testing and measurement, combination frequency differencing (CFD), and illustrates the use of CFD in machine learning applications. This method is particularly well-suited to artificial intelligence and machine learning applications, where training data sets used in learning systems are dependent on the prevalence of various attributes of elements of class and non-class sets. This paper illustrates the use of this method by applying it to analyzing physical unclonable functions (PUFs) for bit combinations that have a disproportionately strong influence on PUF response bit values. Additionally, it is shown that combination frequency differences provide a simple but effective algorithm for classification problems.



combinatorial coverage; combination frequency difference; combinatorial testing; physical unclonable function (PUF); unclonable
Control Families

None selected



Supplemental Material:
Draft White Paper (pdf)

Document History:
12/06/21: CSWP 22 (Draft)