U.S. flag   An official website of the United States government
Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.


Secure .gov websites use HTTPS
A lock (Dot gov) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Combinatorial Testing

Combinatorial coverage - case studies

Combinatorial coverage measures are used in industry for high assurance software used in critical applications.  Industry examples include the following:

Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Frequency Differencing. NIST Cybersecurity Whitepaper.
- D
escribes measures of the frequency of combination coverage and difference between Class and Non-class elements in machine learning classification problems.  Illustrates application of these methods for identifying weaknesses in physical unclonable function implementations. 

Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Coverage Difference Measurement. NIST Cybersecurity Whitepaper.
Introduces a variety of measures that can be applied to understanding differences in combination coverage.

Smith, Riley, et al., "Measuring Combinatorial Coverage at Adobe", 2019 IEEE Intl Conf on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 208-215). 
- "In this paper, we report the practical application of combinatorial testing to the data collection, compression and processing components of the Adobe analytics product. Consequently, the effectiveness of combinatorial testing for this application is measured in terms of new defects found rather than detecting known defects from previous versions. The results of the application show that combinatorial testing is an effective way to improve validation for these components of Adobe Analytics."

Ericsson S, Enoiu E. Combinatorial Modeling and Test Case Generation for Industrial Control Software using ACTS. 2018 IEEE Intl Conf Software Quality, Reliability and Security (QRS) 2018 Jul 16 (pp. 414-425)
- An analysis of expert-developed tests for Bombardier Transportation. "Our results show that not all combinations of algorithms and interaction strengths could generate a test suite within a realistic cut-off time. The results of the modeling process and the efficiency evaluation of ACTS are useful for practitioners considering to use combinatorial testing for industrial control software as well as for researchers trying to improve the use of such combinatorial testing techniques."

Ozcan, M. Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc., Building Technologies Division.   2017 IEEE Intl Conf on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 208-215). IEEE.  
- Applied combinatorial testing to Zero Defect program for industrial control systems, using mixed-strength covering arrays, “resulting in requiring fewer tests for higher strength coverage”. 






Created May 24, 2016, Updated May 10, 2022