U.S. flag   An official website of the United States government
Dot gov

Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Https

Secure .gov websites use HTTPS
A lock (Dot gov) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

Combinatorial Testing

Combinatorial coverage examples

Examples of combinatorial coverage achieved by real-world test suites in various application domains.  The test suites studied in these examples were designed using conventional methods, i.e., they were not developed using ACTS or another covering array tool.  

Application

Config

t = 2

t = 3

t = 4

t = 5

t = 6

Reference

Spacecraft control

132754262

0.940

0.831

0.668

0.536

 

Maximoff, J. R., Kuhn, D. R., Trela, M. D., & Kacker, R. A method for analyzing system state-space coverage within a t-wise testing framework. In 2010 IEEE ICST.

Spacecraft component control 

273241102

0.962

0.917

0.819

0.658

0.467

Price, C., Kuhn, R., Forquer, R., Lagoy, A., Kacker, R., Evaluating the t-way Combinatorial Technique for Determining the Thoroughness of a Test Suite, NASA IV&V Workshop, 2013.  

Aircraft system  test

243142

1.000

1.000

0.987

0.697

 

Kuhn, D. R., Kacker, R. N., & Lei, Y. (2015). Combinatorial coverage as an aspect of test quality. CrossTalk28(2), 19-23.

Apache Lang   .686 .440       Felbinger, H., Wotawa, F., & Nica, M. (2018, April). Adapting unit tests by generating combinatorial test data. In 2018 IEEE ICST (ICSTW) (pp. 352-355). IEEE.
Apache Math   .628 .301       same
JFreeChart   .577 .250       same
Joda-Time   .500 .250       same
Train control  

0.830

0.550

0.320

0.180

0.100

Fifo, M., Enoiu, E., & Afzal, W. (2019, April). On measuring combinatorial coverage of manually created test cases for industrial software. In 2019 IEEE ICST (ICSTW) (pp. 264-267). 

Data analytics

 

0.160

0.030

      Smith, R., Jarman, D., Bellows, J., Kuhn, R., Kacker, R., & Simos, D. Measuring combinatorial coverage at adobe. In 2019 IEEE ICST (ICSTW)

Data analytics

 

0.650

0.390

     

same

Data analytics

 

0.120

0.010

     

same

Data analytics

 

0.120

0.010

     

same

Data analytics

 

0.230

0.060

     

same

Data analytics

 

0.270

0.090

     

same

Created May 24, 2016, Updated November 01, 2022