Combinatorial coverage - case studies
Combinatorial coverage measures are used in industry for high assurance software used in critical applications. Industry examples include the following:
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Frequency Differencing. NIST Cybersecurity Whitepaper.
- Describes measures of the frequency of combination coverage and difference between Class and Non-class elements in machine learning classification problems. Illustrates application of these methods for identifying weaknesses in physical unclonable function implementations.
Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2021). Combinatorial Coverage Difference Measurement. NIST Cybersecurity Whitepaper.
- Introduces a variety of measures that can be applied to understanding differences in combination coverage.
Elyasaf, A., Farchi, E., Margalit, O., Weiss, G., & Weiss, Y. (2022). Combinatorial Sequence Testing Using Behavioral Programming and Generalized Coverage Criteria. arXiv preprint arXiv:2201.00522.
- "We present a new model-based approach to testing systems whose test vectors are sequences of actions and assertions (called events in this paper). Specifically, we propose an end-to-end solution to testing that includes a method for defining tests, a way to quantify testing quality, and a framework for assessing risks. We use behavioral programming to model what to test and present how its composable and incremental nature makes it effective for designing maintainable test plans. We then formalize a framework that generalizes previous approaches for specifying coverage criteria and show how to extract effective test suites from test plans according to these criteria."
Cody, T., Kauffman, J., Krometis, J., Sobien, D., & Freeman, L. (2022, June). Combinatorial coverage framework for machine learning in multi-domain operations. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications IV (Vol. 12113, pp. 467-471). SPIE.
- "This work presents a framework for using combinatorial coverage for multi-domain operations. We discuss how coverage metrics can incorporate multi-modal meta-data and mission context into fusion processes, how coverage is oriented towards identifying gaps in and between sets of data, and how coverage can identify cases where performance is expected to be difficult. We conclude that combinatorial coverage should be considered a core capability for supporting ML in MDO."
Tzoref-Brill, R., Sinha, S., Nassar, A. A., Goldin, V., & Kermany, H. (2022, April). TackleTest: A Tool for Amplifying Test Generation via Type-Based Combinatorial Coverage. In 2022 IEEE Conference on Software Testing, Verification and Validation (ICST) (pp. 444-455). IEEE.
- "TACKLETEST builds on top of two well-known test-generation tools, EvoSuite and Randoop, by adding a new combinatorial- testing-based approach for computing coverage goals that com- prehensively exercises different parameter type combinations of the methods under test, at configurable interaction levels. We describe the tool architecture, the main tool components, and the combinatorial type-based testing technique."
Fifo, M., Enoiu, E., & Afzal, W. (2019, April). On measuring combinatorial coverage of manually created test cases for industrial software. In 2019 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 264-267). IEEE.
- "The results of this paper suggest that manual test cases created by industrial engineers do not achieve a high combinatorial coverage and can be improved by adding more test cases to cover t-wise interactions at the expense of more test cases to execute."
Wang, Z., Guo, J., Chen, Y., & She, F. (2021, December). The Effect of Combinatorial Coverage for Neurons on Fault Detection in Deep Neural Networks. In 2021 IEEE 21st International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 77-82). IEEE.
- "We conduct an empirical study on MNIST dataset to answer how the combinatorial coverage of neurons affects the fault detection in DNNs. The experimental results show a medium or strong correlation between the fixed- strength combinatorial coverage of neurons and the number of adversarial examples for DNNs. Such a results suggests that it is feasible to use the combinatorial coverage of neurons to guide the testing of DNNs."
Smith, Riley, et al., "Measuring Combinatorial Coverage at Adobe", 2019 IEEE Intl Conf on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 208-215).
- "In this paper, we report the practical application of combinatorial testing to the data collection, compression and processing components of the Adobe analytics product. Consequently, the effectiveness of combinatorial testing for this application is measured in terms of new defects found rather than detecting known defects from previous versions. The results of the application show that combinatorial testing is an effective way to improve validation for these components of Adobe Analytics."
Ericsson S, Enoiu E. Combinatorial Modeling and Test Case Generation for Industrial Control Software using ACTS. 2018 IEEE Intl Conf Software Quality, Reliability and Security (QRS) 2018 Jul 16 (pp. 414-425)
- An analysis of expert-developed tests for Bombardier Transportation. "Our results show that not all combinations of algorithms and interaction strengths could generate a test suite within a realistic cut-off time. The results of the modeling process and the efficiency evaluation of ACTS are useful for practitioners considering to use combinatorial testing for industrial control software as well as for researchers trying to improve the use of such combinatorial testing techniques."
Ozcan, M. Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc., Building Technologies Division. 2017 IEEE Intl Conf on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 208-215). IEEE.
- Applied combinatorial testing to Zero Defect program for industrial control systems, using mixed-strength covering arrays, “resulting in requiring fewer tests for higher strength coverage”.
La Manna, V. P., Segall, I., & Greenyer, J. (2015, September). Synthesizing tests for combinatorial coverage of modal scenario specifications. 2015 ACM/IEEE 18th International Conference on Model Driven Engineering Languages and Systems (MODELS) (pp. 126-135). IEEE.
- "we propose a new test coverage criterion based on scenario interactions. Furthermore, we present a novel technique for automatically synthesizing from Modal Sequence Diagram specifications a minimal set of tests that ensures a maximal coverage of possible t-wise scenario interactions. The technique is evaluated on an example specification from an industrial project."
Price, C., Kuhn, R., Forquer, R., Lagoy, A., Kacker, R., Evaluating the t-way Combinatorial Technique for Determining the Thoroughness of a Test Suite, NASA IV&V Workshop, 2013.
