The goal of this project is to provide practitioners and researchers with a foundational understanding of combinatorial testing techniques and applications to testing AI-enabled software systems (AIES). Resources are being developed in these areas:
- Combinatorial testing (CT), applying CT to test traditional software systems, including real-world examples and case studies.
- How Test and Evaluation (T&E) of AIES differ from traditional software systems due to the data-driven nature of these systems and large input space, and how combinatorial testing methods can be applied.
- Role of combinatorial coverage in data assurance across the lifecycle of AIES, including practical exercises with the Coverage of Data Explorer (CoDEX) tool
AI-enabled systems must function correctly in an enormous range of environments. For example, self-driving cars must deal with lighting, rain, fog, pedestrians, animals, other vehicles, road markings, signs, etc. How do we ensure that autonomous systems are safe in such complex and rapidly changing environments, when conventional test coverage and formal verification methods cannot be applied?
Achieving assured autonomy in any environment requires methods for measuring the input space, to show that the test environment adequately covers real-world conditions that may be encountered. Although some statistical and structural coverage metrics are relevant, they are terribly inadequate for many of the challenges in autonomous systems assurance. We are developing new combinatorial measurement methods and tools for input space coverage, to fill this key gap in current software engineering capabilities and provide safety, security, and reliability of AI-enabled systems.
Introductory tutorials:
A kickoff workshop for this project was held Sept 4, 2024, at the Virginia Tech Arlington center: https://sites.google.com/vt.edu/ct-workshop. Participants included staff from Cybersecurity & Infrastructure Security Agency (CISA), Office of Secretary of Defense, Director, Operational Test & Evaluation (DOT&E), George Mason University (GMU), Johns Hopkins University Applied Physics Laboratory (JHU/APL), Institute for Defense Analyses (IDA), Nuclear Regulatory Commission (NRC), the Software Engineering Institute (SEI), NIST, and VIrginia Tech. Attendees considered the workshop a strong success, with useful and practical knowledge provided. A follow-on workshop may be developed in the future, possibly with additional time, more example applications, and a broader range of machine learning examples.
Recent Publications
- Chandrasekaran, J., Lanus, E., Cody, T., Freeman, L. J., Kacker, R. N., Raunak, M. S., & Kuhn, D. R. (2024). Leveraging Combinatorial Coverage in the Machine Learning Product Lifecycle. Computer, 57(7), 16-26.
- Kuhn, D. R., Raunak, M. S., Kacker, R. N., Chandrasekaran, J., Lanus, E., Cody, T., & Freeman, L. (2024). Assured Autonomy Through Combinatorial Methods. Computer, 57(5), 86-90.
- Lanus, E., Freeman, L. J., Kuhn, D. R., & Kacker, R. N. (2021, April). Combinatorial testing metrics for machine learning. In 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 81-84). IEEE.
