Combinatorial Testing for AI-Enabled Systems
Talks from Workshop on Combinatorial Testing for Artificial Intelligence-Enabled Systems
September 4, 2024
Virginia Tech Research Center, Arlington, VA
https://sites.google.com/vt.edu/ct-workshop
The goal of this workshop was to provide practitioners and researchers with a foundational understanding of combinatorial testing techniques and applications to testing AI-enabled software systems (AIES). Participants included staff from Cybersecurity and Infrastructure Security Agency (CISA), Office of Sec. of Defense, Director Operational Test & Evaluation (OSD/DOT&E), George Mason University, Johns Hopkins University Applied Physics Laboratory (JHU/APL), the Institute for Defense Analyses (IDA), and the Nuclear Regulatory Commission (NRC).
Tutorial and discussion topics included:
-
Introduction to combinatorial testing (CT), an overview of applying CT to test traditional software systems, including real-world examples and case studies.
-
How Test and Evaluation (T&E) of AIES differ from traditional software systems due to the data-driven nature of these systems and large input space, and how combinatorial testing methods can be applied.
-
Role of combinatorial coverage in data assurance across the lifecycle of AIES, including practical exercises with the Coverage of Data Explorer (CoDEX) tool.
Presentations:
