The goal of this project is to provide practitioners and researchers with a foundational understanding of combinatorial testing techniques and applications to testing AI-enabled software systems (AIES). Resources are being developed in these areas:
-
Combinatorial testing (CT), applying CT to test traditional software systems, including real-world examples and case studies.
-
How Test and Evaluation (T&E) of AIES differ from traditional software systems due to the data-driven nature of these systems and large input space, and how combinatorial testing methods can be applied.
-
Role of combinatorial coverage in data assurance across the lifecycle of AIES, including practical exercises with the Coverage of Data Explorer (CoDEX) tool.
Introductory tutorials:
A kickoff workshop for this project was held Sept 4, 2024, at the Virginia Tech Arlington center: https://sites.google.com/vt.edu/ct-workshop. Participants included staff from Cybersecurity & Infrastructure Security Agency (CISA), Office of Secretary of Defense, Director, Operational Test & Evaluation (DOT&E), George Mason University (GMU), Johns Hopkins University Applied Physics Laboratory (JHU/APL), Institute for Defense Analyses (IDA), Nuclear Regulatory Commission (NRC), the Software Engineering Institute (SEI), NIST, and VIrginia Tech. Attendees considered the workshop a strong success, with useful and practical knowledge provided. A follow-on workshop may be developed in the future, possibly with additional time, more example applications, and a broader range of machine learning examples.
