Combinatorial Testing

Autonomous Systems Assurance

Autonomous systems must function correctly in an enormous range of environments.  For example, self-driving cars must deal with lighting, rain, fog, pedestrians, animals, other vehicles, road markings, signs, etc.  How do we ensure that autonomous systems are safe in such complex and rapidly changing environments, when conventional test coverage and formal verification methods cannot be applied?  

Achieving assured autonomy in any environment requires methods for measuring the input space, to show that the test environment adequately covers real-world conditions that may be encountered.  Although some statistical and structural coverage metrics are relevant, they are terribly inadequate for many of the challenges in autonomous systems assurance.  NIST is developing new combinatorial measurement methods and tools for input space coverage, to fill this key gap in current software engineering capabilities and provide assured autonomy. 

• Explainable AI, Verification, and Validation
• Rule-based Expert Systems and Formal Methods
• Case studies