Combinatorial Testing for Building Reliable Systems

Combinatorial testing is an approach where test suites are developed by efficiently covering interactions of parameter values and configuration settings. Multiple studies over the years have shown the interesting phenomenon where almost all defects in a system originate from interactions of a few specific parameters or settings. Efficient algorithms compressing these value combinations into a small number of tests have made this method practical for industrial use, providing better testing at lower cost. Through this approach, fault detection close to the level of exhaustive testing can be achieved with a 20x to 700x reduction in the test suite size. Since most defects in systems can be discovered with systematic testing using 2- to 6-way interactions of parameter values, utilizing this approach can help us develop highly reliable systems.