Case studies - combinatorial and pairwise testing

Application Reference Notes
web administrationN. Condori-Fernandez, T. Vos, P.M. Kruse, E. Brosse, A. Bagnato. Analyzing the Applicability of a Combinatorial Testing Tool in an Industrial Environment, Tech. Rpt. UU-CS-2014-008, May 2014, Univ. of Utrecht. "The main outcomes of the presented study are: (1) with the test suite designed with the CTE, the testers were able to find faults that the traditional test suites did not find, one of them a severe fault; (2) the company realized that the current coverage metrics used for evaluating the quality of test suites needs to be changed to a more sophisticated one; (3) SOFTEAM’s motivation to do more case studies with the CTE is high"
 
avionics
 
R. Bartholomew, An Industry Proof-of-Concept Demonstration of Automated Combinatorial Test, 
25th Annual IEEE Software Technology Conf., April 8-10, 2013, Salt Lake City, Utah.

 
 
“Industry proof-of-concept demonstration that used this approach to automate parts of the unit and integration testing of a 196 KSLOC avionics system. The goal was to see if it might cost-effectively reduce rework by reducing the number of software defects escaping into system test – if it was adequately accurate, rigorous, thorough, scalable, mature, easy to learn, easy to use, etc. Overcoming scalability issues required moderate effort, but in general it was effective – e.g., generating 47,040 test cases (input vectors, expected outputs) in 75 seconds, executing and analyzing them in 2.6 hours. It subsequently detected all seeded defects, and achieved nearly 100% structural coverage.”


rich web
applications
Maughan, C.. Test Case Generation Using Combinatorial Based Coverage for Rich Web Applications.  Logan, UT: Utah State Univ (2012).
   
Compared exhaustive (discretized values) w/ CT. 2-way tests found all but one fault found by exhaustive using < 13% of tests required for exhaustive.

MP3 audio
player software
Zhang, Z., Liu, X., & Zhang, J. (2012, April). Combinatorial Testing on ID3v2 Tags of MP3 Files. In Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth International Conference on (pp. 587-590). IEEE.  
 
Most faults detected by 1-way and 2-way tests, with one caused by 4-way interaction. 
product engineering; banking & financial services; insurance M. Mehta, R. Philip, Applications of Combinatorial Testing methods for Breakthrough Results in Software Testing, 2nd Intl Workshop on Combinatorial Testing, Luxembourg, IEEE, Mar. 2013.
   
"Combinatorial Testing (CT) approach has greatly helped our projects from different domains to optimize testing effort without compromising on testing quality. We were able to achieve breakthrough business results. CT based freeware tools such as All Pairs & ACTS are of great help for testing professionals to optimize effort and reduce learning curve." 

various L. Shikh Gholamhossein Ghandehari, M. N. Bourazjany, Yu Lei, R.N. Kacker and D.R. Kuhn, "Applying Combinatorial Testing to the Siemens Suite", 2nd Intl Workshop on Combinatorial Testing, Luxembourg, IEEE, Mar. 2013. 
 
"Modeled the seven programs in the Siemens suite and applied combinatorial testing to these programs. ... The results show that combinatorial testing can detect most faulty versions of the Siemens programs, and is more effective than random testing."
covering array tool Borazjany, M. N., Yu, L., Lei, Y., Kacker, R., & Kuhn, R. (2012, April). Combinatorial Testing of ACTS: A Case Study. In Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth Intl Conference on (pp. 591-600). IEEE.
 
Applied 2-way and 3-way tests; approx. 80% module and branch coverage, 88% statement coverage.  15 faults found; none by 3-way not also found in 2-way.
failure analysis Cunningham, A. M., Hagar, J., & Holman, R. J.  A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment. In Software Testing, Verification and Validation (ICST), 2012 IEEE Fifth Intl Conference on (pp. 630-635). IEEE.
 
Lockheed Martin F-16 ventral fin redesign “The historic analysis was able to determine a set of combinations, which isolated the problem and tested a solution. However, the original effort was expensive, time consuming, and required highly specialized knowledge from the expert to be effective. … In the study, a series of iterations created combinatorial test cases which could have 'replicated' the original highly optimized and successful test program, without the expert.”

aerospace J. Hagar, D.R. Kuhn, R.N. Kacker, Introducing Combinatorial Testing to a Large System-Software Organization, to appear, TBD. 
 
 
CT applied to 8 Lockheed Martin pilot projects in aerospace software.  Results: “Our initial estimate is that this method supported by the technology can save up to 20% of test planning/design costs if done early on a program while increasing test coverage by 20% to 50%.”
 
web browser
DOM modules
C. Montanez, D.R. Kuhn, M. Brady, R. Rivello, J. Reyes, M.K. Powers, Evaluation of Fault Detection Effectiveness for Combinatorial and Exhaustive Selection of Discretized Test Inputs,  Software Quality Professional  - June, 2012.
 

Compared exhaustive (discretized values) w/ CT, 2-way to 6-way.  4-way tests found all faults using < 5% of tests required for exhaustive. 
avionics D. R. Kuhn, V. Okun, "Pseudo-exhaustive Testing For Software, 30th NASA/IEEE Software Engineering Workshop, April 25-27, 2006 Proof-of-concept experiment on pseudo-exhaustive testing, integrating automated test generation with combinatorial testing.  All seeded faults found with 2-way through 5-way tests.