| Application
|
Reference
|
Notes
|
avionics
|
R. Bartholomew, An
Industry Proof-of-Concept Demonstration of Automated Combinatorial Test,
25th
Annual
IEEE Software Technology Conf., April 8-10, 2013, Salt
Lake City, Utah.
|
“Industry proof-of-concept
demonstration that used this approach to automate parts of the unit and
integration testing of a 196 KSLOC avionics system. The goal was to see
if it might cost-effectively reduce rework by reducing the number of
software defects escaping into system test – if it was adequately
accurate, rigorous, thorough, scalable, mature, easy to learn, easy to
use, etc. Overcoming scalability issues required moderate effort, but
in general it was effective – e.g., generating 47,040 test cases (input
vectors, expected outputs) in 75 seconds, executing and analyzing them
in 2.6 hours. It subsequently detected all seeded defects, and achieved
nearly 100% structural coverage.”
|
rich web
applications
|
Maughan,
C.. Test
Case Generation Using Combinatorial Based Coverage for Rich Web
Applications.
Logan, UT: Utah State
Univ (2012).
|
Compared exhaustive (discretized
values) w/ CT. 2-way tests found all but one fault found by exhaustive
using < 13% of tests required for exhaustive.
|
MP3 audio
player software
|
Zhang, Z., Liu, X., &
Zhang, J. (2012, April). Combinatorial Testing on ID3v2 Tags of MP3
Files. In Software Testing, Verification and Validation
(ICST), 2012 IEEE Fifth International Conference on (pp.
587-590). IEEE.
|
Most faults detected by 1-way and
2-way tests, with one caused by 4-way interaction.
|
| product engineering;
banking & financial services; insurance |
M. Mehta, R.
Philip, Applications of Combinatorial Testing methods for Breakthrough
Results in Software Testing, 2nd
Intl
Workshop on Combinatorial Testing, Luxembourg,
IEEE, Mar. 2013.
|
"Combinatorial
Testing (CT) approach has greatly helped our projects from different
domains to optimize testing effort without compromising on testing
quality. We were able to achieve breakthrough business results. CT
based freeware tools such as All Pairs & ACTS are of great help
for
testing professionals to optimize effort and reduce learning curve."
|
| various |
L. Shikh
Gholamhossein Ghandehari, M. N. Bourazjany, Yu Lei, R.N. Kacker and
D.R. Kuhn, "Applying
Combinatorial Testing to the Siemens Suite", 2nd Intl
Workshop on Combinatorial Testing, Luxembourg,
IEEE, Mar. 2013.
|
"Modeled
the seven programs in the Siemens suite and applied combinatorial
testing to these programs. ... The results show that combinatorial
testing can detect most faulty versions of the Siemens programs, and is
more effective than random testing."
|
| covering array tool
|
Borazjany, M. N., Yu, L., Lei, Y.,
Kacker, R., & Kuhn, R. (2012, April). Combinatorial Testing of
ACTS: A Case Study. In Software Testing, Verification and
Validation (ICST), 2012 IEEE Fifth Intl Conference on (pp.
591-600). IEEE.
|
Applied 2-way and 3-way tests;
approx. 80% module and branch coverage, 88% statement coverage. 15 faults found; none by
3-way not also found in 2-way. |
| failure analysis
|
Cunningham, A. M., Hagar, J.,
& Holman, R. J. A System Analysis Study
Comparing Reverse Engineered Combinatorial Testing to Expert Judgment.
In Software Testing, Verification and Validation (ICST), 2012
IEEE Fifth Intl Conference on (pp. 630-635). IEEE.
|
Lockheed Martin F-16 ventral fin
redesign “The historic analysis was able to determine a set of
combinations, which isolated the problem and tested a solution.
However, the original effort was expensive, time consuming, and
required highly specialized knowledge from the expert to be effective.
… In the study, a series of iterations created combinatorial test cases
which could have 'replicated' the original highly optimized and
successful test program, without the expert.”
|
| aerospace
|
J. Hagar, D.R. Kuhn, R.N. Kacker, Introducing
Combinatorial Testing to a Large System-Software Organization, to
appear, TBD.
|
CT
applied to 8 Lockheed Martin pilot projects in aerospace software. Results: “Our initial
estimate is that this method supported by the technology can save up to
20% of test planning/design costs if done early on a program while
increasing test coverage by 20% to 50%.”
|
web browser
DOM modules
|
C. Montanez, D.R. Kuhn, M. Brady,
R. Rivello, J. Reyes, M.K. Powers, Evaluation
of Fault Detection Effectiveness for Combinatorial and Exhaustive
Selection of Discretized Test Inputs, Software Quality
Professional - June, 2012.
|
Compared exhaustive (discretized
values) w/ CT, 2-way to 6-way. 4-way
tests found all faults using < 5% of tests required for
exhaustive.
|
| avionics |
D. R. Kuhn, V.
Okun, "Pseudo-exhaustive
Testing For Software, 30th NASA/IEEE
Software Engineering Workshop, April 25-27, 2006 |
Proof-of-concept
experiment on pseudo-exhaustive testing, integrating automated test
generation with combinatorial testing. All seeded faults
found with 2-way through 5-way tests.
|
|
|
|
|