Computer Security Resource Center

Computer Security Resource Center

Computer Security
Resource Center

Automated Combinatorial Testing for Software

Papers

Papers

  • Kuhn, D.R., Hu, V., Ferraiolo, D., Kacker, R., Lei, Y., Pseudo-exhaustive Testing of Attribute Based Access Control Rules, (Aug. 2015) Preprint, to be submitted Intl Workshop on Combinatorial Testing.
  • Hagar, J. D., Wissink, T. L., Kuhn, D. R., & Kacker, R. N. (2015).Introducing Combinatorial Testing in a Large Organization. IEEE Computer, (4), 64-72.
  • DR Kuhn, R Bryce, F Duan, L Sh. Ghandehari, Yu Lei, RN Kacker,Combinatorial Testing, Theory and Practice, Advances in Computers, vol. 99 (to appear, 2015).
  • Kuhn, D. R., Kacker, R. N., Lei, Y., & Torres-Jimenez, J. (2015, April).Equivalence class verification and oracle-free testing using two-layer covering arrays. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.
  • Duan, F., Lei, Y., Yu, L., Kacker, R. N., & Kuhn, D. R. (2015, April).Improving IPOG's vertical growth based on a graph coloring scheme. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-8). IEEE.
  • Yu, L., Duan, F., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2015, April).Constraint handling in combinatorial test generation using forbidden tuples. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-9). IEEE.
  • Ghandehari, L. S., Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2015, April). BEN: A combinatorial testing-based fault localization tool. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.
  • D.R. Kuhn, R.N. Kacker and Y. Lei, Combinatorial Coverage as an Aspect of Test Quality, Mar/Apr 2015 CrossTalk: Journal of Defense Software Engineering.  Article (pdf) Preprint
  • D.R. Kuhn, R.N. Kacker and Y. Lei, Measuring and Specifying Combinatorial Coverage of Test Input Configurations, submitted toInnovations in Systems and Software Engineering: a NASA journal, [2015].Preprint
  • L. S. Ghandehari, J. Czerwonka, Y. Lei, S. Shafiee, R. Kacker and R. Kuhn, An Empirical Comparison of Combinatorial and Random Testing, Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, pp. 68-77.AbstractDOI: 10.1109/ICSTW.2014.8Preprint
  • J. Hagar, R. Kuhn, R. Kacker, and T. Wissink, Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report [poster], Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p. 153.AbstractDOI: 10.1109/ICSTW.2014.70Preprint
  • R. Kuhn, R Kacker and Y. Lei, Estimating Fault Detection Effectiveness [poster], Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p. 154.AbstractDOI: 10.1109/ICSTW.2014.69Preprint
  • J. Torres-Jimenez, I. Izquierdo-Marquez, Survey of Covering Arrays,15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC 2013), Timisoara, Romania, September 23-26, 2013, pp. 20-27.DOI: 10.1109/SYNASC.2013.10
  • R.N. Kacker, D.R. Kuhn, Y. Lei, and J.F. Lawrence, Combinatorial Testing for Software: an Adaptation of Design of Experiments,Measurement, vol. 46, no. 9, November 2013, pp. 3745-3752.DOI: 10.1016/j.measurement.2013.02.021
  • X. Niu, C. Nie, Y. Lei, A.T.S. Chan, Identifying Failure-Inducing Combinations Using Tuple Relationships, 2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 271-280.DOI: 10.1109/ICSTW.2013.38Preprint
  • D.R. Kuhn, I. Dominguez Mendoza, R.N. Kacker and Y. Lei.Combinatorial Coverage Measurement Concepts and Applications,2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 352-361.AbstractDOI: 10.1109/ICSTW.2013.77Preprint
  • L.S.G. Ghandehari, M.N. Borazjany, Y. Lei, R.N. Kacker and D.R. Kuhn,Applying Combinatorial Testing to the Siemens Suite, 2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 362-371.AbstractDOI: 10.1109/ICSTW.2013.47Preprint
  • M.N. Borazjany, L.S.G. Ghandehari, Y. Lei, R.N. Kacker and D.R. Kuhn,An Input Space Modeling Methodology for Combinatorial Testing,2nd International Workshop on Combinatorial Testing (IWCT 2013), in Proceedings of the Sixth IEEE International Conference on Software, Testing, Verification and Validation (ICST 2013), Luxembourg, March 18-22, 2013, pp. 372-381.AbstractDOI: 10.1109/ICSTW.2013.48Preprint
  • D.R. Kuhn and R.N. Kacker, Measuring Combinatorial Coverage of System State-space for IV&V [extended abstract], NASA IV&V Workshop, Morgantown, West Virginia, September 11-13, 2012. AbstractPresentation
  • S. Manchester, R. Bryce, S. Sampath, N. Samant, D. R. Kuhn and R. Kacker, Applying Higher Strength Combinatorial Criteria to Test Prioritization: a Case Study, to appear in: Journal of Combinatorial Mathematics and Combinatorial Computing.Preprint
  • D.R. Kuhn, R.N. Kacker and Y. Lei. Combinatorial Testing,Encyclopedia of Software Engineering, edited by Phillip A. Laplante. Taylor & Francis, 2012.AbstractDOI: 10.1081/E-ESE-120048153Preprint
  • D.R. Kuhn, J.M. Higdon, J.F. Lawrence, R.N. Kacker and Y. Lei,Efficient Methods for Interoperability Testing Using Event Sequences, CrossTalk (Hill AFB): the Journal of Defense Software Engineering, vol. 25, no. 4, July/August 2012, pp. 15-18.AbstractArticle
  • C. Montanez-Rivera, D.R. Kuhn, M. Brady, R.M. Rivello, J. Reyes and M.K. Powers, Evaluation of Fault Detection Effectiveness for Combinatorial and Exhaustive Selection of Discretized Test Inputs,Software Quality Professional, vol. 14, no. 3, June 2012.AbstractArticlePreprint
  • M.N. Borazjany, L. Yu, Y. Lei, R. Kacker and R. Kuhn, Combinatorial Testing of ACTS: A Case Study, First International Workshop on Combinatorial Testing, in Proceedings of the IEEE Fifth International Conference on Software, Testing, Verification and Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21, 2012, pp. 591-600.AbstractDOI: 10.1109/ICST.2012.146
  • K. Shakya, T. Xie, N. Li, Y. Lei, R. Kacker and R. Kuhn, Isolating Failure-Inducing Combinations in Combinatorial Testing using Test Augmentation and Classification, First International Workshop on Combinatorial Testing, in Proceedings of the IEEE Fifth International Conference on Software, Testing, Verification and Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21, 2012, pp. 620-623.AbstractDOI: 10.1109/ICST.2012.149Preprint
  • D.R. Kuhn, J.M. Higdon, J.F. Lawrence, R.N. Kacker and Y. Lei,Combinatorial Methods for Event Sequence Testing, First International Workshop on Combinatorial Testing, in Proceedings of the IEEE Fifth International Conference on Software, Testing, Verification and Validation (ICST 2012), Montreal, Quebec, Canada, April 17-21, 2012, pp. 601-609.AbstractDOI: 10.1109/ICST.2012.147Preprint
  • V.C. Hu, D.R. Kuhn, T. Xie and J. Hwang, Model Checking for Verification of Mandatory Access Control Models and Properties,International Journal of Software Engineering and Knowledge Engineering, vol. 21, no. 1, February 2011, pp. 103-127.AbstractDOI: 10.1142/S021819401100513XPreprint
  • J. Lawrence, R.N. Kacker, Y. Lei, D.R. Kuhn, and M. Forbes, A Survey of Binary Covering Arrays, Electronic Journal of Combinatorics, vol. 18, no. 1, art. P84, April 7, 2011.AbstractArticle
  • D.R. Kuhn, R.N. Kacker and Y. Lei, Advanced Combinatorial Test Methods for System Reliability, 2010 Annual Technical Report of the IEEE Reliability Society, January 2011.Article
  • C. Montanez, D.R. Kuhn, M. Brady, R. Rivello, J. Reyes, M.K. Powers,An Application of Combinatorial Methods to Conformance Testing for Document Object Model Events, NIST Internal Report (NISTIR) 7773, November 2010, 9 pp.AbstractNISTIR 7773
  • R.C. Bryce, C.J. Colbourn and D.R. Kuhn, Finding Interaction Faults Adaptively using Distance-Based Strategies, 18th IEEE International Conference and Workshops on Engineering of Computer Based Systems (ECBS), Las Vegas, Nevada, April 27-29, 2011, pp.4-13.DOI: 10.1109/ECBS.2011.9
  • J.R. Maximoff, M.D. Trela, D.R. Kuhn and R. Kacker, A Method for Analyzing System State-space Coverage within a t-Wise Testing Framework, 4th Annual IEEE International Systems Conference, April 5-8, 2010, San Diego, California, pp. 598-603.DOI: 10.1109/SYSTEMS.2010.5482481
  • D.R. Kuhn, R. Kacker and Y.Lei, Random vs. Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network, MODSIM World 2009, Virginia Beach, Virginia, October 14-16, 2009. In Selected Papers Presented at MODSIM World 2009 Conference and Expo, edited by T.E. Pinelli, NASA/CP-2010-216205, National Aeronautics and Space Administration, pp. 83-88.AbstractPaper
  • D.R. Kuhn,R. Kacker, Y. Lei, Combinatorial and Random Testing Effectiveness for a Grid Computer Simulator
  • R. Kuhn, R. Kacker, Y. Lei and J. Hunter, Combinatorial Software Testing, IEEE Computer, vol. 42, no. 8, August 2009, pp. 94-96.AbstractDOI: 10.1109/MC.2009.253Preprint
  • R.C. Bryce, Y. Lei, D.R. Kuhn and R. Kacker, Combinatorial Testing, Ch. 14 in Handbook of Research on Software Engineering and Productivity Technologies: Implications of Globalization, edited by M. Ramachandran and R. Atem de Carvalho, IGI Global, 2010, pp.196-208.DOI: 10.4018/978-1-60566-731-7.ch014
  • V.C. Hu, D.R. Kuhn and T. Xie, Property Verification for Generic Access Control Models, 2008 IEEE/IFIP International Symposium on Trust, Security, and Privacy for Pervasive Applications (TSP-08) in Volume 2 of Proceedings of the 2008 IEEE/IFIP International Conference on Embedded and Ubiquitous Computing (EUC-08), Shanghai, China, December 17-20, 2008, pp. 234-250.AbstractDOI: 10.1109/EUC.2008.22Preprint
  • M. Ellims, D. Ince and M. Petre, The Effectiveness of T-Way Test Data Generation, 27th International Conference on Computer Safety, Reliability, and Security (SAFECOMP 2008), Newcastle upon Tyne, U.K., September 22-25, 2008. In Computer Safety, Reliability, and Security, Lecture Notes in Computer Science 5219, pp.16–29.DOI: 10.1007/978-3-540-87698-4_5
  • M. Forbes, J. Lawrence, Y. Lei, R.N. Kacker, and D.R. Kuhn, Refining the In-Parameter-Order Strategy for Constructing Covering Arrays,Journal of Research of the National Institute of Standards and Technology, vol. 113, no. 5, September-October 2008, pp. 287-297.AbstractDOI: 10.6028/jres.113.022
  • D.R. Kuhn, R. Kacker and Y. Lei, Automated Combinatorial Test Methods: Beyond Pairwise Testing, CrossTalk (Hill AFB): the Journal of Defense Software Engineering, vol. 21, no. 6, June 2008, pp.22-26.AbstractArticle Comment: A fairly comprehensive tutorial on combinatorial testing and automated test generation, with a worked example.
  • R. Kessel and R. Kacker, A Test of Linearity Using Covering Arrays for Evaluating Uncertainty in Measurement, Advanced Mathematical and Computational Tools in Metrology and Testing (AMCTM VIII), Paris, France, June 23-25, 2008, Series on Advances in Mathematics for Applied Sciences vol. 78, pp.195-203.AbstractDOI: 10.1142/7212Preprint
  • R. Kuhn, Y. Lei and R. Kacker, Practical Combinatorial Testing: Beyond Pairwise Testing, IT Professional, vol. 10, no. 3, May-June 2008, pp.19-23.AbstractDOI: 10.1109/MITP.2008.54 Comment: Quick introduction to combinatorial testing.
  • Y. Lei, R. Kacker, D. Kuhn, V. Okun and J. Lawrence, IPOG/IPOD: Efficient Test Generation for Multi-Way Combinatorial Testing,Software Testing, Verification, and Reliability, vol. 18, no. 3, September 2008, pp. 125-148.AbstractDOI: 10.1002/stvr.381
  • Y. Lei, R.H. Carver, R. Kacker and D. Kung, A Combinatorial Testing Strategy for Concurrent Programs, Software Testing, Verification, and Reliability, vol. 17, no. 4, December 2007, pp. 207-225.AbstractDOI: 10.1002/stvr.369
  • Y.Lei, R. Kacker, D.R. Kuhn, V. Okun and J. Lawrence, IPOG: a General Strategy for T-way Software Testing, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07), Tucson, Arizona, March 26-29 2007, pp. 549-556.AbstractDOI: 10.1109/ECBS.2007.47Preprint
  • R. Bryce and C.J. Colbourn, Prioritized Interaction Testing for Pair-wise Coverage with Seeding and Constraints, Information and Software Technology, vol. 48, no. 10, October 2006, pp. 960-970.DOI: 10.1016/j.infsof.2006.03.004
  • R. Bryce, A. Rajan and M.P.E. Heimdahl, Interaction Testing in Model-Based Development: Effect on Model-Coverage, 13th Asia-Pacific Software Engineering Conference (APSEC 2006), Kanpur, India, December 6-8, 2006, pp. 259-268.DOI: 10.1109/APSEC.2006.42
  • D. R. Kuhn, V. Okun, Pseudo-exhaustive Testing for Software, 30th Annual IEEE/NASA Software Engineering Workshop (SEW-30), Columbia, Maryland, April 24-28, 2006, pp. 153-158.AbstractDOI: 10.1109/SEW.2006.26Preprint Comment: Proof-of-concept experiment on pseudo-exhaustive testing, integrating automated test generation with combinatorial testing.
  • R.C. Bryce, C.J. Colbourn and M.B. Cohen, A Framework of Greedy Methods for Constructing Interaction Test Suites, 27th International Conference on Software Engineering (ICSE 2005), St. Louis, Missouri, May 15-21, 2005, pp. 146-155.DOI: 10.1109/ICSE.2005.1553557
  • M. Grindal, J. Offutt and S.F. Andler, Combination Testing Strategies: a Survey, Software Testing, Verification and Reliability, vol. 15, no. 3, March 2005, pp. 167-199.DOI: 10.1002/stvr.319Preprint Comment: A survey of combinatorial testing methods and results.
  • D.R. Kuhn, D.R. Wallace, A.M. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Transactions on Software Engineering, vol. 30, no. 6, June 2004, pp. 418-421.AbstractDOI: 10.1109/TSE.2004.24 Comment: Investigates interaction level required to trigger faults in a large distributed database system.
  • D.R. Kuhn and M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th Annual NASA Goddard/IEEE Software Engineering Workshop (SEW ’02), Greenbelt, Maryland, December 5-6, 2002, pp. 91-95.AbstractDOI: 10.1109/SEW.2002.1199454PreprintComment: Investigates interaction level required to trigger faults in open source browser and server.
  • D.R. Wallace and D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality and Safety Engineering, vol. 8, no. 4, December 2001, pp.351-371.AbstractDOI: 10.1142/S021853930100058XPreprint Comment: Categorizes failures by their symptoms and faults, including interaction level required to trigger faults in medical device software.

Created May 24, 2016, Updated January 17, 2018