Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Projects Combinatorial Testing ACTS Library

Combinatorial Testing

Papers

Fundamental background papers:

 

Ongoing research:

2023:

  • Khadka, K., Chandrasekaran, J., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2023, April). Synthetic Data Generation Using Combinatorial Testing and Variational Autoencoder. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 228-236). IEEE.
  • Kuhn, D. R., Raunak, M. S., & Kacker, R. N. (2023, April). Ordered t-way combinations for testing state-based systems. In 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 246-254). IEEE.

2022:

2021:

2020:

2019:

2018:

  • Mouha, N., Raunak, M.S., Kuhn, D.R. and Kacker, R., 2018. Finding bugs in cryptographic hash function implementations. IEEE Transactions on Reliability, 67(3), pp.870-884.
  • DR Kuhn, D Yaga, R Kacker, Y Lei, V Hu, Pseudo-Exhaustive Verification of Rule Based Systems, 30th Intl Conf on Software Engineering and Knowledge Engineering, July 2018.
  • Ghandehari, L.S., Lei, Y., Kacker, R., Kuhn, D.R.R., Kung, D. and Xie, T., 2018. A Combinatorial Testing-Based Approach to Fault Localization. IEEE Transactions on Software Engineering (accepted).
  • Kuhn, R., Raunak, M. and Kacker, R., 2018. Can Reducing Faults Prevent Vulnerabilities?. Computer, 51(7), pp.82-85.
  • Kuhn, R., Raunak, M. and Kacker, R., 2018, April. What proportion of vulnerabilities can be attributed to ordinary coding errors?: poster. In Proceedings of the 5th Annual Symposium and Bootcamp on Hot Topics in the Science of Security (p. 30). ACM.
  • Simos, D.E., Kuhn, R., Lei, Y. and Kacker, R., 2018, April. Combinatorial security testing course. Tutorial  5th Annual Symposium and Bootcamp on Hot Topics in the Science of Security (p. 10). ACM.
  • Feng, H., Chandrasekaran, J., Lei, Y., Kacker, R. and Kuhn, D.R., 2018, December. A Method-Level Test Generation Framework for Debugging Big Data Applications. In 2018 IEEE International Conference on Big Data (Big Data) (pp. 221-230). IEEE.
  • Hu, L., Wong, W.E., Kuhn, D.R. and Kacker, R., 2018, September. MCDC-Star: A White-Box Based Automated Test Generation for High MC/DC Coverage. In 2018 5th International Conference on Dependable Systems and Their Applications (DSA) (pp. 102-112). IEEE.

2017:

  • Vilkomir, S., Alluri, A., Kuhn, D.R. and Kacker, R.N., 2017, July. Combinatorial and MC/DC coverage levels of random testing. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 61-68). IEEE.
  • Chandrasekaran, J., Feng, H., Lei, Y., Kuhn, D.R. and Kacker, R., 2017, March. Applying combinatorial testing to data mining algorithms. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 253-261). IEEE.
  • Kuhn, D.R., Raunak, M.S. and Kacker, R., 2017, July. An Analysis of Vulnerability Trends, 2008-2016. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 587-588). IEEE.
  • Li, D., Hu, L., Gao, R., Wong, W.E., Kuhn, D.R. and Kacker, R.N., 2017, July. Improving MC/DC and Fault Detection Strength Using Combinatorial Testing. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 297-303). IEEE.
  • Duan, F., Lei, Y., Yu, L., Kacker, R.N. and Kuhn, D.R., 2017, March. Optimizing IPOG's vertical growth with constraints based on hypergraph coloring. In 2017 IEEE International Conference on software testing, verification and validation workshops (ICSTW) (pp. 181-188). IEEE.
  • Kuhn, R., Raunak, M. and Kacker, R., 2017. It Doesn’t Have to Be Like This: Cybersecurity Vulnerability Trends. IT Professional, 19(6), pp.66-70.
  • Raunak, M.S., Kuhn, D.R. and Kacker, R., 2017, July. Combinatorial testing of full text search in Web applications. In 2017 IEEE International Conference on Software Quality, Reliability and Security Companion (QRS-C) (pp. 100-107). IEEE.
  • Kuhn, D.R., Kacker, R.N. and Lei, Y., 2017, March. A Model for T-Way Fault Profile Evolution during Testing. In 2017 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 162-170). IEEE.
  • Simos, D., Mekesis, S., Kuhn, D.R. and Kacker, R.N., Combinatorial Coverage Measurement of Test Vectors used in Cryptographic Algorithm Validation. poster High Confidence Systems and Software. 

2016:

  • Kuhn, D.R., Kacker, R.N. and Lei, Y., 2016. Measuring and specifying combinatorial coverage of test input configurations. Innovations in systems and software engineering, 12(4), pp.249-261.
  • Ratliff, Z.B., Kuhn, D.R., Kacker, R.N., Lei, Y. and Trivedi, K.S., 2016, October. The relationship between software bug type and number of factors involved in failures. In 2016 IEEE international symposium on software reliability engineering workshops (ISSREW) (pp. 119-124). IEEE.
  • Wang, W., Sampath, S., Lei, Y., Kacker, R., Kuhn, R. and Lawrence, J., 2016. Using combinatorial testing to build navigation graphs for dynamic web applications. Software Testing, Verification and Reliability, 26(4), pp.318-346.
  • Simos, D.E., Kuhn, R., Voyiatzis, A.G. and Kacker, R., 2016. Combinatorial Methods in Security Testing. IEEE Computer, 49(10), pp.80-83.
  • Kuhn, D.R., Hu, V., Ferraiolo, D.F., Kacker, R.N. and Lei, Y., 2016, April. Pseudo-exhaustive testing of attribute based access control rules. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 51-58). IEEE.
  • Chandrasekaran, J., Ghandehari, L.S., Lei, Y., Kacker, R. and Kuhn, D.R., 2016, April. Evaluating the effectiveness of BEN in localizing different types of software fault. In 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW) (pp. 26-34). IEEE.
  • Kuhn, D.R., Kacker, R.N. and Lei, Y., 2016, June. Estimating t-way fault profile evolution during testing. In 2016 IEEE 40th Annual Computer Software and Applications Conference (COMPSAC) (Vol. 2, pp. 596-597). IEEE.
  • Simos, D.E., Kleine, K., Voyiatzis, A.G., Kuhn, R. and Kacker, R., 2016, August. TLS cipher suites recommendations: A combinatorial coverage measurement approach. In 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS) (pp. 69-73). IEEE.
  • Kuhn, R., Kacker, R., Feldman, L. and Witte, G., 2016. Combinatorial Testing for Cybersecurity and Reliability (No. ITL Bulletin May 2016). National Institute of Standards and Technology.
  • Simos, D.E., Kleine, K., Kuhn, R. and Kacker, R., 2016, May. Combinatorial Coverage Analysis of Subsets of the TLS Cipher Suite Registry. In High Confidence Software and Systems Conference; May 10-12, 2016; Annapolis, Maryland, United States.
  • Kuhn, D.R. and Kacker, R.N., National Institute of Standards and Technology (NIST), 2016. Oracle-free match testing of a program using covering arrays and equivalence classes. U.S. Patent Application 15/019,448.

2015:

  • Kuhn, D.R., Hu, V., Ferraiolo, D., Kacker, R., Lei, Y., Pseudo-exhaustive Testing of Attribute Based Access Control Rules, (Aug. 2015) Preprint, Intl Workshop on Combinatorial Testing.
  • Hagar, J. D., Wissink, T. L., Kuhn, D. R., & Kacker, R. N. (2015). Introducing Combinatorial Testing in a Large Organization. IEEE Computer, (4), 64-72.
  • Kuhn D. R., R Bryce, F Duan, L Sh. Ghandehari, Yu Lei, RN Kacker, Combinatorial Testing, Theory and Practice, Advances in Computers, vol. 99 (to appear, 2015).
  • Kuhn, D. R., Kacker, R. N., Lei, Y., & Torres-Jimenez, J. (2015, April). Equivalence Class Verification and Oracle-free Testing Using Two-layer Covering Arrays. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.
  • Duan, F., Lei, Y., Yu, L., Kacker, R. N., & Kuhn, D. R. (2015, April). Improving IPOG's Vertical Growth Based on a Graph Coloring Scheme. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-8). IEEE.
  • Yu, L., Duan, F., Lei, Y., Kacker, R. N., & Kuhn, D. R. (2015, April). Constraint Handling in Combinatorial Test Generation Using Forbidden Tuples. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-9). IEEE.
  • Ghandehari, L. S., Chandrasekaran, J., Lei, Y., Kacker, R., & Kuhn, D. R. (2015, April). BEN: A Combinatorial Testing-based Fault Localization Tool. In Software Testing, Verification and Validation Workshops (ICSTW), 2015 IEEE Eighth International Conference on (pp. 1-4). IEEE.
  • D.R. Kuhn, R.N. Kacker and Y. Lei, Combinatorial Coverage as an Aspect of Test Quality, Mar/Apr 2015 CrossTalk: Journal of Defense Software Engineering. 
  • D.R. Kuhn, R.N. Kacker and Y. Lei, Measuring and Specifying Combinatorial Coverage of Test Input Configurations, submitted to Innovations in Systems and Software Engineering: a NASA journal, [2015]. 

2014:

  • L. S. Ghandehari, J. Czerwonka, Y. Lei, S. Shafiee, R. Kacker and R. Kuhn, An Empirical Comparison of Combinatorial and Random Testing, Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, pp. 68-77. Preprint
  • J. Hagar, R. Kuhn, R. Kacker, and T. Wissink, Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report [poster], Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p. 153. Preprint
  • R. Kuhn, R Kacker and Y. Lei, Estimating Fault Detection Effectiveness [poster], Third International Workshop on Combinatorial Testing (IWCT 2014), in Proceedings of the Seventh IEEE International Conference on Software, Testing, Verification and Validation (ICST 2014), Cleveland, Ohio, March 31-April 4, 2014, p. 154. Preprint

2013:

2012:

2011:

2001-2010:

  • C. Montanez, D.R. Kuhn, M. Brady, R. Rivello, J. Reyes, M.K. Powers, An Application of Combinatorial Methods to Conformance Testing for Document Object Model Events, NIST Internal Report (NISTIR) 7773, November 2010, 9 pp.
  • J.R. Maximoff, M.D. Trela, D.R. Kuhn and R. Kacker, A Method for Analyzing System State-space Coverage within a t-Wise Testing Framework, 4th Annual IEEE International Systems Conference, April 5-8, 2010, San Diego, California, pp. 598-603.DOI: 10.1109/SYSTEMS.2010.5482481
  • D.R. Kuhn, R. Kacker and Y.Lei, Random vs. Combinatorial Methods for Discrete Event Simulation of a Grid Computer Network, MODSIM World 2009, Virginia Beach, Virginia, October 14-16, 2009. In Selected Papers Presented at MODSIM World 2009 Conference and Expo, edited by T.E. Pinelli, NASA/CP-2010-216205, National Aeronautics and Space Administration, pp. 83-88.
  • D.R. Kuhn,R. Kacker, Y. Lei, Combinatorial and Random Testing Effectiveness for a Grid Computer Simulator
  • R. Kuhn, R. Kacker, Y. Lei and J. Hunter, Combinatorial Software Testing, IEEE Computer, vol. 42, no. 8, August 2009, pp. 94-96. Preprint
  • R.C. Bryce, Y. Lei, D.R. Kuhn and R. Kacker, Combinatorial Testing, Ch. 14 in Handbook of Research on Software Engineering and Productivity Technologies: Implications of Globalization, edited by M. Ramachandran and R. Atem de Carvalho, IGI Global, 2010, pp.196-208.DOI: 10.4018/978-1-60566-731-7.ch014
  • V.C. Hu, D.R. Kuhn and T. Xie, Property Verification for Generic Access Control Models, 2008 IEEE/IFIP International Symposium on Trust, Security, and Privacy for Pervasive Applications (TSP-08) in Volume 2 of Proceedings of the 2008 IEEE/IFIP International Conference on Embedded and Ubiquitous Computing (EUC-08), Shanghai, China, December 17-20, 2008, pp. 234-250.Abstract; DOI: 10.1109/EUC.2008.22; Preprint
  • M. Ellims, D. Ince and M. Petre, The Effectiveness of T-Way Test Data Generation, 27th International Conference on Computer Safety, Reliability, and Security (SAFECOMP 2008), Newcastle upon Tyne, U.K., September 22-25, 2008. In Computer Safety, Reliability, and Security, Lecture Notes in Computer Science 5219, pp.16–29.DOI: 10.1007/978-3-540-87698-4_5
  • M. Forbes, J. Lawrence, Y. Lei, R.N. Kacker, and D.R. Kuhn, Refining the In-Parameter-Order Strategy for Constructing Covering Arrays,Journal of Research of the National Institute of Standards and Technology, vol. 113, no. 5, September-October 2008, pp. 287-297.Abstract; DOI: 10.6028/jres.113.022
  • D.R. Kuhn, R. Kacker and Y. Lei, Automated Combinatorial Test Methods: Beyond Pairwise Testing, CrossTalk (Hill AFB): the Journal of Defense Software Engineering, vol. 21, no. 6, June 2008, pp.22-26.Abstract; Article Comment: A fairly comprehensive tutorial on combinatorial testing and automated test generation, with a worked example.
  • R. Kessel and R. Kacker, A Test of Linearity Using Covering Arrays for Evaluating Uncertainty in Measurement, Advanced Mathematical and Computational Tools in Metrology and Testing (AMCTM VIII), Paris, France, June 23-25, 2008, Series on Advances in Mathematics for Applied Sciences vol. 78, pp.195-203.Abstract; DOI: 10.1142/7212; Preprint
  • R. Kuhn, Y. Lei and R. Kacker, Practical Combinatorial Testing: Beyond Pairwise Testing, IT Professional, vol. 10, no. 3, May-June 2008, pp.19-23.Abstract; DOI: 10.1109/MITP.2008.54 Comment: Quick introduction to combinatorial testing.
  • Y. Lei, R. Kacker, D. Kuhn, V. Okun and J. Lawrence, IPOG/IPOD: Efficient Test Generation for Multi-Way Combinatorial Testing,Software Testing, Verification, and Reliability, vol. 18, no. 3, September 2008, pp. 125-148.Abstract; DOI: 10.1002/stvr.381
  • Y. Lei, R.H. Carver, R. Kacker and D. Kung, A Combinatorial Testing Strategy for Concurrent Programs, Software Testing, Verification, and Reliability, vol. 17, no. 4, December 2007, pp. 207-225.Abstract; DOI: 10.1002/stvr.369
  • Y.Lei, R. Kacker, D.R. Kuhn, V. Okun and J. Lawrence, IPOG: a General Strategy for T-way Software Testing, 14th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems (ECBS’07), Tucson, Arizona, March 26-29 2007, pp. 549-556.Abstract; DOI: 10.1109/ECBS.2007.47; Preprint
  • R. Bryce and C.J. Colbourn, Prioritized Interaction Testing for Pair-wise Coverage with Seeding and Constraints, Information and Software Technology, vol. 48, no. 10, October 2006, pp. 960-970.DOI: 10.1016/j.infsof.2006.03.004
  • R. Bryce, A. Rajan and M.P.E. Heimdahl, Interaction Testing in Model-Based Development: Effect on Model-Coverage, 13th Asia-Pacific Software Engineering Conference (APSEC 2006), Kanpur, India, December 6-8, 2006, pp. 259-268.DOI: 10.1109/APSEC.2006.42
  • D. R. Kuhn, V. Okun, Pseudo-exhaustive Testing for Software, 30th Annual IEEE/NASA Software Engineering Workshop (SEW-30), Columbia, Maryland, April 24-28, 2006, pp. 153-158.Abstract; DOI: 10.1109/SEW.2006.26; Preprint Comment: Proof-of-concept experiment on pseudo-exhaustive testing, integrating automated test generation with combinatorial testing.
  • R.C. Bryce, C.J. Colbourn and M.B. Cohen, A Framework of Greedy Methods for Constructing Interaction Test Suites, 27th International Conference on Software Engineering (ICSE 2005), St. Louis, Missouri, May 15-21, 2005, pp. 146-155.DOI: 10.1109/ICSE.2005.1553557
  • M. Grindal, J. Offutt and S.F. Andler, Combination Testing Strategies: a Survey, Software Testing, Verification and Reliability, vol. 15, no. 3, March 2005, pp. 167-199.DOI: 10.1002/stvr.319; Preprint Comment: A survey of combinatorial testing methods and results.
  • D.R. Kuhn, D.R. Wallace, A.M. Gallo, Jr., Software Fault Interactions and Implications for Software Testing, IEEE Transactions on Software Engineering, vol. 30, no. 6, June 2004, pp. 418-421.Abstract; DOI: 10.1109/TSE.2004.24 Comment: Investigates interaction level required to trigger faults in a large distributed database system.
  • D.R. Kuhn and M.J. Reilly, An Investigation of the Applicability of Design of Experiments to Software Testing, 27th Annual NASA Goddard/IEEE Software Engineering Workshop (SEW ’02), Greenbelt, Maryland, December 5-6, 2002, pp. 91-95.Abstract; DOI: 10.1109/SEW.2002.1199454; PreprintComment: Investigates interaction level required to trigger faults in open source browser and server.
  • D.R. Wallace and D.R. Kuhn, Failure Modes in Medical Device Software: an Analysis of 15 Years of Recall Data, International Journal of Reliability, Quality and Safety Engineering, vol. 8, no. 4, December 2001, pp.351-371.Abstract; DOI: 10.1142/S021853930100058X; Preprint Comment: Categorizes failures by their symptoms and faults, including interaction level required to trigger faults in medical device software.

Contacts

Rick Kuhn
kuhn@nist.gov
Address: https://www.nist.gov/people/d-richard-kuhn

Raghu Kacker
raghu.kacker@nist.gov
301-975-2109
Address: http://math.nist.gov/~RKacker/

M S Raunak
raunak@nist.gov

Topics

Security and Privacy: assurance, modeling, testing & validation

Technologies: software & firmware

Created May 24, 2016, Updated March 08, 2024