Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Combinatorial Methods for Trust and Assurance

Workshop Papers

In 2012, we co-founded the International Workshop on Combinatorial Testing, focused on theory and application of CT.  Papers from previous workshops are listed below. 

IWCT 2023

  • Applying CT-FLA for AEB Function Testing: A Virtual Driving Case Study
    Ludwig Kampel, Michael Wagner, Dimitris Simos, Mihai Nica, Dino Dodig, David Kaufmann, Franz Wotawa
  • Combinatorial Methods for HTML Sanitizer Security Testing
    Jovan Zivanovic, Manuel Leithner, Dimitris Simos, Michael Pitzer, Peter J. Slanina
  • Hints in Unified Combinatorial Interaction Testing
    Cemal Yilmaz, Hanefi Mercan
  • Incremental generation of combinatorial test suites starting from existing seed tests
    Andrea Bombarda, Angelo Gargantini
  • Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error
    Tyler Cody, Laura Freeman
  • Ordered t-way Combinations for Testing State-based Systems
    Rick Kuhn, M S Raunak, Raghu Kacker
  • Summary of In-Parameter-Order strategies for covering perfect hash families
    Michael Wagner, Charles Colbourn, Dimitris Simos
  • Summary of Locating Hardware Trojans using Combinatorial Testing for Cryptographic Circuits
    Ludwig Kampel, Paris Kitsos, Dimitris Simos
  • Synthetic Data Generation Using Combinatorial Testing and Variational Autoencoder
    Krishna Khadka, Jaganmohan Chandrasekaran, Jeff Yu Lei, Raghu Kacker, Richard Kuhn

IWCT 2022

  • A C++ implementation of the IPO algorithm
    Ken'Ya Takemura
  • A Combinatorial Approach to Fairness Testing of Machine Learning Models
    Ankita Ramjibhai Patel, Jaganmohan Chandrasekaran, Jeff Yu Lei, Raghu Kacker, Rick Kuhn
  • A Constrained Covering Array Generator using Adaptive Penalty based Parallel Tabu Search
    Yan Wang, Huayao Wu, Changhai Nie, Xintao Niu, Jiaxi Xu
  • Applying Combinatorial Testing to High-Speed Railway Automatic Train Protection System
    Rui Huang, Chang Rao, Jeff Yu Lei, Jin Guo, Yadong Zhang
  • CA^2: Practical Archival and Compression of Covering Arrays
    Manuel Leithner, Dimitris Simos
  • Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions
    Rick Kuhn, M S Raunak, Charles Prado, Vinay Patil, Raghu Kacker
  • Developing multithreaded techniques and improved constraint handling for the tool CAgen
    Michael Wagner, Manuel Leithner, Dimitris Simos, Rick Kuhn, Raghu Kacker
  • Experience of Combinatorial Testing toward Fault Detection, Isolation and Recovery Functionality
    Naoko Okubo, Shoma Takatsuki, Yasushi Ueda
  • Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams
    Andrea Bombarda, Angelo Gargantini
  • Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing
    Tyler Cody, Erin Lanus, Daniel D. Doyle, Laura Freeman
  • Using Unified Combinatorial Interaction Testing for MC/DC Coverage
    Giray Coskun, Cankut Coskun, Hanefi Mercan, Cemal Yilmaz

IWCT 2021

  • A Combinatorial Approach to Explaining Image Classifiers
    Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn.
  • A Practical Method for API Testing in the Context of Continuous Delivery and Behavior Driven Development  
    Brian Elgaard Bennett.
  • An Environment for Benchmarking Combinatorial Test Suite Generators
    Andrea Bombarda, Edoardo Crippa and Angelo Gargantini.
  • A Combinatorial Approach to Testing Deep Neural Network-based Autonomous Driving Systems
    Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn.
  • SYSMODIS: A Systematic Model Discovery Approach
    Omer Korkmaz and Cemal Yilmaz.
  • Computing Sequence Covering Arrays using Unified Combinatorial Interaction Testing
    Hanefi Mercan and Cemal Yilmaz.
  • Combinatorial Testing Metrics for Machine Learning
    Erin Lanus, Laura J. Freeman, D. Richard Kuhn and Raghu N. Kacker.
  • Combinatorially XSSing Web Application Firewalls
    Bernhard Garn, Daniel Sebastian Lang, Manuel Leithner, Rick Kuhn, Raghu Kacker and Dimitris E. Simos.

IWCT 2020

  • Toward an Encoding Approach to Interaction-based Test Suite Minimization
    Kohei Nishino, Takashi Kitamura, Tomoji Kishi, Cyrille Artho
  • An Automata-Based Generation Method for Combinatorial Sequence Testing of Finite State Machines
    Andrea Bombarda, Angelo Gargantini
  • Combinatorial Methods for Explainable AI
    Rick Kuhn, Raghu Kacker, Jeff Lei, Dimitris Simos
  • Generation of Invalid Test Inputs from Over-Constrained Test Models for Combinatorial Robustness Testing
    Konrad Fögen, Horst Lichter
  • On Using Ontologies for Testing Compilers
    Yihao Li, Franz Wotawa
  • A Comparison Infrastructure for Fault Characterization Algorithms
    Torben Friedrichs, Konrad Fögen, Horst Lichter
  • Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application
    Jan Richter, Bestoun S. Ahmed, Miroslav Bures, Cleber R. Rosa Junior
  • CAGEN: A fast combinatorial test generation tool with support for constraints and higher-index arrays
    Michael Wagner, Kristoffer Kleine, Dimitris Simos, Rick Kuhn, Raghu Kacker


IWCT 2019

  • A Framework for Automated Combinatorial Test Generation
    Joshua Bonn, Konrad Fögen and Horst Lichter
  • On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions
    Jianbo Tao, Yihao Li, Franz Wotawa, Hermann Felbinger and Mihai Nica
  • An Approach to T-way Test Sequence Generation With Constraints
    Feng Duan, Yu Lei, Raghu Kacker and Rick Kuhn
  • Efficient and Guaranteed Detection of t-way Failure-inducing Combinations
    Paolo Arcaini, Angelo Gargantini and Marco Radavelli
  • Using Parameter Mapping to Avoid Forbidden Tuples in a Covering Array
    Chang Rao, Nan Li, Yu Lei, Raghu Kacker, Richard Kuhn and Jin Guo
  • Partitioned Search with Column Resampling for Locating Array Construction
    Erin Lanus, Charles Colbourn and Douglas Montgomery
  • Measuring Combinatorial Coverage at Adobe
    Riley Smith, Darryl Jarman, Richard Kuhn, Raghu Kacker, Dimitris Simos and Jared Bellows
  • On Measuring Combinatorial Coverage of Manually Created Test Cases for Industrial Software  
    Miraldi Fifo, Eduard Paul Enoiu and Wasif Afzal
  • Reduce Test Cost by Reusing Test Oracles through Combinatorial Join 
    Hiroshi Ukai, Xiao Qu, Hironori Washizaki and Yoshiaki Fukazawa
  • Applying Combinatorial Testing to Large-scale Data Processing at Adobe
    Riley Smith, Darryl Jarman, Richard Kuhn, Raghu Kacker, Dimitris Simos, Ludwig Kampel, Manuel Leithner and Gabe Gosney
  • Practical Fault Localization with Combinatorial Test Design
    Dale Blue, Andrew Hicks, Rachel Tzoref-Brill and Ryan Rawlins
  • An Industrial Study on Applications of Combinatorial Testing in Modern Web Development and Cloud Computing   
    Murat Ozcan
  • Crucial Tool Features for Successful Combinatorial Input Parameter Testing in an Industrial Application   
    Christian Eitner and Franz Wotawa
  • Scenario Sampling for Cyber Physical Systems using Combinatorial Testing 
    Akihisa Yamada, Clovis Eberhart, Fuyuki Ishikawa and Nian-Ze Lee


IWCT 2018 

  • A Satisfiability-Based Approach to Generation of Constrained Locating Arrays
    Hao Jin, Takashi Kitamura, Eun-Hye Choi and Tatsuhiro Tsuchiya
  • Optimal Scheduling for Combinatorial Software Testing and Design of Experiments
    Robert V. Binder
  • Enumerator: An Efficient Approach for Enumerating all Valid t-tuples (short 20 mins)
    Hanefi Mercan, Kamer Kaya and Cemal Yilmaz
  • Combinatorial Test Design – smarter way to connect with the business for quality coverage
    Saritha Route and Sudheer Pendela
  • Bringing combinatorial interaction testing modeling and generation to the web
    Angelo Gargantini and Marco Radavelli
  • CAmetrics: A Tool for Advanced Combinatorial Analysis and Measurement of Test Sets
    Manuel Leithner, Kristoffer Kleine and Dimitris E. Simos
  • Combinatorial Testing with Constraints for Negative Test Cases (short 20 mins)
    Konrad Fögen and Horst Lichter
  • Practical Amplification of Condition/Decision Test Coverage by Combinatorial Testing
    Artur Andrzejak and Thomas Bach
  • The effect of time-between-events for sequence interaction testing of a real-time system
    Chek Pin Yang, Gunwant Dhadyalla, James Marco and Paul Jennings
  • Combining Combinatorial Testing and Metamorphic Testing for Testing a Logic-based Non-monotonic Reasoning System (short 20 mins)
    Franz Wotawa
  • Adapting Unit Tests by Generating Combinatorial Test (short 20 mins)
    Hermann Felbinger, Franz Wotawa and Mihai Nica

IWCT 2017 

  • Estimating t-way Fault Profile Evolution During Testing
    D. Richard Kuhn, Raghu N. Kacker and Jeff Lei[slides]
  • Mutation Score, Coverage, Model Inference: Quality Assessment For t-way Combinatorial Test-Suites
    Hermann Felbinger, Franz Wotawa and Mihai Nica[slides]
  • Optimizing IPOG’s Vertical Growth with Constraints Based on Hypergraph Coloring
    Feng Duan, Yu Lei, Linbin Yu, Raghu N. Kacker and D. Richard Kuhn  [slides]
  • Test Case Generation with Regular Expressions and Combinatorial Techniques
    Macario Polo, Francisco Ruiz Romero, Rosana Rodríguez-Bobada Aranda and Ignacio García-Rodríguez
  • Applying Combinatorial Testing to High-Speed Railway Track Circuit Receiver
    Chang Rao, Jin Guo, Nan Li, Jeff Lei, Yadong Zhang, Yao Li and Yaxin Cao  [slides]
  • Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc.
    Murat Ozcan  [slides]
  • Using Timed Base-Choice Coverage Criterion for Testing Industrial Control Software Bergstrom and Eduard Enoiu  [slides]
  • Finding Minimum Locating Arrays Using a SAT Solver
    Tatsuya Konishi, Hideharu Kojima, Hiroyuki Nakagawa and Tatsuhiro Tsuchiya  [slides]
  • Test Optimisation using Combinatorial Test Design
    Saritha Route  [slides]
  • Building Combinatorial Test Input Model from Use Case Artefacts
    Preeti S, Milind B, Medhini S. Narayan and Krishnan Rangarajan  [slides]
  • Combinatorial Methods for Modelling Composed Software Systems
    Ludwig Kampel, Bernhard Garn and Dimitris E. Simos  [slides]
  • Combinatorial Interaction Testing for Automated Constraint Repair
    Angelo Gargantini, Justyna Petke and Marco Radavelli  [slides]
  • A composition-based method for combinatorial test design (short 15 mins)
    Anna Zamansky, Eitan Farchi, Sery Khoury and Amir Schwartz  [slides]
  • Applying Combinatorial Testing to Data Mining Algorithms
    Jaganmohan Chandrasekaran, Huadong Feng, Yu Lei, D. Richard Kuhn and Raghu N. Kacker  [slides]
  • Combinatorial Testing on Implementations of HTML5 Support
    Xi Deng, Tianyong Wu, Jun Yan and Jian Zhang  [slides]
  • Combinatorial Testing on MP3 for Audio Players (short 15 mins)
    Shaojiang Wang, Tianyong Wu, Yuan Yao, Beihong Jin and Liping Ding   [slides]


IWCT 2016 

  • Evaluating the Effectiveness of BEN in Locating Different Types of Software Fault
    Jaganmohan Chandrasekaran, Laleh Sh. Ghandehari, Yu Lei, Raghu Kacker and Richard Kuhn [slides]
  • Analyzing Minimal Failure-Causing Schemas in Siemens Suite 
    Chiya Xu, Yuanchao Qi, Ziyuan Wang and Weifeng Zhang
  • Should We Care about “Don’t Care” Testing Inputs? Empirical Investigation of Pair-Wise TestingSergiy Vilkomir and Galen Pennell  [slides]
  • Towards Automatic Cost Model Discovery for Combinatorial Interaction Testing
    Gulsen Demiroz and Cemal Yilmaz  [slides]
  • Combinatorial Testing Implementations in Solutions Testing
    Anwar Sherif
  • Embedded Functions for Constraints and Variable Strength in Combinatorial Testing
    George Sherwood  [slides]
  • Test Oracles and Test Script Generation in Combinatorial Testing
    Peter M. Kruse  [slides]
  • Pseudo-exhaustive Testing of Attribute Based Access Control Rules
    Rick Kuhn, Vincent Hu, David Ferraiolo, Raghu Kacker and Yu Lei
  • Applying a pairwise coverage criteria to scenario-based testing 
    Lydie Du Bousquet, Mickaël Delahaye and Catherine Oriat [slides]
  • Pairwise Coverage-based Testing with Selected Elements in a Query for Database Applications
    Koji Tsumura, Hironori Washizaki, Yoshiaki Fukazawa, Keishi Oshima and Ryota Mibe


IWCT 2015

  • Embedded Functions in Combinatorial Test Designs 
    George Sherwood. 
  • Introducing Combinatorial Testing in the Organization a Report on a First Attempt
    Douglas Hillmer. 
  • Combining Minimization and Generation for Combinatorial Testing
    Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick. 
  • A First Systematic Mapping Study on Combinatorial Interaction Testing for Software Product Lines
    Roberto Erick Lopez-Herrejon, Stefan Fischer, Rudolf Ramler and Alexander Egyed. 
  • Evaluation of the IPO-Family Algorithms for Test Case Generation in Web Security Testing
    Josip Bozic, Bernhard Garn, Dimitris. E. Simos and Franz Wotawa. 
  • A process for non-functional combinatorial testing (short)
    Siegmar Züfle and Venkataraman Krishnamoorthy. 
  • Combinatorial Interaction Testing of Tangled Configuration Options (short)
    Arsalan Javeed and Cemal Yilmaz. 
  • Relationship between Pair-wise and MC/DC Testing: Initial Experimental Results (short)
    Sergiy Vilkomir and David Anderson. 
  • Why Combinatorial Testing Works: Analyzing Minimal Failure-Causing Schemas in Logic Expressions (short)
    Ziyuan Wang and Yuanchao Qi
  • Constraint Handling In Combinatorial Test Generation Using Forbidden Tuples
    Linbin Yu, Feng Duan, Yu Lei, Raghu N. Kacker and D. Richard Kuhn. 
  • Using Decision Trees to aid Algorithm Selection in Combinatorial Interaction Test Generation
    Angelo Gargantini and Paolo Vavassori. 
  • Flattening or Not of the Combinatorial Interaction Testing Models?(short)
    Christopher Henard, Mike Papadakis and Yves Le Traon. 
  • Improving IPOG’s Vertical Growth Based on a Graph Coloring Scheme
    Feng Duan, Linbin Yu, Yu Lei, Raghu N. Kacker and D. Richard Kuhn. 
  • Equivalence Class Verification and Oracle-free Testing Using Two-layer Covering Arrays (short)
    Rick Kuhn, Raghu Kacker, Yu Lei and Jose Torres-Jimenez. 
  • BEN: A Combinatorial Testing-Based Fault Localization Tool (short) 
    Laleh Shikh Gholamhosseinghandehari, Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn. 


IWCT 2014 

  • Combinatorial Testing for an Automotive Hybrid Electric Vehicle Control System - A Case Study, Gunwant Dhadyalla, Neelu Kumari and Timothy Snell
  • Eris: A Tool for Combinatorial Testing of the Linux System Call Interface,
    Bernhard Garn and Dimitris. E. Simos
  • An Empirical Comparison of Combinatorial and Random Testing,
    Laleh Shikh Gholamhosseingghandehari, Jacek Czerwonka, Yu Lei, Soheil Shafiee, Raghu Kacher and Richard Kuhn
  • Using Combinatorial Approaches for Testing Mobile Applications,
    Sergiy Vilkomir and Brandi Amstutz
  • Combinatorial interaction testing of a Java Card static verifier (short paper),
    Andrea Calvagna, Andrea Fornaia and Emiliano Tramontana
  • Extracting the Combinatorial Test Parameters and Values from UML Sequence Diagrams,
    Preeti Satish, Arinjita Paul and Krishnan Rangarajan
  • Validation of Models and Tests for Constrained Combinatorial Interaction Testing,
    Paolo Arcaini, Angelo Gargantini and Paolo Vavassori
  • Functional Dependence and Equivalence Class Factors in Combinatorial Test Designs,
    George Sherwood
  • Combinatorial Testing with Order Requirements,
    Eitan Farchi, Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick
  • Test Suite Reduction by Combinatorial-based Coverage of Event Sequences (short paper),
    Quentin Mayo, Renee Bryce and Ryan Michaels
  • Test Suite Prioritization by Switching Cost,
    Huayao Wu, Changhai Nie and Fei-Ching Kuo
  • Refining the Randomized Post-optimization Strategy for Covering Arrays,
    Xiaohua Li, Zhao Dong, Huayao Wu, Changhai Nie and Kai-Yuan Cai
  • Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report,
    Jon Hagar, Rick Kuhn, Raghu Kacker and Tom Wissink
  • Estimating Fault Detection Effectiveness,
    Rick Kuhn, Raghu Kacker and Yu Lei


IWCT 2013 

  • Empirically Identifying the Best Greedy Algorithm for Covering Array Generation,
    Changhai Nie, Jing Jiang, Huayao Wu, Hareton Leung and Charles J. Colbourn
  • Evaluation of t-wise Approach for Testing Logical Expressions in Software,
    Sergiy Vilkomir, Oleksii Starov and Ranjan Bhambroo
  • On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs,
    Jacek Czerwonka
  • Cascade: A Test Generation Tool For Combinatorial Testing,
    Yong Zhao, Zhiqiang Zhang, Jun Yan and Jian Zhang
  • Identifying Failure-Inducing Combinations Using Tuple Relationship,
    Xintao Niu, Changhai Nie, Lei Yu and Alvin Chan
  • Better Bounds for Event Sequencing Testing,
    Oded Margalit
  • A Study in Prioritization for Higher Strength Combinatorial Testing,
    Xiao Qu and Myra Cohen
  • Combinatorial Test Architecture Design Using Viewpoint diagram,
    Yasuharu Nishi, Tetsuro Katayama and Satomi Yoshizawa
  • Experimenting with Category Partition's 1-way and 2-way test selection criteria,
    Yvan Labiche and Faezeh Sadeghi
  • Using Projections to Debug Large Combinatorial Models,
    Eitan Farchi, Itai Segall and Rachel Tzoref-Brill
  • Combinatorial Interaction Testing with Multi-Perspective Feature Models,
    Sachin Patel, Priya Gupta and Vipul Shah
  • Deriving Combinatorial Test Design Model from UML Activity Diagram,
    Preeti Satish, Sheeba K and Krishnan Rangarajan
  • Combinatorial Testing for Feature Models Using CitLab,
    Andrea Calvagna, Angelo Gargantini and Paolo Vavassori
  • Applications of Combinatorial Testing methods for Breakthrough Results in Software Testing,
    Manish Mehta and Roji Philip
  • Combinatorial Coverage Measurement Concepts and Applications,
    Rick Kuhn, Itzel Dominguez, Raghu Kacker and Yu Lei
  • Applying Combinatorial Testing to the Siemens Suite,
    Laleh Shikh Gholamhossein Ghandehari, Mehra N. Bourazjany, Yu Lei, Raghu Kacker and Richard Kuhn
  • An Input Space Modeling Methodology for Combinatorial Testing,
    Mehra Nouroz Borazjany, Laleh Shikh Gholamhossein Ghandehari, Yu Lei, Raghu Kacker and Rick Kuhn


IWCT 2012

  • Effectiveness of Pair-wise Testing for Software with Boolean Inputs  
    William Alton Ballance, Sergiy Vilkomir, William Jenkins
  • Combinatorial Testing of ACTS: A Case Study  
    Mehra N.Borazjany, Linbin Yu, Yu Lei, Raghu Kacker, Rick Kuhn
  • A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment 
    Atlee M. Cunningham, Jr., Jon Hagar, Ryan J. Holman
  • CITLAB: a Laboratory for Combinatorial Interaction Testing 
    Angelo Gargantini, Paolo Vavassori
  • Identifying Failure-Inducing Combinations in a Combinatorial Test Set  
    Laleh Shikh Gholamhossein Ghandehari, Yu Lei, Tao Xie, Richard Kuhn, Raghu Kacker
  • Combinatorial Methods for Event Sequence Testing  
    D. Richard Kuhn, James M. Higdon, James F. Lawrence, Raghu N. Kacker, Yu Lei
  • Combinatorial Test Design in the TOSCA Testsuite: Lessons Learned and Practical Implications 
    Rudolf Ramler, Theodorich Kopetzky
  • Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing 
    Elke Salecker, Sabine Glesner
  • Common Patterns in Combinatorial Models 
    Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
  • Simplified Modeling of Combinatorial Test Spaces 
    Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
  • Isolating Failure-Inducing Combinations in Combinatorial Testing using Test Augmentation and Classification 
    Kiran Shakya, Tao Xie, Nuo Li, Yu Lei, Raghu Kacker, Richard Kuhn
  • Combinatorial Testing on ID3v2 Tags of MP3 Files 
    Zhiqiang Zhang, Xiaojian Liu, Jian Zhang

Created May 24, 2016, Updated April 29, 2024