Combinatorial Testing

Workshop Papers

In 2012, we co-founded the International Workshop on Combinatorial Testing, focused on theory and application of CT.  Papers from previous workshops are listed below. 
 

IWCT 2023

• Applying CT-FLA for AEB Function Testing: A Virtual Driving Case Study
  Ludwig Kampel, Michael Wagner, Dimitris Simos, Mihai Nica, Dino Dodig, David Kaufmann, Franz Wotawa
• Combinatorial Methods for HTML Sanitizer Security Testing
  Jovan Zivanovic, Manuel Leithner, Dimitris Simos, Michael Pitzer, Peter J. Slanina
• Hints in Unified Combinatorial Interaction Testing
  Cemal Yilmaz, Hanefi Mercan
• Incremental generation of combinatorial test suites starting from existing seed tests
  Andrea Bombarda, Angelo Gargantini
• Metric Learning Improves the Ability of Combinatorial Coverage Metrics to Anticipate Classification Error
  Tyler Cody, Laura Freeman
• Ordered t-way Combinations for Testing State-based Systems
  Rick Kuhn, M S Raunak, Raghu Kacker
• Summary of In-Parameter-Order strategies for covering perfect hash families
  Michael Wagner, Charles Colbourn, Dimitris Simos
• Summary of Locating Hardware Trojans using Combinatorial Testing for Cryptographic Circuits
  Ludwig Kampel, Paris Kitsos, Dimitris Simos
• Synthetic Data Generation Using Combinatorial Testing and Variational Autoencoder
  Krishna Khadka, Jaganmohan Chandrasekaran, Jeff Yu Lei, Raghu Kacker, Richard Kuhn

IWCT 2022

• A C++ implementation of the IPO algorithm
  Ken'Ya Takemura
• A Combinatorial Approach to Fairness Testing of Machine Learning Models
  Ankita Ramjibhai Patel, Jaganmohan Chandrasekaran, Jeff Yu Lei, Raghu Kacker, Rick Kuhn
• A Constrained Covering Array Generator using Adaptive Penalty based Parallel Tabu Search
  Yan Wang, Huayao Wu, Changhai Nie, Xintao Niu, Jiaxi Xu
• Applying Combinatorial Testing to High-Speed Railway Automatic Train Protection System
  Rui Huang, Chang Rao, Jeff Yu Lei, Jin Guo, Yadong Zhang
• CA^2: Practical Archival and Compression of Covering Arrays
  Manuel Leithner, Dimitris Simos
• Combination Frequency Differencing for Identifying Design Weaknesses in Physical Unclonable Functions
  Rick Kuhn, M S Raunak, Charles Prado, Vinay Patil, Raghu Kacker
• Developing multithreaded techniques and improved constraint handling for the tool CAgen
  Michael Wagner, Manuel Leithner, Dimitris Simos, Rick Kuhn, Raghu Kacker
• Experience of Combinatorial Testing toward Fault Detection, Isolation and Recovery Functionality
  Naoko Okubo, Shoma Takatsuki, Yasushi Ueda
• Parallel Test Generation for Combinatorial Models Based on Multivalued Decision Diagrams
  Andrea Bombarda, Angelo Gargantini
• Systematic Training and Testing for Machine Learning Using Combinatorial Interaction Testing
  Tyler Cody, Erin Lanus, Daniel D. Doyle, Laura Freeman
• Using Unified Combinatorial Interaction Testing for MC/DC Coverage
  Giray Coskun, Cankut Coskun, Hanefi Mercan, Cemal Yilmaz

IWCT 2021

• A Combinatorial Approach to Explaining Image Classifiers
  Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn.
• A Practical Method for API Testing in the Context of Continuous Delivery and Behavior Driven Development  
  Brian Elgaard Bennett.
• An Environment for Benchmarking Combinatorial Test Suite Generators
  Andrea Bombarda, Edoardo Crippa and Angelo Gargantini.
• A Combinatorial Approach to Testing Deep Neural Network-based Autonomous Driving Systems
  Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn.
• SYSMODIS: A Systematic Model Discovery Approach
  Omer Korkmaz and Cemal Yilmaz.
• Computing Sequence Covering Arrays using Unified Combinatorial Interaction Testing
  Hanefi Mercan and Cemal Yilmaz.
• Combinatorial Testing Metrics for Machine Learning
  Erin Lanus, Laura J. Freeman, D. Richard Kuhn and Raghu N. Kacker.
• Combinatorially XSSing Web Application Firewalls
  Bernhard Garn, Daniel Sebastian Lang, Manuel Leithner, Rick Kuhn, Raghu Kacker and Dimitris E. Simos.

IWCT 2020

• Toward an Encoding Approach to Interaction-based Test Suite Minimization
  Kohei Nishino, Takashi Kitamura, Tomoji Kishi, Cyrille Artho
• An Automata-Based Generation Method for Combinatorial Sequence Testing of Finite State Machines
  Andrea Bombarda, Angelo Gargantini
• Combinatorial Methods for Explainable AI
  Rick Kuhn, Raghu Kacker, Jeff Lei, Dimitris Simos
• Generation of Invalid Test Inputs from Over-Constrained Test Models for Combinatorial Robustness Testing
  Konrad Fögen, Horst Lichter
• On Using Ontologies for Testing Compilers
  Yihao Li, Franz Wotawa
• A Comparison Infrastructure for Fault Characterization Algorithms
  Torben Friedrichs, Konrad Fögen, Horst Lichter
• Avocado: Open-Source Flexible Constrained Interaction Testing for Practical Application
  Jan Richter, Bestoun S. Ahmed, Miroslav Bures, Cleber R. Rosa Junior
• CAGEN: A fast combinatorial test generation tool with support for constraints and higher-index arrays
  Michael Wagner, Kristoffer Kleine, Dimitris Simos, Rick Kuhn, Raghu Kacker

IWCT 2019

• A Framework for Automated Combinatorial Test Generation
  Joshua Bonn, Konrad Fögen and Horst Lichter
• On the Industrial Application of Combinatorial Testing for Autonomous Driving Functions
  Jianbo Tao, Yihao Li, Franz Wotawa, Hermann Felbinger and Mihai Nica
• An Approach to T-way Test Sequence Generation With Constraints
  Feng Duan, Yu Lei, Raghu Kacker and Rick Kuhn
• Efficient and Guaranteed Detection of t-way Failure-inducing Combinations
  Paolo Arcaini, Angelo Gargantini and Marco Radavelli
• Using Parameter Mapping to Avoid Forbidden Tuples in a Covering Array
  Chang Rao, Nan Li, Yu Lei, Raghu Kacker, Richard Kuhn and Jin Guo
• Partitioned Search with Column Resampling for Locating Array Construction
  Erin Lanus, Charles Colbourn and Douglas Montgomery
• Measuring Combinatorial Coverage at Adobe
  Riley Smith, Darryl Jarman, Richard Kuhn, Raghu Kacker, Dimitris Simos and Jared Bellows
• On Measuring Combinatorial Coverage of Manually Created Test Cases for Industrial Software  
  Miraldi Fifo, Eduard Paul Enoiu and Wasif Afzal
• Reduce Test Cost by Reusing Test Oracles through Combinatorial Join 
  Hiroshi Ukai, Xiao Qu, Hironori Washizaki and Yoshiaki Fukazawa
• Applying Combinatorial Testing to Large-scale Data Processing at Adobe
  Riley Smith, Darryl Jarman, Richard Kuhn, Raghu Kacker, Dimitris Simos, Ludwig Kampel, Manuel Leithner and Gabe Gosney
• Practical Fault Localization with Combinatorial Test Design
  Dale Blue, Andrew Hicks, Rachel Tzoref-Brill and Ryan Rawlins
• An Industrial Study on Applications of Combinatorial Testing in Modern Web Development and Cloud Computing   
  Murat Ozcan
• Crucial Tool Features for Successful Combinatorial Input Parameter Testing in an Industrial Application   
  Christian Eitner and Franz Wotawa
• Scenario Sampling for Cyber Physical Systems using Combinatorial Testing 
  Akihisa Yamada, Clovis Eberhart, Fuyuki Ishikawa and Nian-Ze Lee

IWCT 2018 

• A Satisfiability-Based Approach to Generation of Constrained Locating Arrays
  Hao Jin, Takashi Kitamura, Eun-Hye Choi and Tatsuhiro Tsuchiya
• Optimal Scheduling for Combinatorial Software Testing and Design of Experiments
  Robert V. Binder
• Enumerator: An Efficient Approach for Enumerating all Valid t-tuples (short 20 mins)
  Hanefi Mercan, Kamer Kaya and Cemal Yilmaz
• Combinatorial Test Design – smarter way to connect with the business for quality coverage
  Saritha Route and Sudheer Pendela
• Bringing combinatorial interaction testing modeling and generation to the web
  Angelo Gargantini and Marco Radavelli
• CAmetrics: A Tool for Advanced Combinatorial Analysis and Measurement of Test Sets
  Manuel Leithner, Kristoffer Kleine and Dimitris E. Simos
• Combinatorial Testing with Constraints for Negative Test Cases (short 20 mins)
  Konrad Fögen and Horst Lichter
• Practical Amplification of Condition/Decision Test Coverage by Combinatorial Testing
  Artur Andrzejak and Thomas Bach
• The effect of time-between-events for sequence interaction testing of a real-time system
  Chek Pin Yang, Gunwant Dhadyalla, James Marco and Paul Jennings
• Combining Combinatorial Testing and Metamorphic Testing for Testing a Logic-based Non-monotonic Reasoning System (short 20 mins)
  Franz Wotawa
• Adapting Unit Tests by Generating Combinatorial Test (short 20 mins)
  Hermann Felbinger, Franz Wotawa and Mihai Nica

IWCT 2017 

• Estimating t-way Fault Profile Evolution During Testing
  D. Richard Kuhn, Raghu N. Kacker and Jeff Lei[slides]
• Mutation Score, Coverage, Model Inference: Quality Assessment For t-way Combinatorial Test-Suites
  Hermann Felbinger, Franz Wotawa and Mihai Nica[slides]
• Optimizing IPOG’s Vertical Growth with Constraints Based on Hypergraph Coloring
  Feng Duan, Yu Lei, Linbin Yu, Raghu N. Kacker and D. Richard Kuhn  [slides]
• Test Case Generation with Regular Expressions and Combinatorial Techniques
  Macario Polo, Francisco Ruiz Romero, Rosana Rodríguez-Bobada Aranda and Ignacio García-Rodríguez
• Applying Combinatorial Testing to High-Speed Railway Track Circuit Receiver
  Chang Rao, Jin Guo, Nan Li, Jeff Lei, Yadong Zhang, Yao Li and Yaxin Cao  [slides]
• Applications of Practical Combinatorial Testing Methods at Siemens Industry Inc.
  Murat Ozcan  [slides]
• Using Timed Base-Choice Coverage Criterion for Testing Industrial Control Software Bergstrom and Eduard Enoiu  [slides]
• Finding Minimum Locating Arrays Using a SAT Solver
  Tatsuya Konishi, Hideharu Kojima, Hiroyuki Nakagawa and Tatsuhiro Tsuchiya  [slides]
• Test Optimisation using Combinatorial Test Design
  Saritha Route  [slides]
• Building Combinatorial Test Input Model from Use Case Artefacts
  Preeti S, Milind B, Medhini S. Narayan and Krishnan Rangarajan  [slides]
• Combinatorial Methods for Modelling Composed Software Systems
  Ludwig Kampel, Bernhard Garn and Dimitris E. Simos  [slides]
• Combinatorial Interaction Testing for Automated Constraint Repair
  Angelo Gargantini, Justyna Petke and Marco Radavelli  [slides]
• A composition-based method for combinatorial test design (short 15 mins)
  Anna Zamansky, Eitan Farchi, Sery Khoury and Amir Schwartz  [slides]
• Applying Combinatorial Testing to Data Mining Algorithms
  Jaganmohan Chandrasekaran, Huadong Feng, Yu Lei, D. Richard Kuhn and Raghu N. Kacker  [slides]
• Combinatorial Testing on Implementations of HTML5 Support
  Xi Deng, Tianyong Wu, Jun Yan and Jian Zhang  [slides]
• Combinatorial Testing on MP3 for Audio Players (short 15 mins)
  Shaojiang Wang, Tianyong Wu, Yuan Yao, Beihong Jin and Liping Ding   [slides]

IWCT 2016 

• Evaluating the Effectiveness of BEN in Locating Different Types of Software Fault
  Jaganmohan Chandrasekaran, Laleh Sh. Ghandehari, Yu Lei, Raghu Kacker and Richard Kuhn [slides]
• Analyzing Minimal Failure-Causing Schemas in Siemens Suite 
  Chiya Xu, Yuanchao Qi, Ziyuan Wang and Weifeng Zhang
• Should We Care about “Don’t Care” Testing Inputs? Empirical Investigation of Pair-Wise TestingSergiy Vilkomir and Galen Pennell  [slides]
• Towards Automatic Cost Model Discovery for Combinatorial Interaction Testing
  Gulsen Demiroz and Cemal Yilmaz  [slides]
• Combinatorial Testing Implementations in Solutions Testing
  Anwar Sherif
• Embedded Functions for Constraints and Variable Strength in Combinatorial Testing
  George Sherwood  [slides]
• Test Oracles and Test Script Generation in Combinatorial Testing
  Peter M. Kruse  [slides]
• Pseudo-exhaustive Testing of Attribute Based Access Control Rules
  Rick Kuhn, Vincent Hu, David Ferraiolo, Raghu Kacker and Yu Lei
• Applying a pairwise coverage criteria to scenario-based testing 
  Lydie Du Bousquet, Mickaël Delahaye and Catherine Oriat [slides]
• Pairwise Coverage-based Testing with Selected Elements in a Query for Database Applications
  Koji Tsumura, Hironori Washizaki, Yoshiaki Fukazawa, Keishi Oshima and Ryota Mibe
  [slides]

IWCT 2015

• Embedded Functions in Combinatorial Test Designs 
  George Sherwood. 
• Introducing Combinatorial Testing in the Organization a Report on a First Attempt
  Douglas Hillmer. 
• Combining Minimization and Generation for Combinatorial Testing
  Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick. 
• A First Systematic Mapping Study on Combinatorial Interaction Testing for Software Product Lines
  Roberto Erick Lopez-Herrejon, Stefan Fischer, Rudolf Ramler and Alexander Egyed. 
• Evaluation of the IPO-Family Algorithms for Test Case Generation in Web Security Testing
  Josip Bozic, Bernhard Garn, Dimitris. E. Simos and Franz Wotawa. 
• A process for non-functional combinatorial testing (short)
  Siegmar Züfle and Venkataraman Krishnamoorthy. 
• Combinatorial Interaction Testing of Tangled Configuration Options (short)
  Arsalan Javeed and Cemal Yilmaz. 
• Relationship between Pair-wise and MC/DC Testing: Initial Experimental Results (short)
  Sergiy Vilkomir and David Anderson. 
• Why Combinatorial Testing Works: Analyzing Minimal Failure-Causing Schemas in Logic Expressions (short)
  Ziyuan Wang and Yuanchao Qi
• Constraint Handling In Combinatorial Test Generation Using Forbidden Tuples
  Linbin Yu, Feng Duan, Yu Lei, Raghu N. Kacker and D. Richard Kuhn. 
• Using Decision Trees to aid Algorithm Selection in Combinatorial Interaction Test Generation
  Angelo Gargantini and Paolo Vavassori. 
• Flattening or Not of the Combinatorial Interaction Testing Models?(short)
  Christopher Henard, Mike Papadakis and Yves Le Traon. 
• Improving IPOG’s Vertical Growth Based on a Graph Coloring Scheme
  Feng Duan, Linbin Yu, Yu Lei, Raghu N. Kacker and D. Richard Kuhn. 
• Equivalence Class Verification and Oracle-free Testing Using Two-layer Covering Arrays (short)
  Rick Kuhn, Raghu Kacker, Yu Lei and Jose Torres-Jimenez. 
• BEN: A Combinatorial Testing-Based Fault Localization Tool (short) 
  Laleh Shikh Gholamhosseinghandehari, Jaganmohan Chandrasekaran, Yu Lei, Raghu Kacker and Richard Kuhn. 

IWCT 2014 

• Combinatorial Testing for an Automotive Hybrid Electric Vehicle Control System - A Case Study, Gunwant Dhadyalla, Neelu Kumari and Timothy Snell
• Eris: A Tool for Combinatorial Testing of the Linux System Call Interface,
  Bernhard Garn and Dimitris. E. Simos
• An Empirical Comparison of Combinatorial and Random Testing,
  Laleh Shikh Gholamhosseingghandehari, Jacek Czerwonka, Yu Lei, Soheil Shafiee, Raghu Kacher and Richard Kuhn
• Using Combinatorial Approaches for Testing Mobile Applications,
  Sergiy Vilkomir and Brandi Amstutz
• Combinatorial interaction testing of a Java Card static verifier (short paper),
  Andrea Calvagna, Andrea Fornaia and Emiliano Tramontana
• Extracting the Combinatorial Test Parameters and Values from UML Sequence Diagrams,
  Preeti Satish, Arinjita Paul and Krishnan Rangarajan
• Validation of Models and Tests for Constrained Combinatorial Interaction Testing,
  Paolo Arcaini, Angelo Gargantini and Paolo Vavassori
• Functional Dependence and Equivalence Class Factors in Combinatorial Test Designs,
  George Sherwood
• Combinatorial Testing with Order Requirements,
  Eitan Farchi, Itai Segall, Rachel Tzoref-Brill and Aviad Zlotnick
• Test Suite Reduction by Combinatorial-based Coverage of Event Sequences (short paper),
  Quentin Mayo, Renee Bryce and Ryan Michaels
• Test Suite Prioritization by Switching Cost,
  Huayao Wu, Changhai Nie and Fei-Ching Kuo
• Refining the Randomized Post-optimization Strategy for Covering Arrays,
  Xiaohua Li, Zhao Dong, Huayao Wu, Changhai Nie and Kai-Yuan Cai
• Introducing Combinatorial Testing in a Large Organization: Pilot Project Experience Report,
  Jon Hagar, Rick Kuhn, Raghu Kacker and Tom Wissink
• Estimating Fault Detection Effectiveness,
  Rick Kuhn, Raghu Kacker and Yu Lei

IWCT 2013 

• Empirically Identifying the Best Greedy Algorithm for Covering Array Generation,
  Changhai Nie, Jing Jiang, Huayao Wu, Hareton Leung and Charles J. Colbourn
• Evaluation of t-wise Approach for Testing Logical Expressions in Software,
  Sergiy Vilkomir, Oleksii Starov and Ranjan Bhambroo
• On Use of Coverage Metrics in Assessing Effectiveness of Combinatorial Test Designs,
  Jacek Czerwonka
• Cascade: A Test Generation Tool For Combinatorial Testing,
  Yong Zhao, Zhiqiang Zhang, Jun Yan and Jian Zhang
• Identifying Failure-Inducing Combinations Using Tuple Relationship,
  Xintao Niu, Changhai Nie, Lei Yu and Alvin Chan
• Better Bounds for Event Sequencing Testing,
  Oded Margalit
• A Study in Prioritization for Higher Strength Combinatorial Testing,
  Xiao Qu and Myra Cohen
• Combinatorial Test Architecture Design Using Viewpoint diagram,
  Yasuharu Nishi, Tetsuro Katayama and Satomi Yoshizawa
• Experimenting with Category Partition's 1-way and 2-way test selection criteria,
  Yvan Labiche and Faezeh Sadeghi
• Using Projections to Debug Large Combinatorial Models,
  Eitan Farchi, Itai Segall and Rachel Tzoref-Brill
• Combinatorial Interaction Testing with Multi-Perspective Feature Models,
  Sachin Patel, Priya Gupta and Vipul Shah
• Deriving Combinatorial Test Design Model from UML Activity Diagram,
  Preeti Satish, Sheeba K and Krishnan Rangarajan
• Combinatorial Testing for Feature Models Using CitLab,
  Andrea Calvagna, Angelo Gargantini and Paolo Vavassori
• Applications of Combinatorial Testing methods for Breakthrough Results in Software Testing,
  Manish Mehta and Roji Philip
• Combinatorial Coverage Measurement Concepts and Applications,
  Rick Kuhn, Itzel Dominguez, Raghu Kacker and Yu Lei
• Applying Combinatorial Testing to the Siemens Suite,
  Laleh Shikh Gholamhossein Ghandehari, Mehra N. Bourazjany, Yu Lei, Raghu Kacker and Richard Kuhn
• An Input Space Modeling Methodology for Combinatorial Testing,
  Mehra Nouroz Borazjany, Laleh Shikh Gholamhossein Ghandehari, Yu Lei, Raghu Kacker and Rick Kuhn

IWCT 2012

• Effectiveness of Pair-wise Testing for Software with Boolean Inputs  
  William Alton Ballance, Sergiy Vilkomir, William Jenkins
• Combinatorial Testing of ACTS: A Case Study  
  Mehra N.Borazjany, Linbin Yu, Yu Lei, Raghu Kacker, Rick Kuhn
• A System Analysis Study Comparing Reverse Engineered Combinatorial Testing to Expert Judgment 
  Atlee M. Cunningham, Jr., Jon Hagar, Ryan J. Holman
• CITLAB: a Laboratory for Combinatorial Interaction Testing 
  Angelo Gargantini, Paolo Vavassori
• Identifying Failure-Inducing Combinations in a Combinatorial Test Set  
  Laleh Shikh Gholamhossein Ghandehari, Yu Lei, Tao Xie, Richard Kuhn, Raghu Kacker
• Combinatorial Methods for Event Sequence Testing  
  D. Richard Kuhn, James M. Higdon, James F. Lawrence, Raghu N. Kacker, Yu Lei
• Combinatorial Test Design in the TOSCA Testsuite: Lessons Learned and Practical Implications 
  Rudolf Ramler, Theodorich Kopetzky
• Combinatorial Interaction Testing for Test Selection in Grammar-Based Testing 
  Elke Salecker, Sabine Glesner
• Common Patterns in Combinatorial Models 
  Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
• Simplified Modeling of Combinatorial Test Spaces 
  Itai Segall, Rachel Tzoref-Brill, Aviad Zlotnick
• Isolating Failure-Inducing Combinations in Combinatorial Testing using Test Augmentation and Classification 
  Kiran Shakya, Tao Xie, Nuo Li, Yu Lei, Raghu Kacker, Richard Kuhn
• Combinatorial Testing on ID3v2 Tags of MP3 Files 
  Zhiqiang Zhang, Xiaojian Liu, Jian Zhang