Computer Security Resource Center

Computer Security Resource Center

Computer Security
Resource Center

Automated Combinatorial Testing for Software

DOs and DON'Ts of testing

DON'T assume that 2-way combinations (pairwise testing) will be enough. Empirical data, documented in papers on this site, show that 2-way combinations are important, but a large proportion of faults involve more than two parameters. 
but
DO consider the appropriate level of t-way combinations to be used.   It is reasonable to expect that 30% or more of the faults that need to be found in testing may require three factors for detection.

 

DON’T try to develop the input model (the parameters and test values) only from use cases.  Considering only use cases is likely to lead to missing some important values, since testers may miss some possibilities in developing ad hoc cases.
but
DO use requirements specifications in developing the input model.  While use cases may give hints of the parameters and values that are important, requirements are generally a better source.

 

DON'T limit the input model to just input parameters. Combinatorial testing can be applied even if there is only one input. 
but
DO consider abstract parameters in developing tests.  These are properties that are not input parameters, but are important in testing. For example, in testing a string search, there may be only one input, but abstract parameters should capture properties such as whether the string contains embedded quotes, spaces, length of string, whether string is present in file, etc. Some faults may only appear with inputs that have combinations of these properties. 

 

DON’T try to use too many values for test parameters.  In general, it is best to limit the number of values per parameter to no more than 10 or 12. This is because the number of tests increases rapidly as more values are added.  
but
DO try to minimize the number of values.  If possible, it is better to keep the number of parameter values to around 4 to 6 in most cases. 

 

DON'T randomly select representative values for continuous-value parameters (such as amounts or distances), or use just low, medium, and high values.
but
DO consider boundary values and proper equivalence partitioning in choosing these values. 

Created May 24, 2016, Updated October 07, 2019