Official websites use .gov
A .gov website belongs to an official government organization in the United States.

Secure .gov websites use HTTPS
A lock ( ) or https:// means you’ve safely connected to the .gov website. Share sensitive information only on official, secure websites.

Combinatorial Methods for Trust and Assurance

DOs and DON'Ts of testing

DON'T assume that 2-way combinations (pairwise testing) will be enough. Empirical data, documented in papers on this site, show that 2-way combinations are important, but a large proportion of faults involve more than two parameters. 
but
DO consider the appropriate level of t-way combinations to be used.   It is reasonable to expect that 30% or more of the faults that need to be found in testing may require three factors for detection.

 

DON’T try to develop the input model (the parameters and test values) only from use cases.  Considering only use cases is likely to lead to missing some important values, since testers may miss some possibilities in developing ad hoc cases.
but
DO use requirements specifications in developing the input model.  While use cases may give hints of the parameters and values that are important, requirements are generally a better source.

 

DON'T limit the input model to just input parameters. Combinatorial testing can be applied even if there is only one input. 
but
DO consider abstract parameters in developing tests.  These are properties that are not input parameters, but are important in testing. For example, in testing a string search, there may be only one input, but abstract parameters should capture properties such as whether the string contains embedded quotes, spaces, length of string, whether string is present in file, etc. Some faults may only appear with inputs that have combinations of these properties. 

 

DON’T try to use too many values for test parameters.  In general, it is best to limit the number of values per parameter to no more than 10 or 12. This is because the number of tests increases rapidly as more values are added.  
but
DO try to minimize the number of values.  If possible, it is better to keep the number of parameter values to around 4 to 6 in most cases. 

 

DON'T randomly select representative values for continuous-value parameters (such as amounts or distances), or use just low, medium, and high values.
but
DO consider boundary values and proper equivalence partitioning in choosing these values. 

Contacts

Rick Kuhn
kuhn@nist.gov
Address: https://www.nist.gov/people/d-richard-kuhn

Raghu Kacker
raghu.kacker@nist.gov
301-975-2109
Address: http://math.nist.gov/~RKacker/

M S Raunak
raunak@nist.gov

Topics

Security and Privacy: assurance, modeling, testing & validation

Technologies: software & firmware

Created May 24, 2016, Updated July 22, 2024