Moreover, we relied on adequate statistical methods in order to reason about data normality and whether we did really find statistical difference between TTR 1.1 and TTR 1.2. A recent SAT-based approach (Yamada et al. 2016), implemented in the Calot tool, performed well in terms of what is combinatorial testing efficiency (time to generate the test suites) and cost (test suite sizes) comparing again with the greedy tools ACTS (Yu et al. 2013) and PICT (Czerwonka 2006). Despite the advantages of the SAT-based approach, ACTS was much more faster than Calot for many 3-way test case examples.

After the construction of the matrix Θ, the initial solution, and the calculation of the goals of all t-tuples, Main sort Θ so that the elements belonging to the parameter interaction with the greatest amount of t-tuples get ahead (line 1). Since the matrix M is being traversed in the loop (line 4), it will be updated every time a t-tuple is combined with some of its test cases (note ⊕ in line 5). The main reasoning of TTR 1.1 is to build an MCA M through the reallocation of t-tuples from a matrix Θ to this matrix M, and then each reallocated t-tuple should cover the greatest number of t-tuples not yet covered, considering a parameter called a goal (ζ).

An algorithm for combinatorial interaction testing: definitions and rigorous evaluations

Thus, it is interesting to think about a new greedy solution for CIT that does not need, at the beginning, to enumerate all t-tuples (such as PICT) and does not demand many auxiliary matrices to operate (as some IPO-based approaches). In the context of CIT, meta-heuristics such as simulated annealing (Garvin et al. 2011), genetic algorithms (Shiba et al. 2004), and Tabu Search Approach (TSA) (Hernandez et al. 2010) have been used. Recent empirical studies show that meta-heurisitic and greedy algorithms have similar performance (Petke et al. 2015). Hence, early fault detection via a greedy algorithm with constraint handling (implemented in the ACTS tool (Yu et al. 2013)) was no worse than a simulated annealing algorithm (implemented in the CASA tool (Garvin et al. 2011)). Moreover, there was not enough difference between test suites generated by ACTS and CASA in terms of efficiency (runtime) and t-way coverage.

The motivation for such a change is because we realized that, in some cases, less test cases were created due to non-ordering of parameters and values. Case studies below are from many types of applications, including aerospace, automotive, autonomous systems, cybersecurity, financial systems, video games, industrial controls, telecommunications, web applications, and others. If this is not done, the final goal will never be matched, since there are no uncovered t-tuples that correspond to this interaction. Before going on with the descriptions of the procedures of our algorithm, we need to define the following operators applied to the structures (set, sequence, matrix) we handle. We also present some examples to better illustrate how such operators work. An empirical comparison of combinatorial testing, random testing and adaptive random testing.

Stay in the loop with the lastest software testing news

This insertion is done in the same way as the initial solution for M is constructed, as described in the section above. In this paper, we present a new algorithm, called T-Tuple Reallocation (TTR), to generate CIT test suites specifically via the Mixed-value Covering Array (MCA) technique. The main reasoning behind TTR is to generate an MCA M by creating and reallocating t-tuples into this matrix M, considering a variable called goal (ζ). In the context of software systems, robustness testing aims to verify whether the Software Under Test (SUT) behaves correctly in the presence of invalid inputs. Therefore, even though an unconstrained CIT-derived test case may seem pointless or even somewhat difficult to execute, it may still be interesting to see how the software will behave in the presence of inconsistent inputs.

We relied on the experimentation process proposed in (Wohlin et al. 2012), using the R programming language version 3.2.2 (Kohl 2015). Both algorithms/tools (TTR 1.1, TTR 1.2) were subjected to each one of the 80 test instances (see Table 11), one at a time. The output of each algorithm/tool, with the number of test cases and the time to generate them, was recorded. Considering the metrics we defined in this work and based on both controlled experiments, TTR 1.2 is a better option if we need to consider higher strengths (5, 6). ‘Enabled’, ‘Choice Type’ and ‘Category’ have a choice range of 2, 3 and 4, respectively. Multiplying the two largest values (3 and 4) indicates that a pair-wise tests would involve 12 tests.

The conclusion validity has to do with how sure we are that the treatment we used in an experiment is really related to the actual observed outcome (Wohlin et al. 2012). One of the threats to the conclusion validity is the reliability of the measures (Campanha et al. 2010). We automatically obtained the measures via the implementations of the algorithms and hence we believe that replication of this study by other researchers will produce similar results. Even if other researchers may get different absolute results, especially related to the time to generate the test suites simply because such results depend on the computer configuration (processor, memory, operating system), we dot not expect a different conclusion validity.

It measures the degree to which combinations of input values have been covered in tests, which is a static property of the test set. Measures such as statement or branch coverage are dynamic properties, as they measure the proportion of statements and branches covered when the program is running. Combinatorial strategies are extended to generate invalid test inputs but the effectiveness of negative test scenarios is yet unclear. Therefore, we conduct a case study and analyze 434 failures reported as bugs of a financial enterprise application. As a result, 51 robustness failures are identified including failures triggered by invalid value combinations and failures triggered by interactions of valid and invalid values.

• Combinatorial testing is being applied successfully in nearly every industry, and is especially valuable for assurance of high-risk software with safety or security concerns.
• We also present some examples to better illustrate how such operators work.
• We can explain this better performance of TTR 1.2 due to the fact that it no longer generates, at the beginning, the matrix of t-tuples but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M.
• The conclusion validity has to do with how sure we are that the treatment we used in an experiment is really related to the actual observed outcome (Wohlin et al. 2012).
• They integrate testing, analysis, and environmental and operational assumptions, from which the set of conditions that testing must cover is determined.
• IPOG-F (Forbes et al. 2008) is an adaptation of the IPOG algorithm (Lei et al. 2007).

To obtain the tools, please send a request to Rick Kuhn –   including your name and the name of your organization. No other information is required, but we like to have a list of organizations so that we can show our management where the software is being used. Software on this site is free of charge and will remain free in the future. It is public domain; no license is required and there are no restrictions on use.

In such experiment, we jointly considered cost (size of test suites) and efficiency (time to generate the test suites) in a multi-objective perspective. We conclude that TTR 1.2 is more adequate than TTR 1.1 especially for higher strengths (5, 6). This is explained by the fact that, in TTR 1.2, we no longer generate the matrix of t-tuples (Θ) but rather the algorithm works on a t-tuple by t-tuple creation and reallocation into M.