A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering

By Andrea Arcuri, Lionel Claude Briand. Software testing, verification & reliability, 2014

650 citations recorded by the source corpus.

Randomized algorithms are widely used to address many types of software engineering problems, especially in the area of software verification and validation with a strong emphasis on test automation. However, randomized algorithms are affected by chance and so require the use of appropriate statistical tests to be properly analysed in a sound manner. This paper features a systematic review regarding recent publications in 2009 and 2010 showing that, overall, empirical analyses involving randomized algorithms in software engineering tend to not properly account for the random nature of these algorithms. Many of the novel techniques presented clearly appear promising, but the lack of soundness in their empirical evaluations casts unfortunate doubts on their actual usefulness. In software engineering, although there are guidelines on how to carry out empirical analyses involving human subjects, those guidelines are not directly and fully applicable to randomized algorithms. Furthermore, many of the textbooks on statistical analysis are written from the viewpoints of social and natural sciences, which present different challenges from randomized algorithms. To address the questionable overall quality of the empirical analyses reported in the systematic review, this paper provides guidelines on how to carry out and properly analyse randomized algorithms applied to solve software engineering tasks, with a particular focus on software testing, which is by far the most frequent application area of randomized algorithms within software engineering. Copyright © 2012 John Wiley & Sons, Ltd.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/b3de8bfff5b2a3ec070fbd021a4544af974542fc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

Decoding the Representation of Code in the Brain: An fMRI Study of Code Review and Expertise (2017, methodology): “Median BAC was compared for each of the three models using nonparametric Wilcoxon rank-sum tests [2].”
REDQUEEN: Fuzzing with Input-to-State Correspondence (2019, neutral): “[7] for evaluation of randomized algorithms.”
Large Language Model guided Protocol Fuzzing (2024, positive): “…achieved in 24 hours ( Improv ), how much faster C HAT AFL can achieve the same transition coverage as the baseline in 24 hours ( Speed-up ), and the probability that a random campaign of C HAT AFL outperforms a random campaign of the baseline ( ˆ A 12 , Vargha-Delaney measure of effect size [5]).”
Directed Greybox Fuzzing (2017, background): “(Â12) is a non-parametric measure of effect size [39] and is also the recommended standard measure for the evaluation of randomized algorithms [1].”
NAUTILUS: Fishing for Deep Bugs with Grammars (2019, result): “[21] to ensure statistical significance and we report the results in Table III.”