A practical guide for using statistical tests to assess randomized algorithms in software engineering

By Andrea Arcuri, Lionel Claude Briand. International Conference on Software Engineering, 2011

1,110 citations recorded by the source corpus.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/92ca9f57ebb0cfdbfa07e5d5956e11509f902f0b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

Citations in the corpus

A Hitchhiker's guide to statistical tests for assessing randomized algorithms in software engineering (2014, background): “This paper is an extension of a conference paper [10] published in the I ternational Conference on Software Engineering (ICSE), 2011.”
Software Architecture Optimization Methods: A Systematic Literature Review (2013, methodology): “analyzing them, the recently published guidelines by Arcuri and Briand [11] can be recommended.”
History Driven Program Repair (2016, background): “io/github/SpoonLabs/astor our approach, PAR and GenProg are all stochastic, multiple trials are needed to properly assess their performances [4].”
The strength of random search on automated program repair (2014, methodology): “We suggest that an advanced repair approach should always be compared against at least the baseline approach to check the advantage of the new approach, which is consistent with the declaration presented in [4, 15, 14].”
Is the cure worse than the disease? overfitting in automated program repair (2015, methodology): “Our work is consistent with the guidelines for evaluating randomized algorithms [5] to enhance the credibility of our findings.”
Fairness testing: testing software for discrimination (2017, methodology): “Apparent discrimination captures this notion, applying group or causal discrimination score measurement to a subset of the input domain, which can be described by an operational profile [7, 58].”
Search Based Software Engineering: Techniques, Taxonomy, Tutorial (2010, background): “However, there has been a recent paper on the topic of statistical testing of randomised algorithms by Arcuri and Briand [8], which provides more detail.”
Mutation-Driven Generation of Unit Tests and Oracles (2010, methodology): “[2] J. H. Andrews, L. C. Briand, and Y. Labiche.”
Parameter tuning or default values? An empirical investigation in search-based software engineering (2013, methodology): “To analyze all these data in a sound manner, we followed the guidelines in [3].”
Sapienz: multi-objective automated testing for Android applications (2016, methodology): “Furthermore, since Sapienz and the other techniques use randomised algorithms, it is widely regarded as best practice to perform an inferential statistical analysis of the performance of each algorithm, reporting statistical signiﬁcance and eﬀect size [18,40].”
Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “In particular, SBSE bears many similarities to LLM-based Software Engineering, sharing with it the need to achieve robust scientific evaluation in the presence of noisy, non-deterministic, and incomplete results [12], [13].”
Evaluating Fuzz Testing (2018, methodology): “Arcuri and Briand [2] suggest that for randomized testing algorithms (like fuzzers), one should use the Mann Whitney U-test to determine the stochastic ranking of A and B , i.e., whether the outcomes of the trials in A ’s data sample are more likely to be larger than outcomes in B ’s. Mann Whitney…”
An Empirical Comparison of Model Validation Techniques for Defect Prediction Models (2017, neutral): “[3].”
Chapter Six - Mutation Testing Advances: An Analysis and Survey (2019, background): “To reduce this problem, researchers usually select multiple sets of test cases and perform an inferential statistical analysis on the set of results as a whole, [399, 400].”
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair (2018, methodology): “run each configuration multiple times and report the mean and standard deviation for the model as recommended for assessment of random algorithms [42].”