Supporting Controlled Experimentation with Testing Techniques: An Infrastructure and its Potential Impact

By Hyunsook Do, Sebastian G. Elbaum, Gregg Rothermel. Empirical Software Engineering, 2005

1,199 citations recorded by the source corpus.

Subfields: Hardware and Architecture, Software

View Paper: https://www.semanticscholar.org/paper/94782b2954ae2451a5963cdc39a954b5bfd077aa

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Hardware and Architecture

In Software

Citations in the corpus

A model for spectra-based software diagnosis (2011, methodology): “The versions of the test suites we used were obtained from the Software Information Repository [Do et al. 2005].”
Statistical Debugging: A Hypothesis Testing-Based Approach (2006, methodology): “6.1 A Controlled Experiment with grep 2.2 We obtained a copy of the grep 2.2 subject program from the “Subject Infrastructure Repository” (SIR) [ 29 ].”
Are mutants a valid substitute for real faults in software testing? (2014, methodology): “2 Commonly Used Artifacts Many research papers use programs from the Siemens benchmark suite [15] or the software-artifact infrastructure repository (SIR) [10] in their evaluation.”
Metallaxis‐FL: mutation‐based fault localization (2015, background): “However, the researchers, created these tests independent to the present study, to support their experimentation [28], [29] and [30].”
Are automated debugging techniques actually helping programmers? (2011, methodology): “As our second subject, we selected NanoXML, a library for XML parsing available from the SIR repository [5].”
The ManyBugs and IntroClass Benchmarks for Automated Repair of C Programs (2015, positive): “" Some evaluations have used the SIR repository, focusing primarily on the subset constituting the Siemens benchmark suite [45], which currently consists of eight programs of 173– 726 lines of code.”
SemFix: Program repair via semantic analysis (2013, methodology): “It is a code excerpt taken from Tcas, traffic collision avoidance system [12].”
Regression testing minimization, selection and prioritization: a survey (2012, methodology): “Our survey also provides evidence to indicate that there is a preponderance of empirical work that draws upon a comparatively small set of subjects (notably those available through the SIR repository).”
A systematic review on regression test selection techniques (2010, background): “We identified a need for empirical studies where concepts are evaluated rather than small variations in technical implementations.”
Software Testing Research: Achievements, Challenges, Dreams (2007, background): “This is obviously true for testing as well, in which controlled experimentation is an indispensable research methodology [26].”
A practical evaluation of spectrum-based fault localization (2009, methodology): “In our study we work with two sets of faults that are available from the Software-artifact Infrastructure Repository (SIR, Do et al., 2005): the Siemens set (Hutchins et al., 1994), which is a widely-used collection of benchmark faults in seven small C programs, and a set of faults in a somewhat lar”
Defects4J: a database of existing faults to enable controlled testing studies for Java programs (2014, background): “In contrast to SIR, Defects4J provides 357 real bugs for 5 large real-world programs ranging between 22,000 and 96,000 LOC.”
Ask the Mutants: Mutating Faulty Programs for Fault Localization (2014, background): “18, and space, all of which are from the SIR benchmark suite [4].”
A Theoretical and Empirical Study of Search-Based Testing: Local, Global, and Hybrid Search (2010, methodology): “space is a program from the European Space Agency and is available from the Software-artifact Infrastructure Repository [1], [16].”
DirectFix: Looking for Simple Program Repairs (2015, background): “Table I shows our subject programs comprised of eighty nine buggy versions of four subject programs from SIR (Software-artifact Infrastructure Repository) [21] (the number of versions for each subject is shown in the #Versions column) and nine buggy versions of Coreutils reported by Cadar et al [”