Defects4J: a database of existing faults to enable controlled testing studies for Java programs

By René Just, Darioush Jalali, Michael D. Ernst. 2014

1,713 citations recorded by the source corpus.

Subfields: Information Systems, Software, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Evaluation & Benchmarks

Citations in the corpus

Break-It-Fix-It: Unsupervised Learning for Program Repair (2021, background): “Several works learn to repair code from labeled datasets of source code edits made by programmers, e.g., error resolution records (Just et al., 2014; Chen et al., 2019; Mesbah et al., 2019; Bader et al., 2019; Tarlow et al., 2020; Ding et al., 2020).”
Graph-based, Self-Supervised Program Repair from Diagnostic Feedback (2020, background): “Another line of work learns from labeled datasets of how programmers edit code (e.g. error resolution records) (Just et al., 2014; Chen et al., 2019).”
A Survey of Machine Learning for Big Code and Naturalness (2017, background): “Defects4J [100]—a curated corpus of bugs—could further prove useful within machine learning for fault prediction.”
Unit Test Case Generation with Transformers (2020, methodology): “For this comparison we select a small but reproducible testbed using defects4j [34].”
SequenceR: Sequence-to-Sequence Learning for End-to-End Program Repair (2018, methodology): “• RQ3: How effective is S EQUENCE R’s sequence-to-sequence learning in ﬁxing bugs in the well-established Defects4J benchmark?”
An Empirical Study on Learning Bug-Fixing Patches in the Wild via Neural Machine Translation (2018, methodology): “To have enough training data, we mined bug-fixes in GitHub repositories rather than using curated bug-fix datasets such as Defects4j [33] or IntroClass [43], useful but very limited in size.”
Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code (2020, methodology): “To assess whether our code NLM is applicable beyond code completion, we compare the ability of diﬀer-ent language models to diﬀerentiate between the two on the well-known Defects4j dataset [53].”
Neural Program Repair by Jointly Learning to Localize and Repair (2019, methodology): “The APR community has also designed some repair bechmarks, such as ManyBugs and IntroClass (Goues et al., 2015), and Defects4J (Just et al., 2014), for test-based program repair techniques.”
Context-Aware Patch Generation for Better Automated Program Repair (2018, methodology): “The benchmark was built to facilitate controlled experiments in software debugging and testing researches [14], which has been widely adopted by recent studies on APR[19, 27, 29, 40, 54].”
Is the cure worse than the disease? overfitting in automated program repair (2015, background): “Empirical studies of fixes of real bugs in open-source projects can also improve repair by helping designers select change operators and search strategies [27, 64].”
Learning semantic program embeddings with graph interval neural network (2020, methodology): “In addition to bugs provided by existing datasets [Just et al. 2014; Saha et al. 2018; Ye et al. 2014], we extract additional bugs from Bugswarm projects [Tomassi et al.”
Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction (2022, methodology): “[7] note that in the widely used Defects4J [8] bug benchmark, bugrevealing tests did not exist prior to the bug report being filed in 96% of the cases.”
Shaping program repair space with existing patches and similar code (2018, methodology): “1 Experiment Setup Weevaluated SimFix onDefects4J [21] v1.”
CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, positive): “CodeScope not only evaluates the capability of LLMs to generate new code snippets but also tests their proficiency in understanding and analyzing existing complex code, reflecting the diverse challenges encountered in real-world programming scenarios.”
A syntax-guided edit decoder for neural program repair (2021, methodology): “For example, let us consider the patch of defect Closure-14 in the Defects4J benchmark [26], as shown in Figure 1.”
An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation (2023, methodology): “In experiments on 5 projects from Defects4J [69], AthenaTest generated 158K test cases, achieving similar test coverage as EvoSuite [16], a popular search-based test generation tool, and covering 43% of all focal methods.”
RepairAgent: An Autonomous, LLM-Based Agent for Program Repair (2024, positive): “We apply RepairAgent to all bugs in the Defects4J dataset [24], which consists of 835 real-world bugs from 17 Java projects, including 395 bugs from 6 projects in Defects4Jv1.”