The Oracle Problem in Software Testing: A Survey

By Earl T. Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, Shin Yoo. IEEE Transactions on Software Engineering, 2015

1,129 citations recorded by the source corpus.

Testing involves examining the behaviour of a system in order to discover potential faults. Given an input for a system, the challenge of distinguishing the corresponding desired, correct behaviour from potentially incorrect behavior is called the “test oracle problem”. Test oracle automation is important to remove a current bottleneck that inhibits greater overall test automation. Without test oracle automation, the human has to determine whether observed behaviour is correct. The literature on test oracles has introduced techniques for oracle automation, including modelling, specifications, contract-driven development and metamorphic testing. When none of these is completely adequate, the final source of test oracle information remains the human, who may be aware of informal specifications, expectations, norms and domain specific information that provide informal oracle guidance. All forms of test oracles, even the humble human, involve challenges of reducing cost and increasing benefit. This paper provides a comprehensive survey of current approaches to the test oracle problem and an analysis of trends in this important area of software testing research and practice.

Subfields: Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/0e17ab6c9d31a481c641f2b1760e09ea51ab76cc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Mark Harman
2. Yue Jia
3. Yuriy Brun
4. Shin Yoo
5. Ke Mao
6. Gordon Fraser
7. Jie M. Zhang
8. Yang Liu

In Software

1. Mark Harman
2. Yue Jia
3. Shin Yoo
4. Yuriy Brun
5. Ke Mao
6. Gordon Fraser
7. Jie M. Zhang
8. Yang Liu

Citations in the corpus

• Metamorphic Testing (2018, background): “In recent surveys of the oracle problem [4, 68, 71, 72], a significant amount of discussion was devoted to MT, which”
• Sapienz: multi-objective automated testing for Android applications (2016, methodology): “It uses a standard, simple-but-effective, default test oracle [22] that regards any input that reveals a crash to be a fault-revealing test sequence.”
• Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction (2022, background): “Similar to our work, some techniques focus on reproducing problems reported by users: a commonly used implicit oracle is that the program should not crash [32].”
• Fairness testing: testing software for discrimination (2017, background): “While prior work has tackled the oracle problem [10, 26, 30] typically using inferred pre- and post-conditions or documentation, our oracle is more precise and easier to compute, but is only applicable to fairness testing.”
• An empirical study on TensorFlow program bugs (2018, methodology): “[1] studied test oracles in software testing including probabilistic test oracle.”
• Software Testing With Large Language Models: Survey, Landscape, and Vision (2023, neutral): “A test oracle is a source of information about whether the output of a software system (or program or function or method) is correct or not [136].”
• Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “Challenges Relating to The Oracle Problem.”
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, methodology): “, often called as the focal method), its corresponding unit test consists of a test prefix and a test oracle [4].”
• Large Language Models for Software Engineering: Survey and Open Problems (2023, methodology): “3) Test Correctness: Traditional software test generation has suffered from the Oracle Problem [6], i.e., they are inhibited by the lack of an automated oracle that determines whether a test outcome is correct.”
• A Survey on Metamorphic Testing (2016, methodology): “This problem is referred to as the oracle problem and it is recognised as one of the fundamental challenges of software testing [1], [2], [3], [4].”
• Trustworthy AI: From Principles to Practices (2021, methodology): “Similar to canonical software testing, the problem to automatically generate the expected ground truth, known as the oracle problem [34], also occurs in the AI software testing scenario.”
• A survey of the use of crowdsourcing in software engineering (2017, background): “5) Oracle Problem: An oracle is typically needed to de- 1200 termine the required output of a program for a given input (Weyuker, 1982; Barr et al., 2015).”
• Exploring the Integration of Large Language Models in Industrial Test Maintenance Processes (2024, methodology): “Each test case performs a series of input actions, records observations of how the software behaved following the application of input, then compares those observations to a set of expectations (known as a test oracle [10]).”
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval (2025, unknown): “These challenges underscore the urgent need for retrieval systems that go beyond surface-level functional matching to explicitly account for code quality, leading to a central research question: Can code retrievers be designed based on not only relevance but also quality?”