Differential Testing for Software

By W. McKeeman. 1998

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, methodology): “Such differential testing techniques have been applied successfully in the past for detecting logic bugs without manual specifications in a wide variety of traditional software [6, 11, 14, 15, 45, 86].”
Synthesizing benchmarks for predictive modeling (2017, methodology): “Our method is based on random differential testing [20], though we emphasize that this is not a general purpose approach and is tailored specifically for our use case.”
Test-case reduction for C compiler bugs (2012, methodology): “McKeeman presented a random C program generator for testing C compilers [11], including a testcase reduction tool that operates at a finer granularity than Bugfind or vpoiso.”
Machine Learning Testing: Survey, Landscapes and Horizons (2019, background): “Differential testing is a traditional software testing technique that detects bugs by observing whether similar applications yield different outputs regarding identical inputs [11], [131].”
Compiler validation via equivalence modulo inputs (2014, unknown): “C; H.3.4 [Programming Languages]: Processors—compilers General Terms Algorithms, Languages, Reliability, Verification Keywords Compiler testing, miscompilation, equivalent program variants, automated testing”
Using Frankencerts for Automated Adversarial Testing of Certificate Validation in SSL/TLS Implementations (2014, background): “Georgiev et al. focus primarily on the incorrect validation of hostnames in server certificates at a different level in the software stack—in applications, transport libraries, and Webservices middleware.”
The Art, Science, and Engineering of Fuzzing: A Survey (2018, background): “Semantic bugs are often discovered using a technique called differential testing [153], which compares the behavior of similar (but not identical) programs.”
SFADiff: Automated Evasion Attacks and Fingerprinting Using Black-box Differential Automata Learning (2016, result): “Differential testing is a way of testing a program without any manually crafted specifications by comparing its outputs to those of other comparable programs for the same set of inputs [23].”
Coverage-directed differential testing of JVM implementations (2016, background): “Differential testing [18, 27] can be a promising approach to revealing defects in JVMs by comparing multiple systems.”
I Am Going MAD: Maximum Discrepancy Competition for Comparing Classifiers Adaptively (2020, unknown): “In addition, MAD can also be linked to the popular technique of differential testing (McKeeman, 1998) in software engineering.”
Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation (2023, methodology): “We use differential testing [32] as the oracle to cross-check the output of the ground-truth and LLM-generated solution.”
Guided differential testing of certificate validation in SSL/TLS implementations (2015, background): “Differential testing is a mature testing technology for large software systems [30, 35]: a test case is randomly generated, and output is compared for a variety of systems.”
Learning to Fuzz: Application-Independent Fuzz Testing with Probabilistic, Generative Models of Input Data (2016, background): “, when creating inputs for differential testing across multiple supposedly equivalent programs [33] or when fuzz testing remote web applications.”
NEZHA: Efficient Domain-Independent Differential Testing (2017, background): “Differential Testing: Differential testing [51] has been very successful in uncovering semantic differences between independent implementations with similar intended functionality.”
A Formally Verified Compiler Back-end (2009, background): “Despite intensive testing, bugs in compilers do occur, causing the compiler to crash at compile time or—much worse—to silently generate an incorrect executable for a correct source program [67,65,31].”
CRADLE: Cross-Backend Validation to Detect and Localize Bugs in Deep Learning Libraries (2019, background): “Differential Testing and Inconsistency Detection: Differential testing [56] consists of testing whether different compilers produce the same results.”