Are mutants a valid substitute for real faults in software testing?

662 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Evaluating and Improving Fault Localization (2017, background): “Artiﬁcial faults such as mutants differ from real faults in many respects, including their size, their distribution in code, and their difﬁculty of being detected by tests [22].”
Taxonomy of Real Faults in Deep Learning Systems (2019, background): “Whether mutants are a valid substitute for real faults has been an ongoing debate for traditional software [11, 14, 18].”
Chapter Six - Mutation Testing Advances: An Analysis and Survey (2019, background): “[43] ’14 statement, mutation Both mutation and statement coverage correlate with fault detection, with mutants having higher correlation.”
Coverage is not strongly correlated with test suite effectiveness (2014, background): “Previous work has also shown that if a test suite detects a large number of simple faults, caused by a single incorrect line of source code, it will detect a large number of harder, multi-line faults [28,32].”
Defects4J: a database of existing faults to enable controlled testing studies for Java programs (2014, background): “However, mutants and hand-seeded faults differ from inadvertently introduced bugs, and thus might not be suitable for evaluating testing techniques [6].”