Citation Needed: A Taxonomy and Algorithmic Assessment of Wikipedia's Verifiability

By Miriam Redi, Besnik Fetahu, Jonathan T. Morgan, Dario Taraborelli. 2019

53 citations recorded by the source corpus.

Wikipedia is playing an increasingly central role on the web, and the policies its contributors follow when sourcing and fact-checking content affect million of readers. Among these core guiding principles, verifiability policies have a particularly important role. Verifiability requires that information included in a Wikipedia article be corroborated against reliable secondary sources. Because of the manual labor needed to curate Wikipedia at scale, however, its contents do not always evenly comply with these policies. Citations (i.e. reference to external sources) may not conform to verifiability requirements or may be missing altogether, potentially weakening the reliability of specific topic areas of the free encyclopedia. In this paper, we aim to provide an empirical characterization of the reasons why and how Wikipedia cites external sources to comply with its own verifiability guidelines. First, we construct a taxonomy of reasons why inline citations are required, by collecting labeled data from editors of multiple Wikipedia language editions. We then crowdsource a large-scale dataset of Wikipedia sentences annotated with categories derived from this taxonomy. Finally, we design algorithmic models to determine if a statement requires a citation, and to predict the citation reason . We evaluate the accuracy of such models across different classes of Wikipedia articles of varying quality, and on external datasets of claims annotated for fact-checking purposes.

Subfields: Artificial Intelligence, Communication, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Communication

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Improving Wikipedia verifiability with AI (2022, background): “Text-based classiﬁers able to detect claim needing citations (Redi et al., 2019; Chou et al., 2020) have received a lot of attention from both the scientiﬁc and the Wikimedia communities.”
Citation Detective : a Public Dataset to Improve and Quantify Wikipedia Citation Quality at Scale (2020, methodology): “A recent research work aimed to fill this gap by designing machine learning classifiers able to detect sentences needing citations in Wikipedia [7]: through a qualitative analysis of the citation guidelines in Wikipedia, the authors created a taxonomy of reasons why inline citations are required in…”
Quantifying Engagement with Citations on Wikipedia (2020, unknown): “, by finding potential citations [15] and Wikipedia statements in need of evidence [48].”