What Question Answering can Learn from Trivia Nerds

By Jordan L. Boyd-Graber. 2019

42 citations recorded by the source corpus.

In addition to the traditional task of machines answering questions, question answering (QA) research creates interesting, challenging questions that help systems how to answer questions and reveal the best systems. We argue that creating a QA dataset—and the ubiquitous leaderboard that goes with it—closely resembles running a trivia tournament: you write questions, have agents (either humans or machines) answer the questions, and declare a winner. However, the research community has ignored the hard-learned lessons from decades of the trivia community creating vibrant, fair, and effective question answering competitions. After detailing problems with existing QA datasets, we outline the key lessons—removing ambiguity, discriminating skill, and adjudicating disputes—that can transfer to QA research and how they might be implemented.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

In Question Answering & Reading Comprehension

Citations in the corpus

What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021, methodology): “While this can be helpful when building diagnostic datasets that focus on speciﬁc types of model failure (Cooper et al., 1996; Naik et al., 2018; Wang et al., 2019b), it is counterproductive when our goal is to build a broad-coverage benchmark dataset to set priorities and guide progress toward the…”
The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants (2023, methodology): “models that use shortcuts or pattern-match) (Boyd-Graber and Börschinger, 2020).”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, background): “The amount of evidence a human needs is a unique metric of how difficult a claim is for humans (although incremental evidence is recommended for question answering systems in Boyd-Graber and Börschinger (2020), to the best of our knowledge it has not been applied to entailment or validation).”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, background): “And leaderboards should also signal when they have outlived their usefulness (Boyd-Graber and Börschinger, 2020).”