Question Answering is a Format; When is it Useful?

By Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, Sewon Min. 2019

55 citations recorded by the source corpus.

Recent years have seen a dramatic expansion of tasks and datasets posed as question answering, from reading comprehension, semantic role labeling, and even machine translation, to image and video understanding. With this expansion, there are many differing views on the utility and definition of "question answering" itself. Some argue that its scope should be narrow, or broad, or that it is overused in datasets today. In this opinion piece, we argue that question answering should be considered a format which is sometimes useful for studying particular phenomena, not a phenomenon or task in itself. We discuss when a task is correctly described as question answering, and when a task is usefully posed as question answering, instead of using some other format.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, background): “The question answering format encompasses a wide variety of tasks (Gardner et al., 2019) ranging”
oLMpics-On What Language Model Pre-training Captures (2019, methodology): “In MC-QA, we phrase the task as a question, letting answer candidates be arbitrary strings, which provides ample expressivity (Gardner et al., 2019b) and facilitates probing question involving complex and commonsense reasoning (Talmor et al., 2019; Gardner et al., 2019a; Talmor and Berant, 2018).”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “Formatting new instances as QA pairs has been shown to be useful for transfer learning from a QA model (Gardner et al., 2019).”
FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization (2020, background): “Question answering is a broad format that subsumes many tasks (Gardner et al., 2019).”
TORQUE: A Reading Comprehension Dataset of Temporal Ordering Questions (2020, background): “However, this setup also increases the risk of leading to trivial solutions (Gardner et al., 2019a).”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, background): “and a wide range of NLP tasks can be cast as QA problems (Levy et al., 2017; McCann et al., 2018; Gardner et al., 2019).”
Learning to Solve Complex Tasks by Talking to Agents (2021, methodology): “We focus on the QA task and use QA agents since the question-answer format can capture a broad range of tasks while also sur-facing the capability of the agent (Gardner et al., 2019).”
CONDAQA: A Contrastive Reading Comprehension Dataset for Reasoning about Negation (2022, unknown): “It is the ﬁrst English reading-comprehension dataset that targets how models process negated statements in paragraphs (Gardner et al., 2019b”
Description Based Text Classification with Reinforcement Learning (2020, methodology): “Inspired by the current trend of formalizing NLP problems as question answering tasks (Levy et al., 2017; McCann et al., 2018; Li et al., 2019a,b; Gardner et al., 2019; Raffel et al., 2019), we pro-pose a new framework for text classiﬁcation by formalizing it as a SQuAD-style machine reading…”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, background): “Although extractive question answering is a flexible format for evaluating modeling approaches on a wide range of phenomena (Gardner et al., 2019; Fisch et al., 2019), focusing on a different benchmark or task (e.g., machine translation on WMT 2014 EN-DE) may significantly change the set of…”