Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering

By Najoung Kim, Ellie Pavlick, Burcu Karagol Ayan, Deepak Ramachandran. 2021

56 citations recorded by the source corpus.

Many Question-Answering (QA) datasets contain unanswerable questions, but their treatment in QA systems remains primitive. Our analysis of the Natural Questions (Kwiatkowski et al. 2019) dataset reveals that a substantial portion of unanswerable questions (~21%) can be explained based on the presence of unverifiable presuppositions. Through a user preference study, we demonstrate that the oracle behavior of our proposed system—which provides responses based on presupposition failure—is preferred over the oracle behavior of existing QA systems. Then, we present a novel framework for implementing such a system in three steps: presupposition generation, presupposition verification, and explanation generation, reporting progress on each. Finally, we show that a simple modification of adding presuppositions and their verifiability to the input of a competitive end-to-end QA system yields modest gains in QA performance and unanswerability detection, demonstrating the promise of our approach.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Question Answering & Reading Comprehension

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Question Answering & Reading Comprehension

Citations in the corpus

Few-Shot Self-Rationalization with Natural Language Prompts (2021, background): “For instance, constraining a model that answers the question “Which linguist invented the lightbulb?” with “none” to also provide the reason—“Thomas Edison is the inventor of the lightbulb and he was not a linguist”— makes the model easier to control and interact with (Kim et al., 2021).”
How Language Model Hallucinations Can Snowball (2023, background): “; Kim et al., 2021, 2022) or that are otherwise misleading (e.g., “Who really caused 9/11?””
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, background): “We limit our study to valid questions, ignoring presupposition cases (Kim et al., 2021) (e.”
Can NLI Models Verify QA Systems' Predictions? (2021, background): “Adding a presup-position checking stage to the question converter could further improve its performance (Kim et al., 2021).”