WIQA: A dataset for “What if...” reasoning over procedural text

By Niket Tandon, Bhavana Dalvi, Keisuke Sakaguchi, Antoine Bosselut, Peter Clark. 2019

116 citations recorded by the source corpus.

We introduce WIQA, the first large-scale dataset of “What if...” questions over procedural text. WIQA contains a collection of paragraphs, each annotated with multiple influence graphs describing how one change affects another, and a large (40k) collection of “What if...?” multiple-choice questions derived from these. For example, given a paragraph about beach erosion, would stormy weather hasten or decelerate erosion? WIQA contains three kinds of questions: perturbations to steps mentioned in the paragraph; external (out-of-paragraph) perturbations requiring commonsense knowledge; and irrelevant (no effect) perturbations. We find that state-of-the-art models achieve 73.8% accuracy, well below the human performance of 96.3%. We analyze the challenges, in particular tracking chains of influences, and present the dataset as an open challenge to the community.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…cg/other Botha et al. 2018 wikisql cg/other Zhong et al. 2017 wino_grande qa/multiple-choice qa Sakaguchi et al. 2020 wiqa qa/multiple-choice qa Tandon et al. 2019 xsum cg/summarization Narayan et al. 2018 yahoo_answers_topics cls/topic (link) yelp_polarity cls/sentiment analysis Zhang et…”
Logic-Guided Data Augmentation and Regularization for Consistent Question Answering (2020, methodology): “Comparison-type questions (Tandon et al., 2019; Tafjord et al., 2019; Yang et al., 2018) ask about relationships between properties of entities or events such as cause-effect, qualitative or quantitative reasoning.”
QED: A Framework and Dataset for Explanations in Question Answering (2020, background): “…keeping with its modularity, we can extend QED to account for these by looking to semantic relations beyond referential equality and predicate entailment, such as set-membership noun phrase (Hearst, 1992) and interclausal (Miltsakaki et al., 2004; Lamm et al., 2018; Tandon et al., 2019) relations.”
Language Models of Code are Few-Shot Commonsense Learners (2022, background): “…a major remaining challenge is how to leverage LLMs for structured commonsense reasoning , including tasks such as generating event graphs (Tandon et al., 2019), reasoning graphs (Madaan et al., 2021a), scripts (Sakaguchi et al., 2021), and argument explanation graphs (Saha et al., 2021).”
GLUCOSE: GeneraLized and COntextualized Story Explanations (2020, background): “Recently, there has been a renewed interest in commonsense reasoning (Talmor et al., 2019; Tandon et al., 2019; Rashkin et al., 2018a; Zellers et al., 2018), further fostered by the increasing need for explainable AI systems (Yang et al.”
Think about it! Improving defeasible reasoning by first modeling the question scenario. (2021, methodology): “Inference graphs were introduced in philosophy by Pollock (2009) to aid defeasible reasoning for humans, and in NLP by Tandon et al. (2019) for a counterfactual reasoning task.”
Thinking Like a Skeptic: Defeasible Inference in Natural Language (2020, methodology): “Finally, joint modeling between defeasible inference and related reasoning tasks such as abductive reasoning (Peirce, 1960; Bhagavatula et al., 2019) and coun-terfactual reasoning (Goodman, 1947; Qin et al., 2019; Tandon et al., 2019) is a potentially fruitful line of inquiry.”
Can Language Models Solve Graph Problems in Natural Language? (2023, background): “In structured commonsense reasoning, where LLMs are given a natural language input and then asked to generate a graph such as an event graph [Tandon et al., 2019], a reasoning-graph [Madaan et al.”