Reasoning Over Paragraph Effects in Situations

By Kevin Lin, Oyvind Tafjord, Peter Clark, Matt Gardner. 2019

125 citations recorded by the source corpus.

A key component of successfully reading a passage of text is the ability to apply knowledge gained from the passage to a new situation. In order to facilitate progress on this kind of reading, we present ROPES, a challenging benchmark for reading comprehension targeting Reasoning Over Paragraph Effects in Situations. We target expository language describing causes and effects (e.g., “animal pollinators increase efficiency of fertilization in flowers”), as they have clear implications for new situations. A system is presented a background passage containing at least one of these relations, a novel situation that uses this background, and questions that require reasoning about effects of the relationships in the background passage in the context of the situation. We collect background passages from science textbooks and Wikipedia that contain such phenomena, and ask crowd workers to author situations, questions, and answers, resulting in a 14,322 question dataset. We analyze the challenges of this task and evaluate the performance of state-of-the-art reading comprehension models. The best model performs only slightly better than randomly guessing an answer of the correct type, at 61.6% F1, well below the human performance of 89.0%.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Cross-Task Generalization via Natural Language Crowdsourcing Instructions (2021, background): “, 2019), ROPES (Lin et al., 2019) and Winogrande (Sakaguchi et al.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “Our work joins a line of previously introduced benchmarks targeting commonsense reasoning 365 [12, 13, 29, 15] as well as probing for other reasoning abilities [32, 33, 34, 35, 36, 37, 38].”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “Recently many reading comprehension datasets requiring complex and compositional reasoning over text have been introduced, including HotpotQA (Yang et al., 2018), DROP (Dua et al., 2019), Quoref (Dasigi et al., 2019), and ROPES (Lin et al., 2019).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, neutral): “They cover a diverse range of topics, including events, careers, biographies, and writing tasks such as reviews, summaries and essays – and this is just looking at the 11% of our LongForm-C dataset with reverse instructions (RI).”
CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP (2021, background): “…al. 2017 reddit_tifu-title cg/summarization Kim et al. 2019 reddit_tifu-tldr cg/summarization Kim et al. 2019 ropes qa/machine reading comprehension Lin et al. 2019 rotten_tomatoes cls/sentiment analysis Pang and Lee 2005 samsum cg/summarization Gliwa et al. 2019 scicite cls/other Cohan et al.…”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, background): “Selsam et al. (2019), Tafjord et al. (2019), Lin et al. (2019), and Khashabi et al. (2020) designed their data collection process to include contrastive examples.”
Evaluating NLP Models via Contrast Sets (2020, methodology): “We create contrast sets for 10 NLP datasets (full descriptions are provided in Section A): • NLVR2 (Suhr et al., 2019) • IMDb sentiment analysis (Maas et al., 2011) • MATRES Temporal RE (Ning et al., 2018) • English UD parsing (Nivre et al., 2016) • PERSPECTRUM (Chen et al., 2019) • DROP (Dua et al.”
PRover: Proof Generation for Interpretable Reasoning over Rules (2020, methodology): “, 2019) and ROPES (Lin et al., 2019).”
Think you have Solved Direct-Answer Question Answering? Try ARC-DA, the Direct-Answer AI2 Reasoning Challenge (2021, background): “, 2019), and ROPES (Lin et al., 2019), are crowdsourced, and thus tend to explore a single, specific style of reasoning in a controlled setting.”
Critical Thinking for Language Models (2020, background): “• ROPES (Lin et al., 2019) is a reading comprehension task that involves applying background knowledge to a given situation (both being presented as paragraph long text).”
On Making Reading Comprehension More Comprehensive (2019, background): “ROPES (Lin et al., 2019) borrowed the idea of “minimal pairs” from linguistic analysis in its construction.”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “T e r m s [ 110 ] M C T A C O [ 302 ] , M u l t i R C [ 109 ] , Q A S C [ 111 ] Q u o r e f [ 46 ] , R O P E S [ 127 ] , W i n o g r a n d e [ 202 ] [ 161 ] c o d e I M D B [ 149 ] C M A C o n t e n t S o c i a l M e d i a A R C E a s y a n d C h a ll e n g e [ 41 ] , R a c e [ 115 ] , P I Q A [ 18…”
UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training (2022, background): “The past few years has seen a lot of activity toward unified model designs (Friedman et al., 2021; Aghajanyan et al., 2021; Gupta et al., 2021; Tafjord and Clark, 2021; Lourie et al., 2021; Jiang et al., 2021; Aribandi et al., 2021; Zhong et al., 2021a; Chen et al., 2021; Zhong et al., 2022).”
Multi-Step Inference for Reasoning over Paragraphs (2020, methodology): “Training Following the settings of prior work (Lin et al., 2019), we fine-tune the ROBERTALARGE pre-trained transformer.”
Polyglot Prompt: Multilingual Multitask PrompTraining (2022, methodology): “We adopt SQuAD 2.0 (Rajpurkar et al., 2016), Quoref (Dasigi et al., 2019), NewsQA (Trischler et al., 2017), and ROPES (Lin et al., 2019).”