Tracking State Changes in Procedural Text: a Challenge Dataset and Models for Process Paragraph Comprehension

By Bhavana Dalvi, Lifu Huang, Niket Tandon, Wen-tau Yih, Peter Clark. 2018

143 citations recorded by the source corpus.

We present a new dataset and models for comprehending paragraphs about processes (e.g., photosynthesis), an important genre of text describing a dynamic world. The new dataset, ProPara, is the first to contain natural (rather than machine-generated) text about a changing world along with a full annotation of entity states (location and existence) during those changes (81k datapoints). The end-task, tracking the location and existence of entities through the text, is challenging because the causal effects of actions are often implicit and need to be inferred. We find that previous models that have worked well on synthetic data achieve only mediocre performance on ProPara, and introduce two new neural models that exploit alternative mechanisms for state prediction, in particular using LSTM input encoding and span prediction. The new models improve accuracy by up to 19%. We are releasing the ProPara dataset and our models to the community.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “, 2019), tracking entity state changes (Mishra et al., 2018; Ostermann et al., 2018) or a particular kind of “multi-step” reasoning over multiple documents (Welbl et al.”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “, 2016), MCScript,4 and ProPara (Mishra et al., 2018) do require additional domain knowledge about everyday events, scripts, and processes, respectively.”
QASC: A Dataset for Question Answering via Sentence Composition (2019, background): “However, MultiRC and other singlepassage datasets (Mishra et al. 2018; Weston et al. 2015) have a stronger emphasis on passage discourse and entity tracking, rather than relation composition.”
What Makes Reading Comprehension Questions Easier? (2018, neutral): “Other datasets that are not covered in our study, but can be analyzed using the same method, include: QA4MRE (Sutcliffe et al., 2013), CNN/Daily Mail (Hermann et al., 2015), Children’s Book Test (Hill et al., 2016), bAbI (Weston et al., 2015), WikiReading (Hewlett et al., 2016), LAMBADA (Paperno et ”
A Survey of Large Language Models (2023, unknown): “Complex Reasoning Knowledge Reasoning CSQA [247], StrategyQA [248], ARC [291], BoolQ [315], PIQA [295], SIQA [316], HellaSwag [317], WinoGrande [318], OpenBookQA [302], COPA [319], ScienceQA [301], proScript [320], ProPara [321], ExplaGraphs [322], ProofWriter [323], EntailmentBank [324], ProOntoQA ”
Commonsense Reasoning for Natural Language Understanding: A Survey of Benchmarks, Resources, and Approaches (2019, background): “…for tasks which require comprehending passages with several state changes or supporting facts, such as bAbI (Weston et al., 2016) or ProPara (Mishra et al., 2018), involves augmenting systems with a dynamic memory which may be maintained over time to represent the changing state of the…”
Video2Commonsense: Generating Commonsense Descriptions to Enrich Video Captioning (2020, background): “On the other hand, challenges such as ProPara (Mishra et al., 2018) and bAbI (Weston et al.”
WIQA: A dataset for “What if...” reasoning over procedural text (2019, background): “While recent systems for procedural text comprehension can answer questions about what events happen, e.g., (Bosselut et al., 2018; Hena ﬀ et al., 2017; Dalvi et al., 2018), the extent to which they understand the inﬂuences between those events remains unclear.”
Language Models of Code are Few-Shot Commonsense Learners (2022, methodology): “We use the PROPARA dataset (Dalvi et al., 2018) as the test-bed for this task.”