Counterfactual Story Reasoning and Generation

By Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, Yejin Choi. 2019

168 citations recorded by the source corpus.

Counterfactual reasoning requires predicting how alternative events, contrary to what actually happened, might have resulted in different outcomes. Despite being considered a necessary component of AI-complete systems, few resources have been developed for evaluating counterfactual reasoning in narratives. In this paper, we propose Counterfactual Story Rewriting: given an original story and an intervening counterfactual event, the task is to minimally revise the story to make it compatible with the given counterfactual event. Solving this task will require deep understanding of causal narrative chains and counterfactual invariance, and integration of such story reasoning capabilities into conditional language generation models. We present TIMETRAVEL, a new dataset of 29,849 counterfactual rewritings, each with the original story, a counterfactual event, and human-generated revision of the original story compatible with the counterfactual event. Additionally, we include 81,407 counterfactual “branches” without a rewritten storyline to support future work on semi- or un-supervised approaches to counterfactual story rewriting. Finally, we evaluate the counterfactual rewriting capacities of several competitive baselines based on pretrained language models, and assess whether common overlap and model-based automatic metrics for text generation correlate well with human scores for counterfactual rewriting.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Prompting & Reasoning

Citations in the corpus

Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “…resources have also been invested in developing much larger and more natural crowdsourced benchmarks such as natural language inference (NLI) tasks (Bowman et al., 2015; Williams, Nangia, and Bowman, 2018; Nie et al., 2020) and commonsense reasoning tasks (Zellers et al., 2019; Qin et al., 2019a).”
Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “that change the groundtruth labels by manually rewriting instances (Gardner et al., 2020; Qin et al., 2019) or defining perturbation functions (Ribeiro et al.”
Don’t Say That! Making Inconsistent Dialogue Unlikely with Unlikelihood Training (2019, methodology): “There is a raft of recent large-scale, high quality data that can be massaged into this form, from natural language inference (NLI) tasks (Bowman et al., 2015; Williams et al., 2018; Welleck et al., 2019b) to commonsense reasoning tasks (Zellers et al., 2019; Qin et al., 2019).”
Deep Learning for Text Style Transfer: A Survey (2020, background): “Qin et al. (2019) propose the first dataset, each sample of which takes an originally five-sentence story, and changes the event in the second sentence to a new, counterfactual event.”
Inverse Scaling: When Bigger Isn't Better (2023, background): “The task provides LMs with a short description of a setting loosely derived from Qin et al. (2019), along with a question about the setting that requires more information in order to be answered.”
InFillmore: Frame-Guided Language Generation with Bidirectional Context (2021, background): “Qin et al. (2019) introduce the task of generative counterfactual reasoning in narratives.”
Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision (2020, background): “Despite being a fundamental mode of human reasoning, modern NLP research paid little attention to non-monotonic reasoning (e.g. Qin et al. 2019; Bhagavatula et al. 2019).”
Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning (2020, methodology): “The associated T IME T RAVEL dataset is based on ﬁctional narratives from ROCStories, for which counterfactual contexts and alternative endings are crowdsourced, yielding 29,849 problem instances.”
Learning to Imagine: Integrating Counterfactual Thinking in Neural Discrete Reasoning (2022, background): “Additionally, in commonsense reasoning, counterfactual samples are also utilized through hyperbole generation (Tian et al., 2021), story generation (Qin et al., 2019) and commonsense QA(Huang et al., 2019), which is also a related yet different strand of research.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, background): “One line of work concerns counterfactuals to a certain event or situation that is still licensed in a default world model (Qin et al., 2019, 2020; Yang et al., 2020; Frohberg and Binder, 2022; i.a. ), in contrast to our counterfactual world states that deviate from the default.”
STORIUM: A Dataset and Evaluation Platform for Machine-in-the-Loop Story Generation (2020, methodology): “The sampling parameters, such as the k in top-k sampling, can signiﬁcantly affect output quality of story generation models (See et al., 2019), so we choose values that worked well in prior work (Qin et al., 2019).”
From LSAT: The Progress and Challenges of Complex Reasoning (2021, background): “For example, we should dive into abductive reasoning [110] and counterfactual reasoning [111, 112] to research cause-and-effect complex reasoning.”
Conversational Neuro-Symbolic Commonsense Reasoning (2020, background): “2020), where the task is to choose a plausible outcome, cause or explanation for an input scenario; and the TimeTravel benchmark (Qin et al. 2019) where the task to revise a story to make it compatible with a given counterfactual event.”
Thinking Like a Skeptic: Defeasible Inference in Natural Language (2020, methodology): “Finally, joint modeling between defeasible inference and related reasoning tasks such as abductive reasoning (Peirce, 1960; Bhagavatula et al., 2019) and coun-terfactual reasoning (Goodman, 1947; Qin et al., 2019; Tandon et al., 2019) is a potentially fruitful line of inquiry.”
Asking without Telling: Exploring Latent Ontologies in Contextual Representations (2020, background): “Recent work also advocates for counterfactual learning and evaluation (Qin et al., 2019; Kaushik et al., 2020) to mitigate confounds, or contrastive evaluation sets (Gardner et al., 2020) to rigorously test local decision boundaries.”
Conversational Multi-Hop Reasoning with Neural Commonsense Knowledge and Symbolic Logic Rules (2021, result): “In contrast to this challenging task and TimeTravel (Qin et al., 2019), most commonsense reasoning benchmarks have traditionally been designed in a multiple choice manner.”
COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamics (2022, unknown): “, 2019), and counterfactual story generation (Qin et al., 2019).”