Moral Stories: Situated Reasoning about Norms, Intents, Actions, and their Consequences

By Denis Emelin, Ronan Le Bras, Jena D. Hwang, Maxwell Forbes, Yejin Choi. 2020

183 citations recorded by the source corpus.

In social settings, much of human behavior is governed by unspoken rules of conduct rooted in societal norms. For artificial systems to be fully integrated into social environments, adherence to such norms is a central prerequisite. To investigate whether language generation models can serve as behavioral priors for systems deployed in social settings, we evaluate their ability to generate action descriptions that achieve predefined goals under normative constraints. Moreover, we examine if models can anticipate likely consequences of actions that either observe or violate known norms, or explain why certain actions are preferable by generating relevant norm hypotheses. For this purpose, we introduce Moral Stories, a crowd-sourced dataset of structured, branching narratives for the study of grounded, goal-oriented social reasoning. Finally, we propose decoding strategies that combine multiple expert models to significantly improve the quality of generated actions, consequences, and norms compared to strong baselines.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “The manual collection relies on human annotators to compose new samples or label existing data (Forbes et al., 2020; Hendrycks et al., 2020; Sap et al., 2020; Lourie et al., 2021; Emelin et al., 2021).”
Ethical-Advice Taker: Do Language Models Understand Natural Language Interventions? (2021, background): “A range of recent works are based on the general idea of models revising their behavior according to changes in their input (Wallace et al., 2019; Gardner et al., 2020; Emelin et al., 2020; Ye and Ren, 2021; Schick and Schütze, 2020; Sheng et al., 2020).”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, background): “The Moral Stories dataset (N = 20, 000) examines whether LMs can generate moral responses under diverse social situations (Emelin et al., 2021).”
DREAM: Uncovering Mental Models behind Language Models (2021, methodology): “…a training dataset for learning mental models: 1) Story commonsense (Rashkin et al., 2018) 2) Social Chemistry (Forbes et al., 2020) 3) Moral stories (Emelin et al., 2020) Statistics about these data sources, and which di-mension(s) they contribute to the training data, are shown in Table 2.”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “While other tasks and datasets (Emelin et al., 2021; Mathew et al., 2021) are also possible choices, we leave them for future work.”
Robust Preference Learning for Storytelling via Contrastive Reinforcement Learning (2022, methodology): “We also test our model on the Moral Stories dataset (Emelin et al., 2021), to demonstrate our”
The Moral Integrity Corpus: A Benchmark for Ethical Dialogue Systems (2022, unknown): “Such metadata is not present in other popular moral datasets (Hendrycks et al., 2020; Lourie et al., 2021; Forbes et al., 2020; Emelin et al., 2020), but this metadata is critical for understanding the variability of moral intuitions (Talat et al.”
On the Machine Learning of Ethical Judgments from Natural Language (2022, background): “M ORAL S TORIES uses S OCIAL -”
Aligning to Social Norms and Values in Interactive Narratives (2022, methodology): “, 2021a), MORAL STORIES (Emelin et al., 2021), SOCIAL BIAS INFERENCE CORPUS (Sap et al.”