Aligning to Social Norms and Values in Interactive Narratives

By Prithviraj Ammanabrolu, Liwei Jiang, Maarten Sap, Hannaneh Hajishirzi, Yejin Choi. 2022

54 citations recorded by the source corpus.

We focus on creating agents that act in alignment with socially beneficial norms and values in interactive narratives or text-based games—environments wherein an agent perceives and interacts with a world through natural language. Such interactive agents are often trained via reinforcement learning to optimize task performance, even when such rewards may lead to agent behaviors that violate societal norms—causing harm either to the agent itself or other entities in the environment. Social value alignment refers to creating agents whose behaviors conform to expected moral and social norms for a given context and group of people—in our case, it means agents that behave in a manner that is less harmful and more beneficial for themselves and others.We build on the Jiminy Cricket benchmark (Hendrycks et al. 2021), a set of 25 annotated interactive narratives containing thousands of morally salient scenarios covering everything from theft and bodily harm to altruism. We introduce the GALAD (Game-value ALignment through Action Distillation) agent that uses the social commonsense knowledge present in specially trained language models to contextually restrict its action space to only those actions that are aligned with socially beneficial values. An experimental study shows that the GALAD agent makes decisions efficiently enough to improve state-of-the-art task performance by 4% while reducing the frequency of socially harmful behaviors by 25% compared to strong contemporary value alignment approaches.

Subfields: Artificial Intelligence, General Social Sciences, Safety Research, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Safety Research

In Natural Language Processing

In Alignment & RLHF

Citations in the corpus

Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, methodology): “Bahdanau et al. (2016) proposed an actor-critic algorithm to address the variance/large action space problems when using REINFORCE for language generation; follow-up works such as KG-A2C (Ammanabrolu & Hausknecht, 2020), TrufLL (Martin et al., 2022), AE-DQN (Zahavy et al., 2018), and GALAD (Ammanabr”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…learning: up-dating models trained on static text using super-vision either from humans (Stiennon et al., 2020; Ouyang et al., 2022; Scheurer et al., 2022) or from proxies for human behavior or social environments (Ammanabrolu et al., 2022a,b) based on broad coverage LLMs (Perez et al., 2022).”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “Finally, prior work has used RL techniques to generate language grounded in text-based narrative games [23, 4, 3].”
Reinforcement Learning for Language Models (2023, background): “Since then, there have also been follow-up papers that attempt to eliminate actions [21] or generate small candidate action sets [22] for more efficient and stable exploration.”