What Would Jiminy Cricket Do? Towards Agents That Behave Morally

By Dan Hendrycks, Mantas Mazeika, Andy Zou, Sahil Patel, Christine Zhu, Jesus Navarro, Dawn Song, Bo Li, Jacob Steinhardt. 2021

76 citations recorded by the source corpus.

When making everyday decisions, people are guided by their conscience, an internal sense of right and wrong. By contrast, artificial agents are currently not endowed with a moral sense. As a consequence, they may learn to behave immorally when trained on environments that ignore moral concerns, such as violent video games. With the advent of generally capable agents that pretrain on many environments, it will become necessary to mitigate inherited biases from environments that teach immoral behavior. To facilitate the development of agents that avoid causing wanton harm, we introduce Jiminy Cricket, an environment suite of 25 text-based adventure games with thousands of diverse, morally salient scenarios. By annotating every possible game state, the Jiminy Cricket environments robustly evaluate whether agents can act morally while maximizing reward. Using models with commonsense moral knowledge, we create an elementary artificial conscience that assesses and guides agents. In extensive experiments, we find that the artificial conscience approach can steer agents towards moral behavior without sacrificing performance.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (2022, methodology): “Similar policy shaping has recently been used to make RL agents more ethical ( Hendrycks et al., 2021b).”
Unsolved Problems in ML Safety (2021, background): “…not pursue undesirable secondary objectives, researchers could try to construct systems that guide models not just to follow rewards but also behave morally [83]; such systems could also be effective at guiding agents not to cause wanton harm within interactive environments and to abide by rules.”
Quark: Controllable Text Generation with Reinforced Unlearning (2022, methodology): “RL has also been used to train language models to align with models of human preferences and values [90, 24, 3].”
Aligning to Social Norms and Values in Interactive Narratives (2022, methodology): “…suggests that the more fine-grained annotations in the benchmark relating to severity might be prone to being more noisy, we modify our metrics from Hendrycks et al. (2021b) and do not consider labels associated with the relative severity of harm and only consider the more high-level annotations…”