Aligning AI With Shared Human Values
By Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Zheng Li, Dawn Song, Jacob Steinhardt. 2020
970 citations recorded by the source corpus.
We show how to assess a language model's knowledge of basic concepts of morality. We introduce the ETHICS dataset, a new benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality. Models predict widespread moral judgments about diverse text scenarios. This requires connecting physical and social world knowledge to value judgements, a capability that may enable us to steer chatbot outputs or eventually regularize open-ended reinforcement learning agents. With the ETHICS dataset, we find that current language models have a promising but incomplete understanding of basic ethical knowledge. Our work shows that progress can be made on machine ethics today, and it provides a steppingstone toward AI that is aligned with human values.
Subfields: Artificial Intelligence, Safety Research, Alignment & RLHF, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Percy Liang
- Yejin Choi
- Jianfeng Gao
- Noah A. Smith
- Graham Neubig
- Hannaneh Hajishirzi
In Safety Research
- Richard Ngo
- Yejin Choi
- William S. Isaac
- Iason Gabriel
- Amanda Askell
- Dawn Song
- Bo Li
- Ruslan Salakhutdinov
In Alignment & RLHF
- Geoffrey Irving
- Richard Ngo
- Jacob Steinhardt
- John Schulman
- Yejin Choi
- Iason Gabriel
- Amanda Askell
- Daniel Klein
In Safety, Robustness & Interpretability
- Jacob Steinhardt
- Percy Liang
- Dawn Song
- Dan Hendrycks
- Nicholas Carlini
- Bo Li
- Adina Williams
- Yulia Tsvetkov
Citations in the corpus
- Measuring Massive Multitask Language Understanding (2020, background): “It also covers moral scenarios, including questions from the ETHICS dataset (Hendrycks et al., 2020) that test a As Seller, an encyclopedia salesman, approached the grounds on which Hermit's house was situated, he saw a sign that said, "No salesmen.”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “The manual collection relies on human annotators to compose new samples or label existing data (Forbes et al., 2020; Hendrycks et al., 2020; Sap et al., 2020; Lourie et al., 2021; Emelin et al., 2021).”
- Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “These existing solutions tackle content-related harms, but future work in reducing discrimination and toxicity in LMs may also benefit from KBs that encode social (Chang et al., 2020), cultural, (Hershcovich et al., 2022), and moral norms (Hendrycks et al., 2021; Jiang et al., 2021).”
- A General Language Assistant as a Laboratory for Alignment (2021, methodology): “The Ethics [HBB + 21] evaluations are mostly binary classification problems, and so naturally belong in our binary category, except for Utilitarianism which compares relative ‘pleasantness’ of scenarios.”
- Towards Understanding and Mitigating Social Biases in Language Models (2021, background): “, 2019) as well as other notions of human-aligned values such as ethics (Hendrycks et al., 2021), social bias implications (Sap et al.”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Note that we do not evaluate the full agent with evidence, as this introduces several issues, most notably that MMLU questions are sourced from the internet, and so a naive Google search will return many questions verbatim.”
- Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections (2021, background): “ETHICS Deontology (Hendrycks et al., 2020).”
- Measuring Coding Challenge Competence With APPS (2021, methodology): “If a model were to perform well on APPS, this would indicate an ability to flexibly use data structures and programming techniques, as well as an ability to correctly interpret diverse task specifications, follow instructions, and understand human intent (Hendrycks et al., 2021a).”
- Unsolved Problems in ML Safety (2021, background): “Researchers could create models that learn wellbeing functions that do not mimic cognitive biases [76].”
- Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities (2021, background): “Formal frameworks for measuring and ensuring fairness (Hendrycks et al. 2020a,b; Sheng et al. 2020) in language are still in nascent stages of development.”
- Taxonomy of Risks posed by Language Models (2022, background): “Several LMs failed to reliably distinguish between ethical or unethical actions, indicating they may advise unethical behaviours [72].”
- Large pre-trained language models contain human-like biases of what is right and wrong to do (2021, neutral): “[44] introduced a Commonsense Morality dataset.”
- SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022, neutral): “The ETHICS dataset contains defined categories of ethics issues spanning justice, well-being, duties, virtues, and commonsense morality (Hendrycks et al., 2021).”
- On Transferability of Prompt Tuning for Natural Language Understanding (2021, methodology): “, 2021) and justice (Hendrycks et al., 2021); (4) Paraphrase identification (PI), including QQP (Sharma et al.”
- Connecting the Dots in Trustworthy Artificial Intelligence: From AI Principles, Ethics, and Key Requirements to Responsible AI Systems and Regulation (2023, background): “Although giving open-world inputs to models can partially tell apart pleasant and unpleasant states, utility values of such states are no ground truth values, and are a result of the model’s own learned utility function [41].”
- Few-shot Learning with Multilingual Language Models (2021, background): “More-over, we contend that studies on other tasks such as stereotype (Nangia et al., 2020; Nadeem et al., 2021), ethics (Hendrycks et al., 2020) would provide a more comprehensive view of model behavior for social value tasks.”