Artificial Intelligence, Values, and Alignment

By Iason Gabriel. 2020

1,067 citations recorded by the source corpus.

This paper looks at philosophical questions that arise in the context of AI alignment. It defends three propositions. First, normative and technical aspects of the AI alignment problem are interrelated, creating space for productive engagement between people working in both domains. Second, it is important to be clear about the goal of alignment. There are significant differences between AI that aligns with instructions, intentions, revealed preferences, ideal preferences, interests and values. A principle-based approach to AI alignment, which combines these elements in a systematic way, has considerable advantages in this context. Third, the central challenge for theorists is not to identify ‘true’ moral principles for AI; rather, it is to identify fair principles for alignment that receive reflective endorsement despite widespread variation in people’s moral beliefs. The final part of the paper explores three ways in which fair principles for AI alignment could potentially be identified.

Subfields: Artificial Intelligence, Health Informatics, Safety Research, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “This refers to the process of ensuring that AI systems can not only successfully optimize the provided surrogate training objectives, but also that their predictions are in line with their intended purpose and adhere to ethical and safety standards provided by humans [4, 5].”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “More work should look into such value alignment challenges for fairness interventions (Gabriel, 2020).”
Alignment of Language Agents (2021, background): “Gabriel (2020) questions the ‘simple thesis’ that it’s possible to work on the technical challenge separately to the normative challenge, drawing on what we currently know about the field of machine learning (ML).”
Foundation Models and Fair Use (2023, background): “AI values alignment might make an agent more in line with certain moral values (Gabriel, 2020).”
Unsolved Problems in ML Safety (2021, background): “There is a large ecosystem of work on addressing societal consequences of machine learning, including AI policy [42], privacy [3, 162], fairness [72], and ethics [56].”
Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence (2020, background): “that were posed in the opening of whose values and goals are represented, and who is empowered to articulate and embed these values—introducing discussions of utilitarian, Kantian and volitional views on behaviour, and on the prevention and avoidance of undesirable and unintended consequences (Gabri”
RL with KL penalties is better viewed as Bayesian inference (2022, background): “5.3-5.3) for an extended discussion and (Gabriel, 2020) for a philosophical examination of the notion of ground truth human preferences.”
The Challenge of Value Alignment: from Fairer Algorithms to AI Safety (2021, background): “Moreover, important normative questions still need to be addressed (Gabriel, 2020).”
Defining and Characterizing Reward Hacking (2022, neutral): “In reinforcement learning, the goal of maximizing (some) reward function is often taken for granted, but a number of authors have expressed reservations about this approach (Gabriel, 2020; Dobbe et al., 2021; Hadfield-Menell et al., 2016b, 2017; Bostrom, 2014).”
SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, positive): “Indeed, the propensity of LLMs to reproduce this objectionable content has invigorated the field of AI alignment [2–4], wherein various mechanisms are used to “align” the output text generated by LLMs with ethical and legal standards [5–7].”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, neutral): “However, such applications may require language models to make subjective judgments that could vary significantly across different social groups (Solaiman & Dennison, 2021; Bender et al., 2021; Johnson et al., 2022; Goyal et al., 2022; Sap et al., 2022).”
Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision (2023, background): “AI Alignment The domain of AI alignment [12] has garnered substantial attention in recent years, with LLMs exhibiting remarkable proficiencies across a wide array of tasks.”
The Ghost in the Machine has an American accent: value conflict in GPT-3 (2022, background): “There is room for increased methodological diversity into the human alignment in AI problem using diverse sociocultural, philosophical, and linguistic perspectives[44, 45], notably in a global pluralist setting[11].”
Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, positive): “…alignment-based fine-tuning methods are being developed to align LLMs with human expectations better, incorporating not just basic standards (e.g., the 3H Standard (Askell et al., 2021)) but also ethical and human-value considerations (Irving et al., 2018; Gabriel, 2020; Casper et al., 2023).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, background): “AI alignment is a field that aims to ensure that AI systems adhere to their intended goals, interests, and values, as envisioned by their designers [2, 25, 63, 20, 24, 43, 7].”
Trust in AI: progress, challenges, and future directions (2024, neutral): “…others worry about the so-called “ alignment problem ” of how we, as a society, and researchers working on arti ﬁ cial intelligence research could ensure that an AGI ’ s interests and values would align with our own (Abbass, 2019c; Bostrom, 2014; Gabriel, 2020; Prasad, 2019; Taylor et al., 2016).”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “Third, we anticipate ConceptFusion to inherit the limitations and biases of foundation models [5, 73], warranting further investigations for potential harm as well as research into AI safety and alignment [74, 75].”