Alignment of Language Agents
By Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, Geoffrey Irving. 2021
227 citations recorded by the source corpus.
For artificial intelligence to be beneficial to humans the behaviour of AI agents needs to be aligned with what humans want. In this paper we discuss some behavioural issues for language agents, arising from accidental misspecification by the system designer. We highlight some ways that misspecification can occur and discuss some behavioural issues that could arise from misspecification, including deceptive or manipulative language, and review some approaches for avoiding these issues.
Subfields: Artificial Intelligence, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Sergey Levine
- Percy Liang
- Quoc V. Le
- Jianfeng Gao
- Yoshua Bengio
- Kyunghyun Cho
In LLM Agents & Tool Use
- Harrison Edwards
- Ilya Sutskever
- Maosong Sun
- Dorsa Sadigh
- Luke S. Zettlemoyer
- Ryan J. Lowe
- Anca Dragan
- Daniel Klein
Citations in the corpus
- Training language models to follow instructions with human feedback (2022, background): “These risks have been extensively documented (Bender et al., 2021; Bommasani et al., 2021; Kenton et al., 2021; Weidinger et al., 2021; Tamkin et al., 2021).”
- LaMDA: Language Models for Dialog Applications (2022, background): “Developing richer definitions and taxonomies of dialog agent behaviors, such as how polite behavior should be operationalized, is important for avoiding misspecification [104] and testing whether model behavior aligns with politeness norms in defined application contexts.”
- TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “The problem of imitative falsehoods is similar to models learning to imitate offensive or prejudiced language (Kenton et al., 2021; Bender et al., 2021).”
- Evaluating Large Language Models Trained on Code (2021, methodology): “In the literature, a model is defined informally as “intent aligned” with a user if (and only if) the model intends to do what the user wants (Chris-tiano, 2018; Kenton et al., 2021).”
- OPT: Open Pre-trained Transformer Language Models (2022, background): “There is a growing body of work detailing ethical and social risks from deploying language models with emergent capabilities at scale (Weidinger et al., 2021a; Bommasani et al., 2021; Dinan et al., 2021; Kenton et al., 2021).”
- Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, background): “[21] addresses harmful content as one component of behavioral issues, and acknowledges existing approaches are varied and the field requires further research.”
- Predictability and Surprise in Large Generative Models (2022, background): “Although we focus on scaling laws, many of our points complement existing views on the societal risks of deploying large models [7, 67, 9, 19, 72, 41].”
- Unsolved Problems in ML Safety (2021, background): “Research on steerable and controllable text generation [104, 99] could help chatbots exhibit virtues such as friendliness and honesty.”
- A Generalist Agent (2022, background): “It may be possible to extend some of the value alignment approaches for language (Ouyang et al., 2022; Kenton et al., 2021) to generalist agents.”
- Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “In light of this, many works have documented both existing and potential harms arising from generated text [11, 102, 62], ranging from misinformation [74] to the reinforcement of social biases through the perpetuation of stereotypes [95].”
- Taxonomy of Risks posed by Language Models (2022, background): “Where a LM prediction causes a false belief in a user, this may threaten personal autonomy and even pose downstream AI safety risks [99].”
- Large Language Model Alignment: A Survey (2023, background): “Instead, we focus on the values to which LLMs, as language agents (Kenton et al., 2021), are supposed to align.”
- Solving math word problems with process- and outcome-based feedback (2022, unknown): “If optimized for total user satisfaction, such an agent may influence users towards preferences which are easier to satisfy (e.g. easier to predict, or generally more amenable to ML-generated proposals), in order to increase user-reported satisfaction (Kenton et al., 2021).”
- Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, unknown): “…2023; Touvron et al., 2023a;b; Anthropic, 2023; Gemini Team, 2023) heavily hinges on AI alignment approaches (Leike et al., 2018; Christian, 2020; Kenton et al., 2021; Leike & Sutskever, 2023; Ji et al., 2023)—typically a mixture of supervised Fine-tuning (SFT) (Wei et al., 2021) and…”
- RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “[15] distinguishes the inaccurate parts in responses via human annotation, and internally discourages the halluci-nated parts by direct preference optimization.”
- A Survey of Large Language Models (2023, unknown): “Besides, there are other alignment criteria for LLMs from different perspectives including behavior, intent, incentive, and inner aspects [207], which are essentially similar (or at least with similar alignment techniques) to the above three criteria.”
- Position: Levels of AGI for Operationalizing Progress on the Path to AGI (2023, unknown): “This framing is more narrow than OpenAI’s definition of economically valuable work and has the additional downside of potentially introducing alignment risks (Kenton et al., 2021) by only targeting fiscal profit.”
- When Combating Hype, Proceed with Caution (2021, background): “…proposed some mechanisms by which we might mitigate these risks, work in this area is still largely exploratory, with no clear research agenda in place to ensure that powerful ML models will be safe (Hadfield-Menell et al., 2016; Irving et al., 2018; Critch and Krueger, 2020; Kenton et al., 2021).”
- Discovering Latent Knowledge in Language Models Without Supervision (2022, methodology): “The increasing deployment of language models in real-world applications opens up exciting possibilities, but it also raises the stakes of AI research and presents new risks (Bommasani et al., 2021; Weidinger et al., 2021; Bender et al., 2021).”