Large Language Model Alignment: A Survey

By Tianhao Shen, Renren Jin, Yufei Huang, Chuang Liu, Weilong Dong, Zishan Guo, Xinwei Wu, Yan Liu, Deyi Xiong. arXiv.org, 2023

326 citations recorded by the source corpus.

Recent years have witnessed remarkable progress made in large language models (LLMs). Such advancements, while garnering significant attention, have concurrently elicited various concerns. The potential of these models is undeniably vast; however, they may yield texts that are imprecise, misleading, or even detrimental. Consequently, it becomes paramount to employ alignment techniques to ensure these models to exhibit behaviors consistent with human values. This survey endeavors to furnish an extensive exploration of alignment methodologies designed for LLMs, in conjunction with the extant capability research in this domain. Adopting the lens of AI alignment, we categorize the prevailing methods and emergent proposals for the alignment of LLMs into outer and inner alignment. We also probe into salient issues including the models' interpretability, and potential vulnerabilities to adversarial attacks. To assess LLM alignment, we present a wide variety of benchmarks and evaluation methodologies. After discussing the state of alignment research for LLMs, we finally cast a vision toward the future, contemplating the promising avenues of research that lie ahead. Our aspiration for this survey extends beyond merely spurring research interests in this realm. We also envision bridging the gap between the AI alignment research community and the researchers engrossed in the capability exploration of LLMs for both capable and safe LLMs.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/749d59f887c8ac83fd4f5178465e8b03e463358c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiling Yan
2. Lifang He
3. Lichao Sun

In Information Systems

1. Lifang He
2. Lichao Sun

Citations in the corpus

• Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “In addressing these challenges, ensuring the trustworthiness of large models has become one of the primary concerns for researchers [135, 107, 110, 136].”
• A Survey on the Memory Mechanism of Large Language Model-based Agents (2024, unknown): “In addition, cognitive psychology has been studied for a long time, so many effective human memory theories and architectures have been accumulated, which can support more advanced capabilities of the agents [147].”
• High-Dimension Human Value Representation in Large Language Models (2024, unknown): “In SFT, the model is fine-tuned by consuming a set of curated conversation data complying with human desired attributes such as human preferences or values [10, 35, 36, 37].”
• Evaluating Alignment Techniques for Enhancing LLM Performance in a Closed-Domain Application: A RAG Bench-Marking Study (2024, unknown): “Model alignment in Large Language Models (LLMs) ensures that a model’s outputs and behaviors align with human values, ethics, and specific requirements [1].”