Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision
By Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David D. Cox, Yiming Yang, Chuang Gan. 2023
457 citations recorded by the source corpus.
Recent AI-assistant agents, such as ChatGPT, predominantly rely on supervised fine-tuning (SFT) with human annotations and reinforcement learning from human feedback (RLHF) to align the output of large language models (LLMs) with human intentions, ensuring they are helpful, ethical, and reliable. However, this dependence can significantly constrain the true potential of AI-assistant agents due to the high cost of obtaining human supervision and the related issues on quality, reliability, diversity, self-consistency, and undesirable biases. To address these challenges, we propose a novel approach called SELF-ALIGN, which combines principle-driven reasoning and the generative power of LLMs for the self-alignment of AI agents with minimal human supervision. Our approach encompasses four stages: first, we use an LLM to generate synthetic prompts, and a topic-guided method to augment the prompt diversity; second, we use a small set of human-written principles for AI models to follow, and guide the LLM through in-context learning from demonstrations (of principles application) to produce helpful, ethical, and reliable responses to user's queries; third, we fine-tune the original LLM with the high-quality self-aligned responses so that the resulting model can generate desirable responses for each query directly without the principle set and the demonstrations anymore; and finally, we offer a refinement step to address the issues of overly-brief or indirect responses. Applying SELF-ALIGN to the LLaMA-65b base language model, we develop an AI assistant named Dromedary. With fewer than 300 lines of human annotations (including<200 seed prompts, 16 generic principles, and 5 exemplars for in-context learning). Dromedary significantly surpasses the performance of several state-of-the-art AI systems, including Text-Davinci-003 and Alpaca, on benchmark datasets with various settings.
Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Graham Neubig
- Yejin Choi
- Percy Liang
- Hannaneh Hajishirzi
- Noah A. Smith
- Jianfeng Gao
In Large Language Models
- Yejin Choi
- Luke S. Zettlemoyer
- Noah A. Smith
- Hannaneh Hajishirzi
- Michael Lewis
- Zhiyuan Liu
- Bill Yuchen Lin
- Maosong Sun
In Alignment & RLHF
- Luke S. Zettlemoyer
- Yejin Choi
- Alexander Wei
- Noah A. Smith
- Yu Meng
- Yilun Du
- Swaroop Mishra
- Hannaneh Hajishirzi
Citations in the corpus
- Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “• GPT-4-pref: In addition to human-pref, we use GPT-4 as a proxy for human preference following prior work (Fu et al., 2023; Chiang et al., 2023; Geng et al., 2023; Sun et al., 2023), and found high correlation (82% for Sentiment Reversal, 68% for Acronym Generation, and 71% for Dialogue Response Ge”
- Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, positive): “At this writing, the central idea of this paper has been adopted in several follow-up works for such endeavors (Taori et al., 2023; Xu et al., 2023; Sun et al., 2023, i.a.).”
- Using In-Context Learning to Improve Dialogue Safety (2023, background): “In contrast to our approach, these filters cannot easily be adapted to new unsafe inputs or new unsafe responses—each undesirable behaviour you wish to mitigate must be reflected in the safety filter’s training corpus. et al., 2023; Bai et al., 2022b).”
- A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “Jin et al. [121] conducted a comprehensive study on LLMs’ understanding of binary code semantics [122] across different architectures and optimization levels, providing key insights for future research in this area.”
- Large Language Models are not Fair Evaluators (2023, background): “Recently, a lot of works (Zheng et al., 2023; Wang et al., 2023a; Sun et al., 2023; Zhou et al., 2023) have been utilizing LLMs such as GPT-4 as evaluators to compare the performance of two AI assistants.”
- Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias (2023, background): “Such applications include generating tabular data [4], relation triplets [8], sentence pairs [46], instruction data [40, 50, 53, 47], etc. .”
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, methodology): “5-Turbo-0613 (i.e. Turbo) and GPT-4-0613 (i.e. GPT-4) on 11 distinct domains of unsafe data (Sun et al., 2023) in both Chinese and English.”
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “Another category of approaches [53, 47, 58, 59, 55, 46] focus on prompting LLMs to generate synthetic instruction datasets, beginning with a small set of human-annotated seed instructions and expanding these through few-shot prompting.”
- A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Models (2024, background): “It helps the model follow human commands for specific tasks (Wang et al., 2023; Chung et al., 2022; Iyer et al., 2023; Sun et al., 2023b) and eventually increases the faithfulness of the model’s outputs.”
- Visual Adversarial Examples Jailbreak Aligned Large Language Models (2023, methodology): “emerging alignment techniques such as Constitutional AI [7] and self-alignment [69].”
- Safe RLHF: Safe Reinforcement Learning from Human Feedback (2023, unknown): “Reinforcement Learning from Human Feedback While LLMs have excelled in various language tasks, they sometimes exhibit unexpected behaviors such as producing inaccurate information or making biased, misleading, and harmful responses (Bai et al., 2022a;b; Koco´n et al., 2023; Sun et al., 2023b).”
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, unknown): “, 2022b) and self-alignment (Sun et al., 2023) are also emerging.”
- A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Recently, distillation-based methods are also proposed (Schick et al., 2023; Zhang, 2023; Patil et al., 2023; Tang et al., 2023a; Qin et al., 2023a; Yuan et al., 2023a; Gao et al., 2023b; Wang et al., 2024; Shen et al., 2024; Yuan et al., 2024b).”
- Large Language Model Alignment: A Survey (2023, background): “Constitutional AI (or principle-guided alignment) (Bai et al., 2022c; Sun et al., 2023b) can be viewed as a scalable oversight approach, where humans provide meta-supervision signals”
- Large Language Models: A Survey (2024, unknown): “…efficient developments of LLMs) includes Inner-Monologue [100], Megatron-Turing NLG [101], LongFormer [102], OPT-IML [103], MeTaLM [104], Dromedary [105], Palmyra [106], Camel [107], Yalm [108], MPT [109], ORCA-2 [110], Gorilla [67], PAL [111], Claude [112], CodeGen 2 [113], Zephyr [114], Grok…”
- What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (2023, background): “While RLHF has underpinned the development of ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023), recent researches suggest that instruction tuning in isolation can offer competitive results (Sun et al., 2024; Zhou et al., 2023).”
- On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, unknown): “For example, TruthfulQA enables the assessment of a model’s ability to identify true claims (Sun et al., 2023); NIV2 is employed to evaluate a model’s language comprehension and reasoning abilities across multiple tasks (Wang et al., 2023e).”
- UltraFeedback: Boosting Language Models with High-quality Feedback (2023, unknown): “Following Sun et al. (2023) and Mukherjee et al. (2023), we define a set of principles for each aspect to prompt models to consider expected aspects in their responses.”