Zephyr: Direct Distillation of LM Alignment

By Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib and 4 more. arXiv.org, 2023

618 citations recorded by the source corpus.

We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets the state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/cdcf3f36866ef1e16eba26d57c2324362247ba84

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Danqi Chen
2. Percy Liang
3. Dan Hendrycks
4. Dacheng Tao
5. Mantas Mazeika
6. Yu Meng
7. Steven Basart
8. Mengzhou Xia

In Information Systems

1. Dacheng Tao
2. Yu Meng
3. Mantas Mazeika
4. Tianyi Zhou
5. Dan Hendrycks
6. Doyen Sahoo
7. Nouha Dziri
8. Danqi Chen

Citations in the corpus

• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, positive): “We use UltraChat as the dataset for the SFT loss, building on the Zephyr codebase (Tunstall et al., 2023).”
• ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “PPO and DPO models were fine-tuned with TRL library (von Werra et al., 2020) on top of SFT models trained for a single epoch on the chosen responses following Rafailov et al. (2023) and Tunstall et al. (2023).”
• Large Language Models: A Survey (2024, neutral): “…[103], MeTaLM [104], Dromedary [105], Palmyra [106], Camel [107], Yalm [108], MPT [109], ORCA-2 [110], Gorilla [67], PAL [111], Claude [112], CodeGen 2 [113], Zephyr [114], Grok [115], Qwen [116], Mamba [30], Mixtral-8x7B [117], DocLLM [118], DeepSeek-Coder [119], FuseLLM-7B [120], TinyLlama-1.”
• SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “For the Base setup, we follow the training pipeline of Zephyr [75].”
• Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, background): “5 and provide the details in Appendix H. (3) SFT on the preferred samples, (4) DPO (Rafailov et al., 2023), (5) RSO (Liu et al., 2023a) 2023), where the authors found that overfitting the preference dataset within certain limit does not hurt the model performance (gold reward) and the strongest…”
• SeaLLMs - Large Language Models for Southeast Asia (2023, background): “5 in most non-Latin SEA languages (Khm, Lao, Mya and Tha) by large margins.”
• RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “We also evaluate popular chat models trained with Direct Preference Optimization (DPO) (Rafailov et al., 2023), for example, Zephyr-β (Tunstall et al., 2023), Qwen-Chat (Bai et al., 2023), StableLM (Bellagente et al., 2024), and T¨ulu 2 (Ivison et al., 2023) to ground recent debates on RLHF methods…”
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, methodology): “Typical alignment methods include Supervised Fine-Tuning (SFT) (Ouyang et al., 2022; Tunstall et al., 2023a) based on human demonstrations, and Reinforcement Learning from Human Feed-back (RLHF) (Christiano et al., 2017; Ziegler et al., 2019; Stiennon et al., 2020; Bai et al., 2022a) based on human…”
• A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…thus methods that learn from AI feedback are also proposed to align with human preferences (Bai et al., 2022a; Kwon et al., 2023; Scheurer et al., 2023; Kim et al., 2023a; Roit et al., 2023; Yang et al., 2024a; Lee et al., 2023a; Tunstall et al., 2023; Cui et al., 2023a; Wang et al., 2023f).”
• RLHF Workflow: From Reward Modeling to Online RLHF (2024, background): “Notable examples include the revolutionary closed-source ChatGPT (OpenAI, 2023), Claude (Anthropic, 2023), and Gemini (Team et al., 2023), as well as the powerful open-source models like Zephyr (Tunstall et al., 2023), Starling (Zhu et al., 2023), and LLaMA-3 (Meta, 2024).”