OpenChat: Advancing Open-source Language Models with Mixed-Quality Data

By Guan Wang, Sijie Cheng, Xianyuan Zhan, Xiangang Li, Sen Song, Yang Liu. 2023

340 citations recorded by the source corpus.

Nowadays, open-source large language models like LLaMA have emerged. Recent developments have incorporated supervised fine-tuning (SFT) and reinforcement learning fine-tuning (RLFT) to align these models with human goals. However, SFT methods treat all training data with mixed quality equally, while RLFT methods require high-quality pairwise or ranking-based preference data. In this study, we present a novel framework, named OpenChat, to advance open-source language models with mixed-quality data. Specifically, we consider the general SFT training data, consisting of a small amount of expert data mixed with a large proportion of sub-optimal data, without any preference labels. We propose the C(onditioned)-RLFT, which regards different data sources as coarse-grained reward labels and learns a class-conditioned policy to leverage complementary data quality information. Interestingly, the optimal policy in C-RLFT can be easily solved through single-stage, RL-free supervised learning, which is lightweight and avoids costly human preference labeling. Through extensive experiments on three standard benchmarks, our openchat-13b fine-tuned with C-RLFT achieves the highest average performance among all 13b open-source language models. Moreover, we use AGIEval to validate the model generalization performance, in which only openchat-13b surpasses the base model. Finally, we conduct a series of analyses to shed light on the effectiveness and robustness of OpenChat. Our code, data, and models are publicly available at https://github.com/imoneoi/openchat and https://huggingface.co/openchat.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Maosong Sun
2. Jie Tang
3. Yankai Lin
4. Zhiyuan Liu
5. Xiao Liu
6. Dacheng Tao
7. Yuxiao Dong
8. Danqi Chen

In Large Language Models

1. Jie Tang
2. Maosong Sun
3. Xiao Liu
4. Yu Meng
5. Yankai Lin
6. Dacheng Tao
7. Jingren Zhou
8. Yuxiao Dong

Citations in the corpus

• UltraFeedback: Boosting Language Models with High-quality Feedback (2023, neutral): “Compared with two critique models, we find that (the community-trained) Shepherd almost always fails to provide high-quality feedback.”
• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, neutral): “As of June 2024, there are over 300 datasets on Hugging Face 1 that are tagged as “syn-thetic”, with many mainstream LLMs leveraging high-quality synthetic data for training, including Alpaca (Taori et al., 2023), Vicuna (Zheng et al., 2023), OpenHermes 2.5, and Openchat 3.5 (Wang et al., 2023a).”
• AgentTuning: Enabling Generalized Agent Abilities for LLMs (2023, positive): “Recent efforts (Chiang et al., 2023; Wang et al., 2023a) aim to distill instruction tuning datasets from proprietary models to improve open-source models’ alignment.”
• How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, methodology): “…SFT on in-domain human-annotated or augmented data (Yuan et al., 2023b; Luo et al., 2023) while diverse and complex human instructions are applauded for aligning human intentions (Wang et al., 2023c; Taori et al., 2023; Xu et al., 2023; Zhou et al., 2023; Wang et al., 2023a; Lu et al., 2023).”
• A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Through KD of LLMs, student evaluators could enhance inference efficiency and achieve more flexible and highly customized evaluation (Wang et al., 2023b; Kim et al., 2024; Xu et al., 2023d; Jiang et al., 2023c; Li et al., 2024a).”
• ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, positive): “We compare UltraLM-13B-PPO with leading open-source models and proprietary models, including LLaMA2-Chat (Touvron et al., 2023b), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023), OpenChat (Wang et al., 2023a), and ChatGPT (OpenAI, 2022).”
• SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “The classical RLHF pipeline typically comprises three phases: supervised fine-tuning [96, 71, 32, 21, 44, 25, 77, 15, 81], reward model training [31, 56, 16, 52, 35, 46], and policy optimization [65, 4].”
• AgentBench: Evaluating LLMs as Agents (2023, neutral): “1 llama2-chat-13b [80] 13B open Meta llama2-chat-7b [80] 7B open openchat-8192-13b [85] 13B open v1 Tsinghua openchat-13b [85] 13B open v3.”
• LLM Web Dynamics: Tracing Model Collapse in a Network of LLMs (2025, unknown): “1-8B-Instruct (Dubey et al., 2024), DeepSeek’s deepseek-llm-7b-chat (Wang et al., 2023), and Mistral’s Mistral-7B-Instruct-v0.”