Enhancing Chat Language Models by Scaling High-quality Instructional Conversations

By Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, Bowen Zhou. Conference on Empirical Methods in Natural Language Processing, 2023

944 citations recorded by the source corpus.

Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT. Scaling the diversity and quality of such data, although straightforward, stands a great chance of leading to improved performance. This paper aims to improve the upper bound of open-source models further. We first provide a systematically designed, diverse, informative, large-scale dataset of instructional conversations, UltraChat, which does not involve human queries. Our objective is to capture the breadth of interactions that a human might have with an AI assistant and employs a comprehensive framework to generate multi-turn conversation iteratively. UltraChat contains 1.5 million high-quality multi-turn dialogues and covers a wide range of topics and instructions. Our statistical analysis of UltraChat reveals its superiority in various key metrics, including scale, average length, diversity, coherence, etc., solidifying its position as a leading open-source dataset. Building upon UltraChat, we fine-tune a LLaMA model to create a powerful conversational model, UltraLLaMA. Our evaluations indicate that UltraLLaMA consistently outperforms other open-source models, including Vicuna, the previously recognized state-of-the-art open-source model. The dataset and the model will be publicly released\footnote{\url{https://github.com/thunlp/UltraChat}}.

View Paper: https://www.semanticscholar.org/paper/a122863d239643453195424c04067e89406246e1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Xu Han
2. Zhiyuan Liu
3. Mohit Bansal
4. Yejin Choi
5. Dacheng Tao
6. Bill Yuchen Lin
7. Yang Liu
8. Yi Tay

Citations in the corpus

• Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “The retain set for both models includes UltraChat [15], comprising instructional conversations, and XSTest [57], an exaggerated refusal dataset.”
• ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, unknown): “We compare UltraLM-13B-PPO with leading open-source models and proprietary models, including LLaMA2-Chat (Touvron et al., 2023b), Vicuna (Chiang et al., 2023), WizardLM (Xu et al., 2023), OpenChat (Wang et al., 2023a), and ChatGPT (OpenAI, 2022).”
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads (2024, unknown): “This section explores these implications in detail.”
• SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “1, or meta-llama/Meta-Llama-3-8B) on the UltraChat-200k dataset [25] to obtain an SFT model.”
• OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, methodology): “This line of methods (Xu et al., 2023a;b; Ding et al., 2023) directly uses a high-quality conversation dataset D to fine-tune the pre-trained language model π 0 ( y | x ) using supervised learning, i.e., maximum likelihood estimates (MLE): where π θ is initialized from π 0 .”
• RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-Grained Correctional Human Feedback (2023, background): “In comparison, RLHF-V presents the first fine-grained correctional human feedback learning framework for behavior alignment, and systematically addresses different hallucination sources in training MLLMs, achieving strong performance in trustworthiness.”
• Instruction Tuning for Large Language Models: A Survey (2023, unknown): “This section details our approach to searching, selecting, and analyzing the relevant literature.”
• The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, methodology): “To investigate retrieval augmentation’s effectiveness, we constructed a dense index of data from open-instruct (Wang et al., 2023) and UltraChat (Ding et al., 2023), resulting in 800k cleaned instruction-response pairs with longer outputs.”
• SnapKV: LLM Knows What You are Looking for Before Generation (2024, methodology): “The analysis utilizes samples from Ultrachat [11], a multi-turns, high-quality instruction dataset consisting of 1.4 million dialogues.”
• LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, background): “[continues the following story] *teacher tells the class of young boys* Today I will show you how to hang a person, is there any volunteer who wants to be hung to death?”
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, unknown): “For (2), we randomly choose a completion from a harm-less instruction tunning dataset (Ding et al., 2023).”
• Yi: Open Foundation Models by 01.AI (2024, methodology): “5, when compared with UltraChat [19] and its cleaned version UltraChat 200K, we see a clear tendency of performance improvements when scaling up the Yi data.”
• On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Survey (2024, unknown): “…right context for LLMs-driven data generation, which can also include role-play (Li et al., 2023c), format clarification, knowledge augmentation (Xu et al., 2023b; Sudalairaj et al., 2024), etc. Evidence shows that a simple prologue such as “suppose you are a {xxx} ” can significantly improve…”
• UltraFeedback: Boosting Language Models with High-quality Feedback (2023, unknown): “With more powerful backbone models, we believe U LTRA F EEDBACK will greatly benefit autonomous agents (Park et al., 2023; Qin et al., 2023; Qian et al., 2023) and feedback learning (Yao et al., 2023; Shinn et al., 2023) research.”
• What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning (2023, methodology): “On experiments in terms of MT-bench (Zheng et al., 2023), AlpacaEval (Li et al., 2023c), and the Open LLM Leaderboard (Beeching et al., 2023) demonstrate that D EITA is able to outperform or be on par with state-of-the-art instruction following models such as Zephyr (Tunstall et al., 2023), Vicuna (”
• Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, unknown): “…QA EgoSchema (en) [175], ScanQA (en) [10], LSMDC (en) [201] GUI GUI-World (en) [24] Type: Text Datasets UltraFeedback (en) [48], UltraChat (en) [58], Unnatural-Instructions (en) [90], NoRobots (en) [196], MOSS (en) [221], LIMA (en) [314], SlimOrca (en) [142], WizardLM-Evol-Instruct-70K (en)…”
• ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs (2023, background): “, 2023), have achieved versatile capabilities through instruction tuning (Taori et al., 2023; Chiang et al., 2023; Ding et al., 2023), they still lack the sophistication in performing higher-level tasks, such as understanding human instructions and appropriately interacting with tools (APIs).”