OpenAssistant Conversations - Democratizing Large Language Model Alignment
By Andreas Kopf, Yannic Kilcher, Dimitri von Rutte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, A. Barhoum, Nguyen Minh Duc, Oliver Stanley, Rich'ard Nagyfi and 8 more. 2023
893 citations recorded by the source corpus.
Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models. We release our code and data under a fully permissive licence.
Subfields: Artificial Intelligence, LLM Agents & Tool Use
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Mohit Bansal
- Yejin Choi
- Noah A. Smith
- Percy Liang
- Hannaneh Hajishirzi
- Jianfeng Gao
- Graham Neubig
In LLM Agents & Tool Use
- Yejin Choi
- Douwe Kiela
- Maosong Sun
- Edward Grefenstette
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Michael Lewis
- Zhiyuan Liu
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “, 2022) based on FLAN-T5-xl, the Open Assistant (Köpf et al., 2023) reward model based on DeBERTa V3 Large (He et al.”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “• Conversational benchmarks , like CoQA [35], MMDialog [15] and OpenAssistant [23], are closest to our intended use cases.”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “OASST1 The OpenAssistant dataset [31] was collected via crowd-sourcing.”
- A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “Early methods mainly utilize human feedback for the alignment of human preferences (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021; Ouyang et al., 2022; Bai et al., 2022b; K¨opf et al., 2023; Yuan et al., 2023b).”
- TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “…that can promote diversity, including: WildChat [Zhao et al., 2024], which is a large source of real-user interaction with models; Open Assistant [Köpf et al., 2024], which is created by volunteer workers for general [Rajani et al., 2023], which is annotated by expert workers for a broad range…”
- Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (2023, background): “Alpaca-7B (Taori et al., 2023b) 0.43 OpenAssistant-12B (Köpf et al., 2023) 0.50 Koala-13B (Geng et al., 2023) 0.51 Vicuna-13B (Chiang et al., 2023) 0.54 UltraLLaMA 0.54 tasks.”
- Evaluating Large Language Models at Evaluating Instruction Following (2023, background): “Authors who manually curate LLMB AR are NOT involved in the experiment as they know the gold labels.”
- Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models (2023, background): “These open-source LLMs have significantly benefited the community and lowered entry barriers by eliminating the substantial costs associated with developing such models from scratch (Kopf et al., 2023).”
- Effective Long-Context Scaling of Foundation Models (2023, background): “In fact, most existing open-source instruction datasets (Conover et al., 2023; Köpf et al., 2023) predominantly consist of short samples.”
- Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference (2024, background): “These include Ope-nAssistant (Köpf et al., 2023), HH-RLHF (Bai et al., 2022), LMSYS-Chat-1M (Zheng et al., 2023a), and synthetic approximations of human preferences like UltraFeedback (Cui et al., 2023) and Nectar (Zhu et al., 2023).”
- WildChat: 1M ChatGPT Interaction Logs in the Wild (2024, background): “In contrast, W ILD C HAT and Open Assistant exhibit a greater linguistic diversity with only 52.94% and 56.02% of their turns in English.”
- How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources (2023, methodology): “The test set consists of 805 instructions, with 252 instructions from the Self-Instruct evaluation [47], 188 from the Open Assistant evaluation [26], 129 from the helpful evaluation by Anthropic [3], 80 from the Vicuna evaluation [8], and 156 from the Koala evaluation [19].”
- Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, neutral): “…52k samples GPT4-Alpaca (Peng et al., 2023b) 52k samples Baize (Xu et al., 2023a) 210k samples Dolly (Conover et al., 2023) 15k samples Open-assistant (Köpf et al., 2023) 34k samples LIMA (Zhou et al., 2023a) 1k samples LLMs on this data using maximum likelihood estimation (MLE) (Wei et al., 2021).”
- Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “To create datasets for alignment, previous studies have collected human interactions with LLMs [14, 64, 65, 66, 26].”
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “This is even more difficult for multilingual data and most efforts to date have focused primarily on English datasets (Köpf et al., 2023; Conover et al., 2023; Zhou et al., 2023).”