How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources
By Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy and 1 more. 2023
510 citations recorded by the source corpus.
In this work we explore recent advances in instruction-tuning language models on a range of open instruction-following datasets. Despite recent claims that open models can be on par with state-of-the-art proprietary models, these claims are often accompanied by limited evaluation, making it difficult to compare models across the board and determine the utility of various resources. We provide a large set of instruction-tuned models from 6.7B to 65B parameters in size, trained on 12 instruction datasets ranging from manually curated (e.g., OpenAssistant) to synthetic and distilled (e.g., Alpaca) and systematically evaluate them on their factual knowledge, reasoning, multilinguality, coding, and open-ended instruction following abilities through a collection of automatic, model-based, and human-based metrics. We further introduce T\"ulu, our best performing instruction-tuned model suite finetuned on a combination of high-quality open resources. Our experiments show that different instruction-tuning datasets can uncover or enhance specific skills, while no single dataset (or combination) provides the best performance across all evaluations. Interestingly, we find that model and human preference-based evaluations fail to reflect differences in model capabilities exposed by benchmark-based evaluations, suggesting the need for the type of systemic evaluation performed in this work. Our evaluations show that the best model in any given evaluation reaches on average 87% of ChatGPT performance, and 73% of GPT-4 performance, suggesting that further investment in building better base models and instruction-tuning data is required to close the gap. We release our instruction-tuned models, including a fully finetuned 65B T\"ulu, along with our code, data, and evaluation framework at https://github.com/allenai/open-instruct to facilitate future research.
Subfields: Artificial Intelligence, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Hannaneh Hajishirzi
- Yejin Choi
- Yoshua Bengio
- Sergey Levine
In Large Language Models
In Alignment & RLHF
Citations in the corpus
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [8] and other concurrent or follow-up work [5, 28, 14, 12, 51, 18, 32, 39, 7, 42].”
- SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “Further research has also demonstrated that RLHF can lead to biased outcomes, such as verbose outputs from the model [28, 66, 80].”
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “More recently, (Chen et al., 2023; Wang et al., 2023e) have proposed to tackle more challenging college-level science and math problems.”
- LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “Open-Instruct (Wang et al., 2023c) provides recipes for training instruct models.”
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “The most common pattern of reward hacking in practice is verbosity: the language models generate more tokens to make the response appear more detailed or better formatted after RLHF (usually for helpfulness) but the actual quality does not improve [Singhal et al., 2023, Wang et al., 2023b].”
- LoRA+: Efficient Low Rank Adaptation of Large Models (2024, background): “On many tasks such as instruction finetuning, LoRA has been shown to achieve comparable or better performance compared with full-finetuning (Wang et al., 2023; Liu et al., 2023), although on complicated, long form generation tasks, it is not always as performant.”
- ORPO: Monolithic Preference Optimization without Reference Model (2024, background): “Instruction-tuning (Wei et al., 2022; Taori et al., 2023; Wang et al., 2023; Zhou et al., 2023a) trains models to follow task descriptions given in natu-ral language, which enables models to generalize well to previously unseen tasks.”
- AgentTuning: Enabling Generalized Agent Abilities for LLMs (2023, unknown): “Held-in/out Tasks We select six held-in tasks from AgentBench (Liu et al., 2023): ALF-World (Shridhar et al., 2020), WebShop (Yao et al., 2022), Mind2Web (Deng et al., 2023), and three others, using AgentBench metrics.”
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “To sample diverse input-output pairs, we sample instances of the Open-Instruct (Wang et al., 2023) dataset.”
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “Task diversity (Longpre et al., 2023a; Wang et al., 2023b; Chung et al., 2022), complexity (Xu et al., 2023; Luo et al., 2023b,a) and quality (Zhou et al., 2023; Taori et al., 2023b; Muennighoff et al., 2023a; Zhuo et al., 2024) are three critical axes for successful instruction tuning.”
- GraphGPT: Graph Instruction Tuning for Large Language Models (2023, unknown): “This significant reduction in token consumption translates to a substantial decrease in training and inference resource requirements.”
- An Empirical Study of Scaling Instruct-Tuned Large Multimodal Models (2023, methodology): “1, which is the best performance for the 70B model size, according to [17] and the Chatbot Arena Leaderboard (8).”
- Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, methodology): “Notably, we modified the official Alpaca dataset by identifying and removing 1,902 safety-related training samples via sensitive phrase matching (Wang et al., 2023b), resulting in a 50K-sized uncensored Alpaca dataset.”