Instruction Tuning with GPT-4
By Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, Jianfeng Gao. 2023
837 citations recorded by the source corpus.
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yejin Choi
- Yoshua Bengio
- Noah A. Smith
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Ming-Hsuan Yang
- Yu Qiao
- Andrew Zisserman
- Jitendra Malik
In Large Language Models
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Noah A. Smith
- Percy Liang
- Danqi Chen
- Jianfeng Gao
- Nan Duan
- Yu Meng
In Alignment & RLHF
- Tianyi Zhang
- Noah A. Smith
- Ari Holtzman
- Yu Meng
- Percy Liang
- Pieter Abbeel
- Carlos Guestrin
- Sergey Levine
Citations in the corpus
- The False Promise of Imitating Proprietary LLMs (2023, background): “…proprietary models for specific tasks (Sun et al., 2023; Hsieh et al., 2023; Honovich et al., 2022) and broadly imitate models, e.g., Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), GPT4ALL (Anand et al., 2023), and more (Wang et al., 2022a; Peng et al., 2023).”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, background): “Most openly accessible datasets are comprised of synthetic data of instructions automatically generated by querying language models [8, 9, 10, 11, 12].”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [8] and other concurrent or follow-up work [5, 28, 14, 12, 51, 18, 32, 39, 7, 42].”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “For the evaluation of chatbots approaches that use GPT-4 instead of costly human annotation have been developed [10, 45].”
- Visual Instruction Tuning (2023, positive): “Alpaca [48], Vicuna [9], GPT-4-LLM [38] utilize various machine-generated high-quality instruction-following samples to improve the LLM’s alignment ability, reporting impressive performance compared with proprietary LLMs. Importantly, this line of work is text-only .”
- TALLRec: An Effective and Efficient Tuning Framework to Align Large Language Model with Recommendation (2023, methodology): “Considering that instruction tuning is core to letting the LLM learn to solve different tasks and have strong generalization ability [22, 23, 37], we propose a lightweight tuning framework to adapt LLMs for recommendations, named TALLRec.”
- Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models (2023, methodology): “To conduct a comprehensive comparison, we compare various instruction datasets: (1) Self-Instruct (Wang et al., 2022a) employs vanilla GPT3 (Brown et al., 2020) to generate instruction data automatically; (2) Alpaca (Taori et al., 2023) introduces refinements to the pipeline of Self-Instruct to gene”
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, background): “More recently, multiple works (Wang et al., 2023h; Xu et al., 2023; Peng et al., 2023; Zhou et al., 2023b; Wang et al., 2023g) propose to utilize synthetic instruction following data distilled from GPT-3/4 to align open-source LLMs.”
- MultiModal-GPT: A Vision and Language Model for Dialogue with Humans (2023, methodology): “We employ the Dolly 15k and Alpaca GPT4 datasets [12] as resources for assessing language-only instruction-following capabilities.”
- EarthGPT: A Universal Multimodal Large Language Model for Multisensor Image Comprehension in Remote Sensing Domain (2024, neutral): “…techniques in EarthGPT are elaborated, involving the visual-enhanced perception mechanism (see Section III-B), the language perception (see Section III-C), the cross-modal mutual comprehension approach (see Section III-D), and the unified instruction tuning method for RS (see Section III-E).”
- Hallusionbench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models (2023, background): “Most recently, researchers have been creating high-quality, diverse multi-modal datasets from GPT4 and GPT-4V [48] to fine-tune open-source LVLMs, including LLaVA [33], MiniGPT4 [63], Mplug-Owl [50], LRV-Instruction [29], LLaVAR [60] and other works [12, 25, 37, 52].”
- An Embodied Generalist Agent in 3D World (2023, neutral): “See the results in Appendix I.4.”
- Large Language Models for Information Retrieval: A Survey (2023, neutral): “This intriguing area remains open for further investigation and advancement.”
- RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “Since the introduction of GPT-3 [73], prompt learning has evolved from traditional discrete [69] and continuous prompt construction [59], [70] to large-scale model-oriented in-context learning [32], instruction-tuning [74], [75], [76], and chain-of-thought approaches [77], [78], [79].”
- OpenChat: Advancing Open-source Language Models with Mixed-Quality Data (2023, background): “Peng et al. (2023) follow Alpaca’s setup but replace GPT-4 as the distillation teacher.”
- Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “Various methods have been developed for fine-tuning LLMs, with popular approaches including Supervised Fine-Tuning (SFT) (Peng et al., 2023) and Reinforcement Learning from Human Feedback (RLHF) (Ziegler et al., 2019; Stiennon et al., 2020; Ouyang et al., 2022).”
- Vision-Language Foundation Models as Effective Robot Imitators (2023, neutral): “This benchmark requires a 7-DOF Franka Emika Panda robot arm with a parallel gripper, utilizing onboard sensors and images from two camera views to successfully complete sequences of up to five language instructions consecutively.”