LongForm: Effective Instruction Tuning with Reverse Instructions

By Abdullatif Köksal, Timo Schick, Anna Korhonen, Hinrich Schütze. 2023

58 citations recorded by the source corpus.

Instruction tuning enables language models to more effectively generalize and better follow user intent. However, obtaining instruction data is costly and challenging. Prior work employs methods such as expensive human annotation, crowd-sourced datasets with alignment issues, and generating noisy examples via LLMs. We introduce the LongForm-C dataset, which is created by reverse instructions. We generate instructions via LLMs for human-written corpus examples using reverse instructions. First we select a diverse set of human-written documents from corpora such as C4 and Wikipedia; then we generate instructions for these documents via LLMs. This approach provides a cheaper and cleaner instruction-tuning dataset with natural output and one suitable for long text generation. Our models outperform 10x larger language models without instruction tuning on tasks such as story/recipe generation and long-form question answering. Moreover, LongForm models outperform prior instruction-tuned models such as FLAN-T5 and Alpaca by a large margin, and improve language understanding capabilities further. We publicly release our data and models: https://github.com/akoksal/LongForm.

Subfields: Education, Information Systems, Media Technology, Large Language Models, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

1. Luke S. Zettlemoyer

In Media Technology

1. Ari Holtzman
2. Luke S. Zettlemoyer

In Large Language Models

1. Tim Dettmers
2. Luke S. Zettlemoyer
3. Ari Holtzman
4. Artidoro Pagnoni
5. Yejin Choi
6. Mohit Bansal
7. Michael Lewis
8. Xiang Ren

In Alignment & RLHF

1. Ari Holtzman
2. Luke S. Zettlemoyer
3. Tim Dettmers
4. Artidoro Pagnoni

Citations in the corpus

• QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Longform The LongForm dataset [30] is based on an English corpus augmented with instructions and as such is a hybrid human-generated dataset.”
• Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models (2023, methodology): “, 2022b) and (9) LongForm (Köksal et al., 2023)) to compare with in-domain datasets.”
• MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, unknown): “This section details the steps used to produce MURI-IT.”