Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks

By Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, Eshaan Pathak and 29 more. 2022

156 citations recorded by the source corpus.

Subfields: Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Prompting & Reasoning

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We used the 6.7B parameter scale for the pretraining objective ablations (Wang et al., 2022a) and the 1.3B scale for the rest including position embeddings, activations, and layer normalization (Le Scao et al., 2022).”
The False Promise of Imitating Proprietary LLMs (2023, background): “This is considerably lower than that of other datasets such as Super-NaturalInstructions (Wang et al., 2022b), which is at 61% BLEU similarity for a similarly sized set of examples.”
Scaling Instruction-Finetuned Language Models (2022, methodology): “Finally, whereas much of prior work ﬁnetunes on either zero-shot without exemplars (Zhong et al., 2021; Wei et al., 2021; Sanh et al., 2021; Wang et al., 2022a) or few-shot with exemplars (Ye et al., 2021; Wei et al., 2021; Min et al., 2022; Wang et al., 2022c), our process ﬁnetunes on a mixture of…”
Learning by Distilling Context (2022, methodology): “Prior work (Wang et al., 2022a) has used this dataset to train instruction-tuned models on prompts consisting of only 2-positive examples or only 2-positive and negative examples with explanations.”
Visual Instruction Tuning (2023, methodology): “…OPT [60] to follow natural language instructions and complete real-world tasks, researchers have explored methods for LLM instruction-tuning [37, 52, 51], leading to instruction-tuned counterparts such as InstructGPT [37]/ChatGPT [35], FLAN-T5 [11], FLAN-PaLM [11], and OPT-IML [22],…”
Instruction Tuning with GPT-4 (2023, methodology): “This is implemented by either ﬁnetuning the model on a wide range of tasks using human-annotated prompts and feedback (Ouyang et al., 2022), or supervised ﬁnetuning using public benchmarks and datasets augmented with manually or automatically generated instructions (Wang et al., 2022b).”
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, positive): “This is likely because of the zero-shot format used during multitask prompted fine-tuning and corroborates a recent finding by [10].”
Second Thoughts are Best: Learning to Re-Align With Human Values from Text Edits (2023, methodology): “…includes the labor-free decomposition (i.e., the editing steps) into the training object, whereas prior works either train on costly manually-created decomposition (Ouyang et al., 2022; Wang et al., 2022) or, rather than training, prompt with such decomposition (Wei et al., 2022; Nye et al., 2021).”
TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding (2023, unknown): “Inspired by the recent success of instruction tuning on LLMs [34, 47, 48], researchers have adopted vision-language instruction tuning to improve instruction following capabilities of Multimodal LLMs [3, 18, 28, 29, 57, 61].”
Instruction Induction: From Few Examples to Natural Language Task Descriptions (2022, methodology): “Subsequent work further improves the performance on such tasks using a self-consistency ensemble [Wang et al., 2022a], which samples a set of diverse chain-of-thought reasoning paths, taking the majority vote over all generated answers.”
VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks (2023, unknown): “Following this idea, subsequent works, such as InstructGPT [36], FLAN [13, 59], and OPT-IML [23], explore the instruction-tuning method [58, 57] and demonstrate that this simple approach effectively enhances the zero-shot and few-shot capabilities of LLMs.”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, unknown): “In NLP, all task instructions can follow the same uniform schema, composed of task definition and positive/negative examples [36, 55].”