Multitask Prompted Training Enables Zero-Shot Task Generalization
By Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey and 30 more. 2022
242 citations recorded by the source corpus.
Subfields: Natural Language Processing, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
Citations in the corpus
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, neutral): “However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and human-written completions [23, 36, 13, 39].”
- MetaICL: Learning to Learn In Context (2021, neutral): “…(e.g., a question answering format) (Khashabi et al., 2020; Zhong et al., 2021), or rely heavily on task-specific templates (Mishra et al., 2022; Wei et al., 2022; Sanh et al., 2022) which are difficult to engineer due to high variance in performance from very small changes (Mishra et al., 2021).”
- MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, positive): “…of structured, task-based datasets [42,56], we also opt to include the training sets of various downstream tasks – adding the train sets of VQA [2], GQA [23], OKVQA [31], VizWiz [17], Hateful Memes [25], and CoCo Captions [9], resulting in a final dataset of around 25 million image-text pairs.”
- Improving In-Context Few-Shot Learning via Self-Supervised Training (2022, unknown): “…cross-task generalization abilities for in-context few-shot or zero-shot learning using human-annotated datasets (Ye et al., 2021; Wei et al., 2022; Sanh et al., 2022; Min et al., 2021; Xu et al., 2022) via instructions (Weller et al., 2020; Efrat and Levy, 2020; Mishra et al., 2021; Ouyang et…”
- Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “For more detailed introduction, please refer to Appendix A.1.”
- MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, unknown): “Instruction tuning refines large language models (LLMs) based on user intentions, enhancing their ability to generalize across tasks and align with human preferences (Ouyang et al., 2022; Sanh et al., 2022; Muennighoff et al., 2023; Wang et al., 2022).”
- An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-Tuning (2023, unknown): “Scialom et al. (2022) [3] investigate the CF issues of T0 models during continual instruction tuning.”
- mGPT: Few-Shot Learners Go Multilingual (2022, unknown): “…Understanding Scao et al. (2023) compared the performance of BLOOM 176B , mGPT 1.3B , OPT 175B (Zhang et al., 2022), GPT-J 6B (Wang and Komatsuzaki, 2021), and T0 11B (Victor et al., 2022) on subset of tasks from the Super-GLUE benchmark (Wang et al., 2019) in the zero-shot and one-shot settings.”
- A Survey on Large Language Models for Code Generation (2024, unknown): “…section explores three fine-tuning techniques: prompt engineering, which optimizes LLM outputs by crafting effective input instructions; feedback refinement, which reduces errors by incorporating corrections; and domain-specific dataset tuning, which improves LLM performance in specialized areas.”
- Is ChatGPT a General-Purpose Natural Language Processing Task Solver? (2023, unknown): “One particularly fruitful direction has been training LLMs with supervision (Sanh et al., 2021b; Wei et al., 2021; Mishra et al., 2022; Chung et al., 2022) and/or human feedback (Ouyang et al., 2022).”