Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
By Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J. Liu. Journal of machine learning research, 2019
26,606 citations recorded by the source corpus.
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, we explore the landscape of transfer learning techniques for NLP by introducing a unified framework that converts every language problem into a text-to-text format. Our systematic study compares pre-training objectives, architectures, unlabeled datasets, transfer approaches, and other factors on dozens of language understanding tasks. By combining the insights from our exploration with scale and our new "Colossal Clean Crawled Corpus", we achieve state-of-the-art results on many benchmarks covering summarization, question answering, text classification, and more. To facilitate future work on transfer learning for NLP, we release our dataset, pre-trained models, and code.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing
View Paper: https://arxiv.org/abs/1910.10683
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yoshua Bengio
- Quoc V. Le
- Yejin Choi
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Mohit Bansal
- Timo Schick
- Furu Wei
- Jianfeng Gao
- Hannaneh Hajishirzi
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Yejin Choi
- Graham Neubig
- Jianfeng Gao
- Percy Liang
- Noah A. Smith
- Hannaneh Hajishirzi
Citations in the corpus
- Scaling Laws for Neural Language Models (2020, background): “Deep learning has recently seen rapid progress in language modeling, with state of the art models [RNSS18, DCLT18, YDY + 19, LOG + 19, RSR + 19] approaching human-level performance on many specific tasks [WPN + 19], including the composition of coherent multi-paragraph prompted text samples [RWC +…”
- Training Compute-Optimal Large Language Models (2022, neutral): “In Appendix C, we reproduce the IsoFLOP analysis on two additional datasets: C4 [40] and GitHub code [38].”
- PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Raffel et al. (2020) then pre-trained and fine-tuned up to 11B parameter encoder-decoder models, setting a new bar in transfer learning.”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We thus included the publicly available C4 dataset (Raffel et al., 2020) in our data.”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, background): “Just like T5 [45] or BART, RAG can be fine-tuned on any seq2seq task, whereby both the sequence generator and retriever are jointly learned.”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “Raffel et al. (2019) showed that generative models are competitive for reading comprehension tasks such as SQuAD (Rajpurkar et al., 2016), where answers are spans.”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Documents are greedily split into chunks of up to 288 BERT wordpieces, resulting in just over 13 million retrieval candi- 4We initially conducted our own T5 experiments using the code from https://tinyurl.com/t5-openqa-colab (Raffel et al., 2019).”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “For the language model, we rely on the T5 sequence-to-sequence architecture (Raffel et al., 2019).”
- Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “Pre-training methods which learn directly from raw text have revolutionized NLP over the last few years (Dai & Le, 2015; Peters et al., 2018; Howard & Ruder, 2018; Radford et al., 2018; Devlin et al., 2018; Raffel et al., 2019).”
- Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “One recent line of work aims to improve the ability of language models to perform a task by providing instructions that describe the task (Raffel et al., 2020; Wei et al., 2022; Ouyang et al., 2022; Sanh et al., 2022; Wang et al., 2022b).”
- How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “T5 model (Raffel et al., 2019)—the system that boasted the highest score on SuperGLUE at the time of writing—achieved an average accuracy of 88.”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “, 2019b); often such attempts to build a single model for multiple NLP tasks underperform the standard pre-training plus finetuning setup (a model per task) (Raffel et al., 2020).”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “, 2019b), and trained using larger batches (Raffel et al., 2019).”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “Big, deep neural language models 1 that have been pre-trained on unlabeled text have proven to be extremely performant when fine-tuned on down-stream Natural Language Processing (NLP) tasks (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lan et al., 2019; Raffel et al., 2019).”
- Multilingual Denoising Pre-training for Neural Machine Translation (2020, background): “Existing approaches (Lample and Conneau, 2019; Edunov et al., 2019; Lewis et al., 2019; Raffel et al., 2019) have been proposed either to partially pre-train the model or to only focus on English corpora.”
- Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “, 2019) and T5 (Raffel et al., 2019), the idea of providing task descriptions has become feasible for neural architectures: We can simply append such descriptions in natural language to an input and let the language model ar X iv :2 00 1.”