The Natural Language Decathlon: Multitask Learning as Question Answering

By Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, Richard Socher. 2018

665 citations recorded by the source corpus.

Presented on August 28, 2018 at 12:15 p.m. in the Pettit Microelectronics Research Center, Room 102 A/B.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, background): “To better understand the challenged posed by GLUE, we conduct experiments with simple base-lines and state-of-the-art sentence representation models.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “The Transformer was initially shown to be effective for machine translation, but it has subsequently been used in a wide variety of NLP settings [Radford et al., 2018; Devlin et al., 2018; McCann et al., 2018; Yu et al., 2018].”
Learning Transferable Visual Models From Natural Language Supervision (2021, background): “The development of “text-to-text” as a standardized input-output interface (McCann et al., 2018; Radford et al., 2019; Raffel et al., 2019) has enabled taskagnostic architectures to zero-shot transfer to downstream datasets removing the need for specialized output heads or dataset specific…”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “McCann et al. (2018) also present tasks in the form of natural language sentences as part of their Natural Language Decathlon; however, their motivation is not to enable few-shot learning, but to provide a unified framework for many different types of NLP tasks.”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “Other similarly-motivated benchmarks include SentEval (Conneau and Kiela, 2018), which evaluates fixed-size sentence embeddings, and DecaNLP (McCann et al., 2018), which recasts a set of target tasks into a general question-answering format and prohibits task-specific parameters.”
Learning and Evaluating General Linguistic Intelligence (2019, methodology): “McCann et al. (2018) train a question-answering language model on many different NLP tasks (e.g., machine translation, summarization, sentiment analysis, etc.) by formulating every task as question answering in order to make the input and output of the model be consistent.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “…the degree of control available in image generation as well as the recent progress in text generation (Radford et al., 2019) and multitask learning McCann et al. (2018), we train a language model that is conditioned on a variety of control codes (Pfaff, 1979; Poplack, 1980) that make desired…”
Unifying Question Answering and Text Classification via Span Extraction (2019, methodology): “Joint models for classification and question answering have already been studied (Collobert et al., 2011; McCann et al., 2018; Radford et al., 2019) with an even broader set of tasks that require text generation and more general architectures.”
AI and the Everything in the Whole Wide World Benchmark (2021, background): “The notion of task diversity, not size, is repeated quite frequently by GLUE authors [Wang et al., 2019a, p. 1,2,3,5] as a main differentiator from predecessors’ benchmarks, SentEval [Conneau and Kiela, 2018] and decaNLP [McCann et al., 2018].”
Finetuned Language Models Are Zero-Shot Learners (2021, methodology): “For instance, McCann et al. (2018) cast ten NLP tasks as QA and train a model on a collection of tasks formulated with natural language prompts; they report transfer learning gains on finetuning tasks as well as zero-shot domain adaptation results on SNLI (Bowman et al., 2015) and Amazon/Yelp…”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “This shortfall can be seen in benchmarks like DecaNLP (McCann et al., 2018) and GLGE (Liu et al.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, background): “Similar unifying frameworks were proposed by Keskar et al. (2019) and McCann et al. (2018).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “The former two settings evaluates the cross-task generalization of instruction-tuning while the last setting evaluates the generalization of supervised multi-task learning (McCann et al., 2018).”
Emergent Abilities of Large Language Models (2022, neutral): “For GPT-3, Gopher, and Chinchilla, models of ∼ 10 22 training FLOPs ( ∼ 10B parameters) or smaller do not perform better than guessing on average over all the topics, scaling up to 3 – 5 · 10 23 training FLOPs (70B–280B parameters) enables performance to substantially surpass random.”
AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “SentEval (Conneau and Kiela, 2018) and De-caNLP (McCann et al., 2018) also set benchmarks for evaluating models’ general capabilities.”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “…numerical class labels), the task-general nature of these generative models became the precursor to many multitask transfer learning studies (McCann et al., 2018; Khashabi et al., 2020; Ye et al., 2021; Vu et al., 2020), which in turn led to the first wave of instruction tuning as…”