BAM! Born-Again Multi-Task Networks for Natural Language Understanding

By Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D. Manning, Quoc V. Le. 2019

243 citations recorded by the source corpus.

It can be challenging to train multi-task neural networks that outperform or even match their single-task counterparts. To help address this, we propose using knowledge distillation where single-task models teach a multi-task model. We enhance this training with teacher annealing, a novel method that gradually transitions the model from distillation to supervised learning, helping the multi-task model surpass its single-task teachers. We evaluate our approach by multi-task fine-tuning BERT on the GLUE benchmark. Our method consistently improves over standard single-task and multi-task training.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “Exploiting commonality across machine learning tasks has a rich history studied under transfer learning (Caruana, 1997; Clark et al., 2019b).”
Technical report on Conversational Question Answering (2019, methodology): “KD can be used for both single-task and multi-task models (Clark et al., 2019).”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, background): “We collected data to produce conservative estimates for human performance on several tasks that we did not ultimately include in our benchmark, including GAP (Webster et al., 2018), PAWS (Zhang et al., 2019), Quora Insincere Questions, 8 Ultraﬁne Entity Typing (Choi et al., 2018b), and Empathetic Re”
Unifying Question Answering and Text Classification via Span Extraction (2019, background): “Further improvements can be obtained through heuristics such as knowledge distillation as demonstrated in (Anonymous, 2019).”
Finetuned Language Models Are Zero-Shot Learners (2021, background): “Reading comprehension datasets: BoolQ, MultiRC, OBQA.”
Rethinking Few-Shot Image Classification: a Good Embedding Is All You Need? (2020, methodology): “ts for optimization and knowledge transfer between tasks. BAN [13] introduced sequential distillation, which also improved the performance of teacher models. In natural language processing (NLP), BAM [7] used BAN to distill from single-task models to a multi-task model, helping the multi-task model ”
Knowledge Distillation: A Survey (2020, methodology): “, 2016) and multitask distillation (Liu et al., 2019e; Yang et al., 2020b; Sanh et al., 2019; Clark et al., 2019).”
Lifelong Language Knowledge Distillation (2020, background): “…idea about distilling knowledge from equal-size models has also been studied in born-again neural network (Furlanello et al., 2018), multitask learning (Clark et al., 2019) and lifelong computer vision learning (Hou et al., 2018), but never been explored in lifelong language learning research.”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, methodology): “For a different purpose of deriving a single model for multiple tasks through distillation, Clark et al. (2019) used a pre-trained student model of the same size as multiple teacher models.”
Parameter-Efficient Transfer Learning with Diff Pruning (2020, background): “Several works have shown that a single BERT model can obtain good performance across multiple tasks when jointly trained (Liu et al., 2019b; Clark et al., 2019; Stickland & Murray, 2019).”
MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices (2020, methodology): “Attention Transfer (AT) The attention mechanism greatly boosts the performance of NLP and becomes a crucial building block in Transformer and BERT (Clark et al., 2019a; Jawahar et al., 2019).”
Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks (2021, methodology): “For CB and BoolQ, since test sets are not available, we divide the validation sets in half, using one half for validation and the other for testing.”
With Little Power Comes Great Responsibility (2020, neutral): “, 2020), BAM(large) (Clark et al., 2019a), RoBERTa (large) (Liu et al.”
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models (2019, background): “(Clark et al., 2019) propose to use knowledge distillation so that single task models can teach a multi-task model.”
Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks (2020, methodology): “To tackle these challenges, several novel self-distillation frameworks [43], [44], [57], [72], [93], [117], [145], [164], [238], [246], [265] have been proposed recently.”
Universal Representation Learning from Multiple Domains for Few-shot Classification (2021, background): “Born-Again Neural Networks [15] uses KD proposes to consecutively distill knowledge from an identical teacher network to a student network, which is further applied to few-shot learning in [51] and multi-task learning in [10].”