ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
By Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, Radu Soricut. 2019
7,601 citations recorded by the source corpus.
Increasing model size when pretraining natural language representations often results in improved performance on downstream tasks. However, at some point further model increases become harder due to GPU/TPU memory limitations and longer training times. To address these problems, we present two parameter-reduction techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows that our proposed methods lead to models that scale much better compared to the original BERT. We also use a self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and \squad benchmarks while having fewer parameters compared to BERT-large. The code and the pretrained models are available at this https URL.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Mohit Bansal
- Yejin Choi
- Hannaneh Hajishirzi
- Timo Schick
- Jianfeng Gao
- Furu Wei
In Efficiency & Compression
- Kurt Keutzer
- Tri Dao
- Zi-Mei Yao
- Canwen Xu
- Dan Alistarh
- Julian McAuley
- Luke S. Zettlemoyer
- A. Gholami
Citations in the corpus
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Some current work along these lines include distillation [Hinton et al., 2015; Sanh et al., 2019; Jiao et al., 2019], parameter sharing [Lan et al., 2019], and conditional computation [Shazeer et al., 2017].”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, methodology): “Since ALBERT involves increasing the model hidden size from 1024 to 2048/4096 and thus substantially increases the amount of computation in terms of FLOPs, we exclude ALBERT from the following results as it is hard to lead to scientific conclusions.”
- Scaling Laws for Neural Language Models (2020, neutral): “This suggests that the embedding matrix can be made smaller without impacting performance, as has been seen in recent work [LCG + 19].”
- UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, background): “The datasets used from this format are: NarrativeQA/NarQA (Kociský et al., 2018), the open-domain version of Natu-ralQuestions/NatQA (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019b).”
- Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “We use ALBERT (Lan et al., 2020) and also a distilled version of BERT, DistilBERT (Sanh et al.”
- Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers (2020, methodology): “Similar to Lan et al. (2020), we experiment with two types of layer sharing: sharing all layers and sharing only the attention layers.”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “Big, deep neural language models that have been pre-trained on unlabeled text have proven to be extremely performant when fine-tuned on down-stream Natural Language Processing (NLP) tasks (Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Lan et al., 2019; Raffel et al., 2019).”
- BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, background): “Recent work has shown that very strong performance can be achieved by training for longer (Liu et al., 2019), by tying parameters across layers (Lan et al., 2019), and by masking spans instead of words (Joshi et al., 2019).”
- Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “To address this degradation, the authors of that work (Lan et al., 2019) introduced parameter sharing and showed that that their models scale much better compared to the original BERT model.”
- Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Large Language Models (LMs) achieve remarkable accuracy and generalization ability when fine tuned for NLP tasks (Peters et al., 2018; Devlin et al., 2019; Liu et al., 2019; Lan et al., 2020; Raffel et al., 2020).”
- Measuring Massive Multitask Language Understanding (2020, methodology): “ALBERT-xxlarge attains an accuracy of 27 .”
- Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, methodology): “Finally, our Reasoning Order Prediction (ROP) is inspired by Sentence Order Prediction (SOP) (Lan et al., 2019).”
- RoFormer: Enhanced Transformer with Rotary Position Embedding (2021, background): “…position encoding through a pre-defined function Vaswani et al. [2017] was added to the contextual representations, while a trainable absolute position encoding Gehring et al. [2017], Devlin et al. [2019], Lan et al. [2020], Clark et al. [2020], Radford et al. [2019], Radford and Narasimhan [2018].”
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “Other approaches include compressing along the sequence dimension to attend to multiple tokens at once [50, 55, 76, 86].”