R-Drop: Regularized Dropout for Neural Networks

By Xiaobo Liang, Lijun Wu, Juntao Li, Yue Wang, Qi Meng, Tao Qin, Wei Chen, M. Zhang, Tie-Yan Liu. Neural Information Processing Systems, 2021

568 citations recorded by the source corpus.

Dropout is a powerful and widely used technique to regularize the training of deep neural networks. In this paper, we introduce a simple regularization strategy upon dropout in model training, namely R-Drop, which forces the output distributions of different sub models generated by dropout to be consistent with each other. Specifically, for each training sample, R-Drop minimizes the bidirectional KL-divergence between the output distributions of two sub models sampled by dropout. Theoretical analysis reveals that R-Drop reduces the freedom of the model parameters and complements dropout. Experiments on $\bf{5}$ widely used deep learning tasks ($\bf{18}$ datasets in total), including neural machine translation, abstractive summarization, language understanding, language modeling, and image classification, show that R-Drop is universally effective. In particular, it yields substantial improvements when applied to fine-tune large-scale pre-trained models, e.g., ViT, RoBERTa-large, and BART, and achieves state-of-the-art (SOTA) performances with the vanilla Transformer model on WMT14 English$\to$German translation ($\bf{30.91}$ BLEU) and WMT14 English$\to$French translation ($\bf{43.95}$ BLEU), even surpassing models trained with extra large-scale data and expert-designed advanced variants of Transformer models. Our code is available at GitHub{\url{https://github.com/dropreg/R-Drop}}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/520bd2331cca8d5a9c032c186a2a0f7704ead6ff

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding (2022, methodology): “The team in second (yxb) used a multilingual T5 model (Xue et al., 2021) with various data augmentation techniques including: back-translation; synonym replacement; random insertion, swap, and deletion.”
ST++: Make Self-trainingWork Better for Semi-supervised Semantic Segmentation (2021, methodology): “However, limited by the computational burden of Bayesian inference, some other methods adopt Dropout [19, 35] and data augmentations [3] to measure the uncertainty.”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, background): “Closed book WebQA achieves a 47.4 accuracy (prior best of 42.8 from Roberts et al. (2020) and exceeds results from the zero-shot performance of the ERNIE 3.0 Titan 260B dense parameter model (Wang et al., 2021)).”
Designing Effective Sparse Expert Models (2022, methodology): “For both summarization datasets, XSum and CNN-DM, our model achieves state-of-the-art without additional changes to training or ﬁne-tuning (Raffel et al., 2019; Liang et al., 2021).”
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, background): “Many efforts have been made to compensate for the phenomenon (Liang et al., 2021; Xu et al., 2021; Yuan et al., 2023a).”
Defense without Forgetting: Continual Adversarial Defense with Anisotropic & Isotropic Pseudo Replay (2024, methodology): “The alignment process is implemented in the R-Drop [47] way, where f 1 w t and f 2 w t represent that the inputs are fed into the model with random Dropout [42] twice, respectively.”