Fraternal Dropout

By Konrad Zolna, Devansh Arpit, Dendi Suhubdy, Yoshua Bengio. 2017

54 citations recorded by the source corpus.

Recurrent neural networks (RNNs) are important class of architectures among neural networks useful for language modeling and sequential prediction. However, optimizing RNNs is known to be harder compared to feed-forward neural networks. A number of techniques have been proposed in literature to address this problem. In this paper we propose a simple technique called fraternal dropout that takes advantage of dropout to achieve this goal. Specifically, we propose to train two identical copies of an RNN (that share parameters) with different dropout masks while minimizing the difference between their (pre-softmax) predictions. In this way our regularization encourages the representations of RNNs to be invariant to dropout mask, thus being robust. We show that our regularization term is upper bounded by the expectation-linear dropout objective which has been shown to address the gap due to the difference between the train and inference phases of dropout. We evaluate our model and achieve state-of-the-art results in sequence modeling tasks on two benchmark datasets - Penn Treebank and Wikitext-2. We also show that our approach leads to performance improvement by a significant margin in image captioning (Microsoft COCO) and semi-supervised (CIFAR-10) tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Do Not Have Enough Data? Deep Learning to the Rescue! (2019, methodology): “They also include data noising techniques , such as altering words in the input of self-encoder networks in order to generate a different sentence (Xie et al. 2017; Zolna et al. 2017; Li et al. 2018), or introducing noise on the word-embedding level.”
Direct Output Connection for a High-Rank Language Model (2018, methodology): “8 AWD-LSTM + Fraternal Dropout (Zolna et al., 2018) 34M 66.”
Gmail Smart Compose: Real-Time Assisted Writing (2019, background): “In recent years with the rapid development in deep learning techniques, a multitude of neural network-based language models have been proposed, significantly improving state-of-theart performance [3, 26, 36, 37, 40, 49, 51].”
Pushing the bounds of dropout (2018, background): “…results in deep learning (Hinton et al. 2012; Srivastava et al. 2014), and its application has received much attention in the form of optimisation (Wang & Manning 2013) and attempts at explaining or improving its approximation properties (Baldi & Sadowski 2013; Zolna et al. 2017; Ma et al. 2016).”
Co-regularized Alignment for Unsupervised Domain Adaptation (2018, background): “has resemblance to Π-model [22] and fraternal dropout [46], which were recently proposed in the context of (semi-)supervised learning.”
R-Drop: Regularized Dropout for Neural Networks (2021, methodology): “Existing literature [41, 81] has revealed the possible side effect of dropout that there is an unnegligible inconsistency between training and inference stage of dropout models, i.e., the randomly sampled sub model (caused by dropout) during training is inconsistent with the full model (without…”