Flipout: Efficient Pseudo-Independent Weight Perturbations on Mini-Batches

By Yeming Wen, Paul Vicol, Jimmy Ba, Dustin Tran, Roger Baker Grosse. 2018

353 citations recorded by the source corpus.

Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limiting the variance reduction effect of large mini-batches. We introduce flipout, an efficient method for decorrelating the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each example. Empirically, flipout achieves the ideal linear variance reduction for fully connected networks, convolutional networks, and RNNs. We find significant speedups in training neural networks with multiplicative Gaussian perturbations. We show that flipout is effective at regularizing LSTMs, and outperforms previous methods. Flipout also enables us to vectorize evolution strategies: in our experiments, a single GPU with flipout can handle the same throughput as at least 40 CPU cores using existing methods, equivalent to a factor-of-4 cost reduction on Amazon Web Services.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning (2021, methodology): “We use in particular a multiplicative Gaussian perturbation that rescales the parameters, as the information content of the weights is said to be invariant to their scale (Wen et al., 2018).”
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021, background): “Comparing with the classical strategy that perturbs weight in the entire space [40, 16, 42, 10], we focus on characterizing the weight loss landscape on the new task with respect to the important subspace representing old task.”
Efficient and Scalable Bayesian Neural Nets with Rank-1 Factors (2020, methodology): “Although 2x the parameter count of other methods, we also tune a vanilla BNN baseline for CIFAR that uses Gaussian priors and approximate posteriors over the full set of weights and Flipout (Wen et al., 2018) for estimating expectations.”
BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning (2020, methodology): “This is achieved by manipulating the matrix computations for a mini-batch (Wen et al., 2018).”
Reliable Uncertainty Estimates in Deep Neural Networks using Noise Contrastive Priors (2018, methodology): “As baseline, we compare to Flipout (Wen et al., 2018), an improved version of BBB that decorrelates minibatch gradients.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “…trained independently on the entire dataset using random initialization (Lakshminarayanan et al., 2017) (we set M = 10 in experiments below) • ( SVI ) Stochastic Variational Bayesian Inference for deep learning (Blundell et al., 2015; Graves, 2011; Louizos & Welling, 2017, 2016; Wen et al., 2018).”
Deep Ensembles: A Loss Landscape Perspective (2019, methodology): “…(MacKay, 1992), Markov chain Monte Carlo methods (Neal, 1996; Welling & Teh, 2011; Springenberg et al., 2016), variational Bayesian methods (Graves, 2011; Blundell et al., 2015; Louizos & Welling, 2017; Wen et al., 2018) and Monte-Carlo dropout (Gal & Ghahramani, 2016; Srivastava et al., 2014).”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, methodology): “For BNN ( VI ), we use Flipout [45] to reduce gradient variances and follow [43] for variational inference on deep ResNets.”
Hyperparameter Ensembles for Robustness and Uncertainty Quantification (2020, background): “com/google/uncertainty-baselines), variational inference [70] leads to (NLL, ACC, ECE)=(0.”
Adversarial Weight Perturbation Helps Robust Generalization (2020, methodology): “Different from the commonly used random weight perturbation (sampling a random direction) [50, 18, 15], we propose the Adversarial Weight Perturbation (AWP), along which the adversarial loss increases dramatically.”
The k-tied Normal Distribution: A Compact Parameterization of Gaussian Mean Field Posteriors in Bayesian Neural Networks (2020, positive): “The model uses the Flipout estimator (Wen et al., 2018) and a constraint on the maximum value of the GMFVI posterior standard deviations of 0.2.”
Laplace Redux - Effortless Bayesian Deep Learning (2021, methodology): “We compare them with the MAP network (MAP) and various popular and strong Bayesian baselines: Deep Ensemble [DE, 14], mean-field variational Bayes [VB, 11, 12] with the flipout estimator [66], cyclical stochastic-gradient Hamiltonian Monte Carlo [CSGHMC / HMC, 67], and SWAG [SWG, 15].”
Uncertainty-Aware Audiovisual Activity Recognition Using Deep Bayesian Variational Inference (2018, methodology): “We use Flipout [48], which is an efﬁ-cient method that decorrelates the gradients within a mini-batch by implicitly sampling pseudo-independent weight perturbations for each input.”
A Systematic Comparison of Bayesian Deep Learning Robustness in Diabetic Retinopathy Tasks (2019, methodology): “In addition, we use a Monte Carlo estimate of the KL-divergence term of the ELBO in order to reduce the time complexity of a forward pass to O(D) in the number of weights.”
Functional Variational Bayesian Neural Networks (2019, background): “Local reparameterization trick (Kingma et al., 2015) and Flipout (Wen et al., 2018) try to decorrelate the gradients within a mini-batch for reducing variances during training.”
Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, methodology): “6b, is sometimes used as it allows to compress the number of optimisation parameters when using variational inference [92].”
Variational Beam Search for Novelty Detection (2021, methodology): “We also used the Flipout estimator (Wen et al., 2018) to reduce variance in the gradient estimator.”
Action and Perception as Divergence Minimization (2020, background): “…Sun et al., 2019), ﬂexible posteriors (Louizos and Welling, 2016; Sun et al., 2017; Louizos and Welling, 2017; Zhang et al., 2018; Chang et al., 2019), low rank posteriors (Izmailov et al., 2018; Dusenberry et al., 2020), and improved inference algorithms (Wen et al., 2018; Immer et al., 2020).”
Bayesian Layers: A Module for Neural Network Uncertainty (2018, methodology): “, 2015), Flipout (Wen et al., 2018), and deterministic variational inference (Wu et al.”