Learning with Pseudo-Ensembles

By Philip Bachman, Ouais Alsharif, Doina Precup. 2014

665 citations recorded by the source corpus.

We formalize the notion of a pseudo-ensemble, a (possibly infinite) collection of child models spawned from a parent model by perturbing it according to some noise process. E.g., dropout [9] in a deep neural network trains a pseudo-ensemble of child subnetworks generated by randomly masking nodes in the parent network. We examine the relationship of pseudo-ensembles, which involve perturbation in model-space, to standard ensemble methods and existing notions of robustness, which focus on perturbation in observation-space. We present a novel regularizer based on making the behavior of a pseudo-ensemble robust with respect to the noise process generating it. In the fully-supervised setting, our regularizer matches the performance of dropout. But, unlike dropout, our regularizer naturally extends to the semi-supervised setting, where it produces state-of-the-art results. We provide a case study in which we transform the Recursive Neural Tensor Network of [19] into a pseudo-ensemble, which significantly improves its performance on a real-world sentiment analysis benchmark.

Subfields: Artificial Intelligence, Computational Mathematics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Mathematics

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “Consistency regularization has been very successful for computer vision applications (Bachman et al., 2014; Laine and Aila, 2017; Tarvainen and Valpola, 2017).”
Unsupervised Data Augmentation for Consistency Training (2019, background): “In a nutshell, consistency training methods simply regularize model predictions to be invariant to small noise applied to either input examples [6, 7, 8] or hidden states [2, 4].”
Gaussian Error Linear Units (GELUs) (2016, background): “Some stochastic regularizers can make the network behave like an ensemble of networks, a pseudoensemble (Bachman et al., 2014), and can lead to marked accuracy increases.”
Self-Training With Noisy Student Improves ImageNet Classification (2019, background): “Apart from self-training, another important line of work in semi-supervised learning [9, 85] is based on consistency training [6, 4, 53, 36, 70, 45, 41, 51, 10, 12, 49, 2, 38, 72, 74, 5, 81].”
Big Self-Supervised Models are Strong Semi-Supervised Learners (2020, methodology): “Another family of methods are based on label consistency regularization [60–62, 39, 14, 13, 63, 15], where unlabeled examples are directly used as a regularizer to encourage task prediction consistency.”
Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, methodology): “While previous methods based on bootstrapping have used pseudo-labels [16], cluster indices [17] or a handful of labels [18, 19, 20], we propose to directly bootstrap the representations.”
FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “This idea was first proposed in [2] and popularized by [46, 24], where the model is trained both via a standard supervised classification loss and on unlabeled data via the loss function”
Unsupervised and Semi-supervised Learning with Categorical Generative Adversarial Networks (2015, background): “While on ﬁrst glance these appear only weakly related, they are strongly connected to entropy minimization as, for example, concisely explained in Bachman et al. (2014).”
Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning (2017, methodology): “Bachman et al. [3] studied random perturbation in the setting of semi-supervised learning.”
Temporal Ensembling for Semi-Supervised Learning (2016, methodology): “As such, their technique follows the general pseudo-ensemble agreement (PEA) regularization framework of Bachman et al. (2014). In addition, they employ a mutual exclusivity loss term (Sajjadi et al.”
Distributional Smoothing with Virtual Adversarial Training (2015, methodology): “Instead of computing the computationally expensive full Jacobian, they approximated the Jabobian by the sum of the 100 600 1000 3000 SVM (Weston et al., 2012) 23.44 8.85 7.77 4.21 TSVM (Weston et al., 2012) 16.81 6.16 5.38 3.45 EmbedNN (Weston et al., 2012) 16.9 5.97 5.73 3.59 *MTC (Rifai et al., 20”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, background): “The Jensen-Shannon Consistency Loss impels to model to be stable, consistent, and insensitive across a diverse range of inputs (Bachman et al., 2014; Zheng et al., 2016; Kannan et al., 2018).”
Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations (2016, background): “Zoneout, like dropout, trains a pseudo-ensemble [Bachman et al., 2014], but uses a stochastic “identity-mask” rather than a zero-mask.”
Dark Experience for General Continual Learning: a Strong, Simple Baseline (2020, background): “It is worth noting that combining data augmentation with our regularization objective enforces an implicit consistency loss [2, 3], which aligns predictions for the same example subjected to small data transformations.”
Rethinking Pre-training and Self-training (2020, methodology): “Self-training is related to the method of pseudo labels [60–63] and consistency training [64–82].”
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average (2018, background): “Currently the best semi-supervised results are obtained by consistency-enforcing approaches (Bachman et al., 2014; Laine and Aila, 2017; Tarvainen and Valpola, 2017; Miyato et al., 2017; Park et al., 2017).”
Data Distillation: Towards Omni-Supervised Learning (2017, methodology): “[1] employ the pseudoensemble regularization term to train models robust on input perturbations.”
End-to-End Semi-Supervised Object Detection with Soft Teacher (2021, methodology): “There are several ways to implement perturbations, including perturbing the model [1], augmenting the images [23] or adversarial training [19].”
ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring (2019, neutral): “First proposed in (Bachman et al., 2014), (Sajjadi et al.”
Consistency Regularization for Generative Adversarial Networks (2019, methodology): “In this paper, we examine a technique called consistency regularization (Bachman et al., 2014; Sajjadi et al., 2016; Laine & Aila, 2016; Zhai et al., 2019; Xie et al., 2019; Hu et al., 2017) in contrast to gradient-based regularizers.”