Deep Learners Benefit More from Out-of-Distribution Examples

By Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger-Lewandowski, Thomas Breuel, Youssouf Chherawala, Moustapha Cissé, Myriam Côté, Dumitru Erhan, Jeremy Eustache and 7 more. 2011

169 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

In Scaling & Learning Theory

Citations in the corpus

How transferable are features in deep neural networks? (2014, background): “We are interested in the answers to these questions because, to the extent that features within a network are general, we will be able to use them for transfer learning (Caruana, 1995; Bengio et al., 2011; Bengio, 2011).”
Revisiting Natural Gradient for Deep Networks (2013, methodology): “(2010), using the NISTP dataset introduced in Bengio et al. (2011) (which is just the NIST dataset plus deformations) and use 32.”
Stochastic Optimization of Plain Convolutional Neural Networks with Simple Methods (2020, background): “[5] did apply a large variety of transformations to a handwritten character dataset, for example, local elastic deformation, motion blur, Gaussian smoothing, Gaussian noise, salt and pepper noise, pixel permutation and affine transformations[4].”
Improved Regularization of Convolutional Neural Networks with Cutout (2017, background): “Of these techniques ours is closest to the occlusions applied in [1], however their occlusions generally take the form of scratches, dots, or scribbles that overlay the target character, while we use zero-masking to completely obstruct an entire region.”
Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, neutral): “For character recoginition task (Bengio et al., 2011) image transformations and occlusions for their handwritten characters were combined.”
Albumentations: fast and flexible image augmentations (2018, background): “Image augmentations have also been shown to improve convergence [7], generalization and robustness on out-of-distribution samples [8,9], and to overall have more advantages compared to other regularization techniques [10].”
Dataset Augmentation in Feature Space (2017, background): “Another option is dataset augmentation, wherein the existing data is transformed in some way to create new data that appears to come from the same (conditional) data generating distribution (Bengio et al., 2011).”
Adversarial Diversity and Hard Positive Generation (2016, background): “[2] demonstrated that deep networks benefit even more from out-of-distribution examples including perturbed versions of the original training examples.”
The Origins and Prevalence of Texture Bias in Convolutional Neural Networks (2019, methodology): “Data augmentation is widely used to improve generalization of CNNs on data drawn from the same distribution as the training data [62, 16, 8, 57, 22, 19, 64, 20, 103, 45].”
Tradeoffs in Data Augmentation: An Empirical Study (2021, background): “Since early uses of data augmentation in training neural networks, there has been an assumption that effective transforms for data augmentation are those that produce images from an “overlapping but different" distribution (Bengio et al., 2011; Bellegarda et al., 1992).”
Further Advantages of Data Augmentation on Convolutional Neural Networks (2018, background): “Bengio et al. Bengio et al. (2011) focused on the importance of data augmentation for recognizing handwritten digits through greedy layer-wise unsupervised pre-training Bengio et al. (2007).”
Cosaliency Detection Based on Intrasaliency Prior Transfer and Deep Intersaliency Mining (2016, background): “The generation of the final cosaliency map is introduced in Section III-E.”
Survey on Deep Neural Networks in Speech and Vision Systems (2019, methodology): “The study subsequently proposes a pipeline that leverages AlexNet [36] through transfer learning [117] for feature extraction and then uses a pool of SVM’s for scale invariant classification.”
SynthSeg: Segmentation of brain MRI scans of any contrast and resolution without retraining (2021, unknown): “Therefore, there is a clear need for a fast, accurate, and reproducible automated method, for segmentation of brain scans of any contrast and resolution, and that can adapt to a wide range of populations.”
A Group-Theoretic Framework for Data Augmentation (2019, background): “Other works have demonstrated connections between data augmentation and kernel classifiers [24], marginalized corrupted features [51], network architecture [9], margins [55], certain complexity measures [38], optimization landscape [47], and biological plausibility [35].”
On the Expressive Power of Deep Architectures (2011, background): “For example, some of the features that are useful for one task may be useful for another, making Deep Learning particularly well suited for transfer learning and multi-task learning (Caruana, 1995; Collobert and Weston, 2008b; Bengio et al., 2011; Bengio, 2011).”