Learning Ordered Representations with Nested Dropout

By Oren Rippel, Michael A. Gelbart, Ryan P. Adams. 2014

129 citations recorded by the source corpus.

In this paper, we present results on ordered representations of data in which different dimensions have different degrees of importance. To learn these representations we introduce nested dropout, a procedure for stochastically removing coherent nested sets of hidden units in a neural network. We first present a sequence of theoretical results for the special case of a semilinear autoencoder. We rigorously show that the application of nested dropout enforces identifiability of the units, which leads to an exact equivalence with PCA. We then extend the algorithm to deep models and demonstrate the relevance of ordered representations to a number of applications. Specifically, we use the ordered property of the learned codes to construct hash-based data structures that permit very fast retrieval, achieving retrieval in time logarithmic in the database size and independent of the dimensionality of the representation. This allows codes that are hundreds of times longer than currently feasible for retrieval. We therefore avoid the diminished quality associated with short codes, while still performing retrieval that is competitive in speed with existing methods. We also show that ordered representations are a promising way to learn adaptive compression for efficient online data reconstruction.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Spectral Representations for Convolutional Neural Networks (2015, background): “Note that this can be regarded as an application of nested dropout (Rippel et al., 2014) on both dimensions of the frequency decomposition of our input.”
Stick-Breaking Variational Autoencoders (2016, methodology): “es from a particular deﬁnition of the energy function of the Gibbs distribution, not from an inﬁnite dimensional Bayesian prior. Lastly, our training procedure bears some semblance to Nested Dropout (Rippel et al., 2014), which removes all hidden units at an index lower than some threshold index. Th”
Variable Bitrate Discrete Neural Representations via Causal Self-Attention (2021, methodology): “We compare two methods of imposing order in the latent codes: training our Perceiver Quantizer with Nested Dropout [25] ( Nest ), and with causal attention masking ( Causal ).”
Variable-rate discrete representation learning (2021, background): “Examples of such approaches are the Indian buffet process (Ghahramani & Grifﬁths, 2005), Stick-breaking VAEs (Nalisnick & Smyth, 2017), Inﬁnite restricted Boltzmann machines (Cˆot´e & Larochelle, 2016) and nested dropout (Rippel et al., 2014).”
Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks (2018, methodology): “rk RNN, would learn a multi-scale hierarchical recurrence. However, the model still has a pre-determined depth to the hierarchy, depending on the number of layers. Our work is more closely related to Rippel et al. (2014), which propose to induce a hierarchy in the representation units by applying “n”
Deep Convolutional Neural Networks for Image Classification: A Comprehensive Review (2017, methodology): “Inspired by its application to unsupervised autoencoders (Rippel et al., 2014), Finn et al. (2015) used it to train, by standard backpropagation, compact DCNNs that adapt to different tasks and data complexity.”
Deep Generative Models with Stick-Breaking Priors (2016, methodology): “es from a particular deﬁnition of the energy function of the Gibbs distribution, not from an inﬁnite dimensional Bayesian prior. Lastly, our training procedure bears some semblance to Nested Dropout (Rippel et al., 2014), which removes all hidden units at an index lower than some threshold index. Th”
DyLoRA: Parameter-Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation (2022, methodology): “Inspired by the dropout (Hinton et al., 2012), nested drop-out (Rippel et al., 2014) is a stochastic regularization technique that targets enforcing ordered representations in training auto-encoders.”
Compressing Neural Networks with the Hashing Trick (2015, methodology): “Rippel et al. (2014) propose a novel dropout method, nested dropout, to give an order of importance for hidden neurons.”
Regularizing Deep Convolutional Neural Networks with a Structured Decorrelation Constraint (2016, methodology): “To reduce overfitting, several regularizers, optimization methods and network structures have been proposed, including Early Stopping [13], Weight Decay [14], Model Ensemble [15–17], Data Augmentation [18], Dropout [19, 20], DropConnect [21], Maxout [22], Batch Normalization [23], Residual Networks ”
Learning with Pseudo-Ensembles (2014, methodology): “For example, [17] develops a method for learning “ordered representations” by applying dropout/masking noise in a deep autoencoder while enforcing a particular “nested” structure among the random masking variables in ξ, and [2] relies heavily on random perturbations when training Generative Stochast”