Information Dropout: learning optimal representations through noise

By Alessandro Achille, Stefano Soatto. 2017

27 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Variational Dropout Sparsifies Deep Neural Networks (2017, methodology): “Then we describe Variational Dropout, a recently proposed Bayesian regularization technique (Kingma et al., 2015).”
Emergence of Invariance and Disentanglement in Deep Representations (2017, methodology): “In parallel developments, the IB Lagrangian was used as a regularized loss function for learning representation, leading to new information theoretic regularizers (Achille and Soatto, 2016; Alemi et al., 2017). In this paper, we introduce the use of the IB Lagrangian between the weights of a network”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Data-dependent PAC-Bayes priors via differential privacy (2017, background): “Earlier work by Achille and Soatto (2016) introduced this objective in the deﬁnition of information dropout , an extension of variational dropout (Kingma, Salimans, and Welling, 2015) motivated from the information bottleneck perspective.”
Entropy-SGD optimizes the prior of a PAC-Bayes bound: Generalization properties of Entropy-SGD and data-dependent priors (2017, background): “f Hinton and Camp. Their objective takes the form of regularized empirical cross entropy Rˆ S(Q)+bKL(QjjP); (2) where Q and P are the prior and posterior on the weights, respectively. Earlier work by Achille and Soatto (2016) introduced this objective in the deﬁnition of information dropout, an exte”
Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives (2019, background): “This type of input compression has been shown to improve generalization [39, 1, 2].”
Deep relaxation: partial differential equations for optimizing deep neural networks (2017, methodology): “Based on empirical results such as Dauphin et al. (2014), authors in Bray and Dean (2007); Fyodorov and Williams (2007); Choromanska et al. (2015) and Chaudhari and Soatto (2015) have modeled the energy landscape of deep neural networks as a high-dimensional Gaussian random ﬁeld.”