Relevant sparse codes with variational information bottleneck

By Matthew Chalk, Olivier Marre, Gasper Tkacik. 2016

93 citations recorded by the source corpus.

In many applications, it is desirable to extract only the relevant aspects of data. A principled way to do this is the information bottleneck (IB) method, where one seeks a code that maximizes information about a 'relevance' variable, Y, while constraining the information encoded about the original data, X. Unfortunately however, the IB method is computationally demanding when data are high-dimensional and/or non-gaussian. Here we propose an approximate variational scheme for maximizing a lower bound on the IB objective, analogous to variational EM. Using this method, we derive an IB algorithm to recover features that are both relevant and sparse. Finally, we demonstrate how kernelized versions of the algorithm can be used to address a broad range of problems with non-linear relation between X and Y.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Deep Variational Information Bottleneck (2017, background): “We recently discovered Chalk et al. (2016), who independently developed the same variational lower bound on the IB objective as us.”
Mutual Information Neural Estimation (2018, background): “In order to overcome the intractability of I ( X ; Z ) in the continuous setting, Alemi et al. (2016); Kolchinsky et al. (2017); Chalk et al. (2016) exploit the variational bound of Barber & Agakov (2003) to approximate the conditional entropy in I ( X ; Z ) .”
On the information bottleneck theory of deep learning (2018, background): “But for fundamentally stochastic or quantized networks, the information bottleneck principle could yield new training algorithms where compression is explicitly encouraged with appropriate regularization terms (Chalk et al 2016, Alemi et al 2017, Kolchinsky et al 2017).”
Meta reinforcement learning as task inference (2019, methodology): “However, overﬁtting could signiﬁcantly impair performance, and so we regularize some of our architectures using an information bottleneck (IB) similar to (Alemi et al., 2017; Chalk et al., 2016): We add a stochastic layer on top of the LSTM in the value function, policy, and belief network (see Fig.”
InfoBot: Transfer and Exploration via the Information Bottleneck (2019, background): “…inspired by the information bottleneck approach (Tishby et al., 1999) of training deep neural networks for supervised learning (Alemi et al., 2017; Chalk et al., 2016; Achille and Soatto, 2016; Kolchinsky et al., 2017), where classiﬁers are trained to achieve high accuracy while simultaneously…”
Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach (2018, background): “ not in the context of domain adaptation. Mohamed & Rezende [32] utilized the variational bounds on MI, and apply it to deep neural networks in the context of reinforcement learning. Chalk et al. [12] and Amini et al. [4], developed the same variational lower bound In the context of Information ”
A high-bias, low-variance introduction to Machine Learning for physicists (2018, unknown): “to the IB objective function have been developed (Alemi et al., 2016; Chalk et al., 2016).”
The effectiveness of layer-by-layer training using the information bottleneck principle (2018, methodology): “…and infinite term I(X;L) were overcome by (i) training stochastic DNNs which ensure it is finite, and (ii) using a variational approximation of the MI which makes it tractable (Alemi et al., 2018; Kolchinsky et al., 2017; Chalk et al., 2016; Achille & Soatto, 2018; Belghazi et al., 2018).”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “, choose hidden layer mappings that provide compressed representations of input data (Alemi et al., 2016; Chalk et al., 2016; Kolchinsky et al., 2017; Achille & Soatto, 2018).”
MINE: Mutual Information Neural Estimation (2018, methodology): “ in the case of jointly Gaussian random variables in Chechik et al. (2005). In order to overcome the intractability of I(X;Z) in the continuous setting, Alemi et al. (2016); Kolchinsky et al. (2017); Chalk et al. (2016) exploit the variational bound of (Barber & Agakov, 2003) to approximate the ”
Toward a unified theory of efficient, predictive, and sparse coding (2017, methodology): “For example, we previously used a “kernel” encoding model, where neural responses are described by a nonparametric and nonlinear function of the input (31).”