Learning and generalization with the information bottleneck

By Ohad Shamir, Sivan Sabato, Naftali Tishby. 2008

251 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Deep Variational Information Bottleneck (2017, background): “For more discussion about the generalization behavior of these IB curves see: Shamir et al. (2010); Still & Bialek (2004).”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, neutral): “The work of [29] is closely related to our results under log-loss.”
Deep learning and the information bottleneck principle (2015, background): “Nonetheless, it has been shown in [8] that it is possible to generalize using the IB principle as a learning objective from ﬁnite samples, as long as the representational complexity (i.e. the cardinality of ˆ X ) is limited.”
Flow of Renyi information in deep neural networks (2016, background): “The motivation for maximizing I(X;Y) stems from the fact that I(X;Y) bounds the prediction error in classification tasks [13], while minimizing I(X;X) could capture the most concise representation.”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “Finally, our results are also independent of issues related to the relationship between IB, ﬁnite data sampling, and generalization error (Shamir et al., 2010; Tishby & Zaslavsky, 2015; Vera et al., 2018).”
An Information-Theoretic Analysis of Deep Latent-Variable Models (2017, background): “The information bottleneck framework (Tishby et al., 1999; Shamir et al., 2010; Tishby & Zaslavsky, 2015; Alemi et al., 2017; Achille & Soatto, 2016; 2017) allows a model to smoothly trade off the minimality of the learned representation (Z) from data (X) by minimizing their mutual information, I(X;”
Lossy Compression for Lossless Prediction (2021, background): “First, minimizing the information I[A(X);Z] arises from our desire to perform compression rather than to (optimally [65]) help generalization [111, 112].”
A Lagrangian Perspective on Latent Variable Generative Models (2018, background): “…Zhao et al., 2017; Li et al., 2017a; Chen et al., 2016a) so that the features (latent variables) z can capture as much information as possible about x, or minimize mutual information (Zhao et al., 2017; Higgins et al., 2016; Tishby & Zaslavsky, 2015; Shamir et al., 2010) to achieve compression.”
Fixing a Broken ELBO (2017, background): “The information bottleneck framework leverages information theory to learn robust representations (Tishby et al., 1999; Shamir et al., 2010; Tishby & Zaslavsky, 2015; Alemi et al., 2017; Achille & Soatto, 2016; 2017).”
What makes for good views for contrastive learning (2020, background): “This can lead to worse generalization on the downstream task [2, 47].”
Cognitive Control (2012, unknown): “To be more precise, if all the information about the parameters of such distributions can be captured by some functions of a statistical sample, those functions will be considered as sufficient statistics [29].”
On the Robustness of Information-Theoretic Privacy Measures and Mechanisms (2018, background): “[43] established similar upper bounds for mutual information that scale as O(log(n)/ √ n).”
Partial Or Complete, That’s The Question (2019, unknown): “The usage of mutual information also aligns well with the information bottleneck framework (Shamir et al., 2010; ShwartzZiv and Tishby, 2017; Yu and Principe, 2018), although a more recent paper challenges the interpretation of information bottleneck (Saxe et al.”
Predictive Information Accelerates Learning in RL (2020, background): “It is well-known that compressed representations can generalize better in many machine learning settings [36, 3, 8], including RL [20, 12, 11].”