Caveats for information bottleneck in deterministic scenarios
By Artemy Kolchinsky, Brendan D. Tracey, Steven Van Kuyk. 2018
94 citations recorded by the source corpus.
Information bottleneck (IB) is a method for extracting information from one random variable $X$ that is relevant for predicting another random variable $Y$. To do so, IB identifies an intermediate "bottleneck" variable $T$ that has low mutual information $I(X;T)$ and high mutual information $I(Y;T)$. The "IB curve" characterizes the set of bottleneck variables that achieve maximal $I(Y;T)$ for a given $I(X;T)$, and is typically explored by maximizing the "IB Lagrangian", $I(Y;T) - \beta I(X;T)$. In some cases, $Y$ is a deterministic function of $X$, including many classification problems in supervised learning where the output class $Y$ is a deterministic function of the input $X$. We demonstrate three caveats when using IB in any situation where $Y$ is a deterministic function of $X$: (1) the IB curve cannot be recovered by maximizing the IB Lagrangian for different values of $\beta$; (2) there are "uninteresting" trivial solutions at all points of the IB curve; and (3) for multi-layer classifiers that achieve low prediction error, different layers cannot exhibit a strict trade-off between compression and prediction, contrary to a recent proposal. We also show that when $Y$ is a small perturbation away from being a deterministic function of $X$, these three caveats arise in an approximate way. To address problem (1), we propose a functional that, unlike the IB Lagrangian, can recover the IB curve in all cases. We demonstrate the three caveats on the MNIST dataset.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Medhi Bennis
- Jihong Park
- Sumudu Samarakoon
- Merouane Debbah
- Benjamin Bloem-Reddy
- Nam H. Nguyen
- Brian Kingsbury
- Ziv Goldfeld
Citations in the corpus
- Wireless Network Intelligence at the Edge (2018, background): “7(b)] [92] and the training objective function [93], which can be partly rectified by modifying the objective function and/or inserting noise [94].”
- Estimating Information Flow in Neural Networks (2018, background): “Both continuous and discrete degeneracies were previously observed (Amjad & Geiger, 2018; Kolchinsky et al., 2019).”
- Lossy Compression for Lossless Prediction (2021, background): “3), it should be beneficial to use R[M(X) |Z](2) to ensure that sweeping over β is equivalent to sweeping over δ [22].”