Uncertainty in the Variational Information Bottleneck

By Alexander A. Alemi, Ian S. Fischer, Joshua V. Dillon. 2018

112 citations recorded by the source corpus.

We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-of-distribution data. Without explicitly being designed to do so, VIB gives two natural metrics for handling and quantifying uncertainty.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (2018, background): “[2] motivate use of the variational information bottleneck.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “Methods which model the joint distribution p ( y, x ) , e.g. deep hybrid models (Kingma et al., 2014; Alemi et al., 2018; Nalisnick et al., 2019; Behrmann et al., 2018).”
Preventing Posterior Collapse with delta-VAEs (2019, methodology): “(2017); Chen et al. (2017); Parmar et al. (2018). Unlike prior models, we use sequential latent variables to generate the image row by row. This differs from Gregor et al. (2016), where the latent variables are sequential but the entire image is generated at each timestep.”
Likelihood Ratios for Out-of-Distribution Detection (2019, methodology): “These three metrics are commonly used for evaluating OOD detection methods (Hendrycks & Gimpel, 2016; Hendrycks et al., 2018; Alemi et al., 2018).”
Hybrid Models with Deep and Invertible Features (2019, unknown): “It is not surprising then that many previous approaches have had to rely on approximate inference (Alemi et al., 2018; Kingma et al., 2014) for efficient model fitting.”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, unknown): “…held-out test sets and computing the following threshold independent metrics (Hendrycks & Gimpel, 2016; Liang et al., 2017; Hendrycks et al., 2018; Alemi et al., 2018; Ren et al., 2019): (i) The area under the ROC curve (AUROC ↑ ), (ii) the area under the precision-recall curve (AUPRC ↑ ), and…”
Generative Ensembles for Robust Anomaly Detection (2018, methodology): “Variational information bottleneck (VIB) (Alemi et al., 2018) performs divergence estimation in latent space to detect OoD, but is still a model-dependent technique because the latent code is trained jointly with the downstream classification task.”
Further Analysis of Outlier Detection with Deep Generative Models (2020, unknown): “There is also work on outlier detection in supervised learning tasks, where auxiliary label information is available; see, e.g. [26–32].”
TabTransformer: Tabular Data Modeling Using Contextual Embeddings (2020, methodology): “For VIB model, we implemented it as described in Alemi, Fischer, and Dillon (2018).”
Unsupervised Class-Incremental Learning Through Confusion (2021, methodology): “To address this problem, research related to out-of-distribution (OOD) detection have utilized supervised methods (Liang et al. (2017); Alemi et al. (2018)) and unsupervised methods (Choi & Jang (2018); Hendrycks et al. (2018); Serrà et al. (2019)).”
Input complexity and out-of-distribution detection with likelihood-based generative models (2019, methodology): “A number of methods have been proposed to perform OOD detection under a classiﬁcation-based framework (Hendrycks & Gimpel, 2017; Lakshminarayanan et al., 2017; Liang et al., 2018; Alemi et al., 2018; Lee et al., 2018; Hendrycks et al., 2019).”
Caveats for information bottleneck in deterministic scenarios (2018, unknown): “…the amount of compression, one can tune desired characteristics of trained models such as generalization error (Shamir et al., 2010; Tishby & Zaslavsky, 2015; Vera et al., 2018), robustness to adversarial inputs (Alemi et al., 2016), and detection of out-of-distribution data (Alemi et al., 2018).”
Learning Representations for Neural Network-Based Classification Using the Information Bottleneck Principle (2018, methodology): “4 is relevant and valid, but also that the promising performance of [5], [6], [8], [9], [27] in terms of various operational goals can at least partly be attributed to the remedies we propose in Sec.”
WAIC, but Why? Generative Ensembles for Robust Anomaly Detection (2018, methodology): “Setting β 1 is a commonly prescribed fix for encouraging VAEs to approach the “autoencoding limit” and avoid posterior collapse (Alemi et al., 2018a).”
Unsupervised Multi-Target Domain Adaptation: An Information Theoretic Approach (2018, result): “Similar to our approach, there have been some recent works [ 3 – 5 , 12 , 32 ] to approximate the MI by applying variational bounds on MI , though not in the context of domain adaptation.”