Input complexity and out-of-distribution detection with likelihood-based generative models
By Joan Serrà, David Alvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, Jordi Luque. International Conference on Learning Representations, 2019
315 citations recorded by the source corpus.
Likelihood-based generative models are a promising resource to detect out-of-distribution (OOD) inputs which could compromise the robustness or reliability of a machine learning system. However, likelihoods derived from such models have been shown to be problematic for detecting certain types of inputs that significantly differ from training data. In this paper, we pose that this problem is due to the excessive influence that input complexity has in generative models' likelihoods. We report a set of experiments supporting this hypothesis, and use an estimate of input complexity to derive an efficient and parameter-free OOD score, which can be seen as a likelihood-ratio, akin to Bayesian model comparison. We find such score to perform comparably to, or even better than, existing OOD detection approaches under a wide range of data sets, models, model sizes, and complexity estimates.
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/1322719978980a831e1aee78aa80a141379c44dd
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “[171] finds that the likelihood exhibits a strong bias towards the input complexity and proposes a likelihood ratio-based method to compensate the influence of input complexity.”
- CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, methodology): “Several work thus proposed modified scores, e . g ., WAIC [7], likelihood ratio [52], and input complexity [57], or utilized unnormalized likelihood ( i . e ., energy) [12, 17].”
- Hybrid Models for Open Set Recognition (2020, background): “The open set recognition is naturally related to some other problem settings such as out-of-distribution detection [36, 32, 18], outlier detection [28], and novelty detection [25], etc.”
- Unsupervised Class-Incremental Learning Through Confusion (2021, methodology): “To address this problem, research related to out-of-distribution (OOD) detection have utilized supervised methods (Liang et al. (2017); Alemi et al. (2018)) and unsupervised methods (Choi & Jang (2018); Hendrycks et al. (2018); Serrà et al. (2019)).”
- CMG: A Class-Mixed Generation Approach to Out-of-Distribution Detection (2022, background): “Many other score functions have also been proposed, e.g., likelihood ratio [46], input complexity [50] and typicality [37].”
- Semantically Coherent Out-of-Distribution Detection (2021, methodology): “Existing OOD detection methods mostly focus on calibrating the distribution of the softmax layer [4] through temperature scaling [7], generative models [8, 9], or ensemble methods [10, 11].”
- Why Normalizing Flows Fail to Detect Out-of-Distribution Data (2020, background): “While there has been empirical progress in improving OOD detection with flows [29, 7, 30, 39, 40, 48], the fundamental reasons for why flows fail at OOD detection in the first place are not fully understood.”
- Further Analysis of Outlier Detection with Deep Generative Models (2020, unknown): “In Section 3 we have introduced the work of [20], and demonstrated that its assumption does not always hold.”
- On the Importance of Gradients for Detecting Distributional Shifts in the Wild (2021, background): “A plethora of literature has emerged to utilize generative models for OOD detection [19, 37, 39, 40, 45, 46, 48].”
- Hands-On Bayesian Neural Networks—A Tutorial for Deep Learning Users (2020, background): “, Generative Adversarial Networks) meant to encode complex data distributions [79].”