Likelihood Ratios for Out-of-Distribution Detection

By Jie Jessie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua V. Dillon, Balaji Lakshminarayanan. 2019

840 citations recorded by the source corpus.

Discriminative neural networks offer little or no performance guarantees when deployed on data not generated by the same process as the training distribution. On such out-of-distribution (OOD) inputs, the prediction may not only be erroneous, but confidently so, limiting the safe deployment of classifiers in real-world applications. One such challenging application is bacteria identification based on genomic sequences, which holds the promise of early detection of diseases, but requires a model that can output low confidence predictions on OOD genomic sequences from new bacteria that were not present in the training data. We introduce a genomics dataset for OOD detection that allows other researchers to benchmark progress on this important problem. We investigate deep generative model based approaches for OOD detection and observe that the likelihood score is heavily affected by population level background statistics. We propose a likelihood ratio method for deep generative models which effectively corrects for these confounding background statistics. We benchmark the OOD detection performance of the proposed method against existing approaches on the genomics dataset and show that our method achieves state-of-the-art performance. We demonstrate the generality of the proposed method by showing that it significantly improves OOD detection when applied to deep generative models of images.

Subfields: Computer Vision and Pattern Recognition, Molecular Biology, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Molecular Biology

In Safety, Robustness & Interpretability

Citations in the corpus

GAN Inversion: A Survey (2021, background): “We refer to this ability as out-of-distribution generalizability [110], [111], [112].”
Simple and Scalable Epistemic Uncertainty Estimation Using a Single Deep Deterministic Neural Network (2020, methodology): “Lee et al. (2018), LL ratio by Ren et al. (2019).”
Towards neural networks that provably know when they don't know (2019, methodology): “Following Ren et al. (2019) we use a single-layer version of Lee et al. (2018c) on our networks’ penultimate layers because the multi-layer version in the original code does not support gradient-based attacks.”
Out-of-Domain Detection for Natural Language Understanding in Dialog Systems (2019, methodology): “These perturbed samples are constructed in a similar way as in the previous Likelihood Ratio baseline.”
Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift (2019, methodology): “• We also compared the set of methods on a real-world challenging genomics problem from Ren et al. (2019).”
Energy-based Out-of-distribution Detection (2020, background): “These approaches are stronger baselines than pure generative-modeling-based OOD detection methods [4, 28, 33], due to the use of labeling information during training.”
Generalized Out-of-Distribution Detection: A Survey (2021, neutral): “Several works attempt to solve the problems using likelihood ratio (Ren et al., 2019).”
Widar3.0: Zero-Effort Cross-Domain Gesture Recognition With Wi-Fi (2021, methodology): “4) The Likeli-hood-ratio method [64] contrasts the likelihood of a generative model against a background model and employs the likelihood ratio between the two models as outlier indicator.”
CODEs: Chamfer Out-of-Distribution Examples against Overconfidence Issue (2021, methodology): “Recent works improve OOD detection by using the ODIN score [31, 23], Mahalanobis distance [28], energy score [32], ensemble of multiple classifiers [42, 48], residual flow [51], generative models [38], self-supervised learning [20, 35] and gram matrices [39].”
Surprisal-Triggered Conditional Computation with Neural Networks (2020, background): “Surprisal-based models and autoregressive models Surprisal is a useful notion for many tasks, such as anomaly detection [43] and comparing the difﬁculty of modeling different languages [44].”
The Pitfalls of Simplicity Bias in Neural Networks (2020, background): “Recent works thus aim to detect OOD examples using generative models [54], statistical tests [56], and model confidence scores [28, 38, 37].”
Detecting Out-of-Distribution Inputs to Deep Generative Models Using Typicality (2019, methodology): “[46] use likelihood ratios, but they do not perform a hypothesis test.”
Generalized ODIN: Detecting Out-of-Distribution Image Without Learning From Out-of-Distribution Data (2020, background): “The selection is inspired by the ﬁnding in the line of works that uses a generative model for OoD detection [32, 26, 4].”
Anomalous Instance Detection in Deep Learning: A Survey (2020, methodology): “The National Center for Biotechnology Information microbial genome dataset is utilized in [23] in the following manner.”
MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Space (2021, background): “Several strategies have been proposed to mitigate this issue, such as improved metrics [5], likelihood ratio [39, 42], and modified training techniques [17].”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “Unlike OOD generalization tasks, which primarily focus on performance under distribution shifts, the OOD detection community [211, 152, 98, 244] concentrates more on detecting samples from unseen distributions.”
Bayesian Variational Autoencoders for Unsupervised Out-of-Distribution Detection (2019, methodology): “2 for the FashionMNIST vs. MNIST experiment (Ren et al., 2019), and µ = 0 .”
Deep Learning for Anomaly Detection (2020, methodology): “ome exciting emerging research applications and problem settings, into which there could be some important opportunities of extending deep detection methods. First, outof-distribution (OOD) detection [61, 80, 128] is another closely related area, which detects data instances that are anomalous or si”