Sketching Curvature for Efficient Out-of-Distribution Detection for Deep Neural Networks
By Apoorva Sharma, Navid Azizan, Marco Pavone. 2021
54 citations recorded by the source corpus.
In order to safely deploy Deep Neural Networks (DNNs) within the perception pipelines of real-time decision making systems, there is a need for safeguards that can detect out-of-training-distribution (OoD) inputs both efficiently and accurately. Building on recent work leveraging the local curvature of DNNs to reason about epistemic uncertainty, we propose Sketching Curvature of OoD Detection (SCOD), an architecture-agnostic framework for equipping any trained DNN with a task-relevant epistemic uncertainty estimate. Offline, given a trained model and its training data, SCOD employs tools from matrix sketching to tractably compute a low-rank approximation of the Fisher information matrix, which characterizes which directions in the weight space are most influential on the predictions over the training data. Online, we estimate uncertainty by measuring how much perturbations orthogonal to these directions can alter predictions at a new test input. We apply SCOD to pre-trained networks of varying architectures on several tasks, ranging from regression to classification. We demonstrate that SCOD achieves comparable or better OoD detection performance with lower computational burden relative to existing baselines.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Erik A. Daxberger
- Youngjae Min
- Agustinus Kristiadi
- Kwangjun Ahn
- Matthias Bauer
- Runa Eschenhagen
- Alexander Immer
- Navid Azizan
Citations in the corpus
- One-Pass Learning via Bridging Orthogonal Gradient Descent and Recursive Least-Squares (2022, positive): “The main idea behind the NTK regime is that when the width of the neural network is chosen large enough, the model is well-approximated by its first-order approximation around the initialization: In particular, Lee et al. [11] discussed sufficient conditions (in terms of the width of the network) for…”
- SketchOGD: Memory-Efficient Continual Learning (2023, unknown): “Sketching has also been successfully employed for summarizing the curvature of neural networks for the purpose of out-of-distribution detection [30], motivating its application to OGD.”
- Laplace Redux - Effortless Bayesian Deep Learning (2021, background): “Low-rank Instead of only approximating each block by a low-rank structure, the entire Hessian or GGN can also be approximated by a low-rank structure [48, 47].”