Automated Scalable Bayesian Inference via Hilbert Coresets

By Trevor Campbell, Tamara Broderick. 2017

138 citations recorded by the source corpus.

The automation of posterior inference in Bayesian data analysis has enabled experts and nonexperts alike to use more sophisticated models, engage in faster exploratory modeling and analysis, and ensure experimental reproducibility. However, standard automated posterior inference algorithms are not tractable at the scale of massive modern datasets, and modifications to make them so are typically model-specific, require expert tuning, and can break theoretical guarantees on inferential quality. Building on the Bayesian coresets framework, this work instead takes advantage of data redundancy to shrink the dataset itself as a preprocessing step, providing fully-automated, scalable Bayesian inference with theoretical guarantees. We begin with an intuitive reformulation of Bayesian coreset construction as sparse vector sum approximation, and demonstrate that its automation and performance-based shortcomings arise from the use of the supremum norm. To address these shortcomings we develop Hilbert coresets, i.e., Bayesian coresets constructed under a norm induced by an inner-product on the log-likelihood function space. We propose two Hilbert coreset construction algorithms---one based on importance sampling, and one based on the Frank-Wolfe algorithm---along with theoretical guarantees on approximation quality as a function of coreset size. Since the exact computation of the proposed inner-products is model-specific, we automate the construction with a random finite-dimensional projection of the log-likelihood functions. The resulting automated coreset construction algorithm is simple to implement, and experiments on a variety of models with real and synthetic datasets show that it provides high-quality posterior approximations and a significant reduction in the computational cost of inference.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Information-theoretic Online Memory Selection for Continual Learning (2022, neutral): “Coreset selection (Campbell & Broderick, 2019) aims to find a weighted set whose maximum likelihood estimator approximates that of the whole dataset.”
GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, neutral): “Coresets enable efﬁcient and scalable Bayesian inference [30, 31, 32, 33, 34] by approximating the model logit sum of the entire data.”
Online Coreset Selection for Rehearsal-based Continual Learning (2021, critical): “It is worth noting that all the rehearsal-based baselines that utilize their coreset selection criteria for current task adaptation decrease the performance (1.0 - 9.8%p ↓), except Grad Matching (0.5%p ↑) on Balanced Rotated MNIST.”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, neutral): “Several approaches relax the uniform approximation guarantees: the Hilbert coreset [7] which poses the subset selection problem as a vector approximation in a normed vector space, and data selection techniques that were successfully employed in active learning with convolutional neural networks [56,”
Efficient posterior sampling for high-dimensional imbalanced logistic regression. (2019, background): “The Bayesian literature has largely overlooked the use of biased sub-sampling in efficient algorithm design, though recent coreset approaches address a related problem [18, 9].”
Explainable artificial intelligence: a comprehensive review (2021, background): “The recent work on so-called Bayesian coresets is a typical example of the data squashing that is expressed in the Bayesian learning environment (Campbell and Broderick 2019).”
Sparse Variational Inference: Bayesian Coresets from Scratch (2019, methodology): “Further, even if an appropriate inner product is speciﬁed, computing it exactly is typically intractable, requiring the use of ﬁnite-dimensional projections for approximation [31].”
Bayesian Coresets: Revisiting the Nonconvex Optimization Perspective (2021, methodology): “The experiment was proposed by Campbell & Broderick (2019), and is used in (Campbell & Broderick, 2018) (GIGA) and (Campbell & Beronov, 2019) (SparseVI).”
Selection Via Proxy: Efficient Data Selection For Deep Learning (2019, background): “…selection attempts to ﬁnd a representative subset of points to speed up learning or clustering; such as k -means and k -medians [Har-Peled and Kushal, 2007], SVM [Tsang et al., 2005], Bayesian logistic regression [Huggins et al., 2016], and Bayesian inference [Campbell and Broderick, 2017, 2018].”