Coresets for Scalable Bayesian Logistic Regression

By Jonathan Huggins, Trevor Campbell, Tamara Broderick. 2016

236 citations recorded by the source corpus.

The use of Bayesian methods in large-scale data settings is attractive because of the rich hierarchical models, uncertainty quantification, and prior specification they provide. Standard Bayesian inference algorithms are computationally expensive, however, making their direct application to large datasets difficult or infeasible. Recent work on scaling Bayesian inference has focused on modifying the underlying algorithms to, for example, use only a random data subsample at each iteration. We leverage the insight that data is often redundant to instead obtain a weighted subset of the data (called a coreset) that is much smaller than the original dataset. We can then use this small coreset in any number of existing posterior inference algorithms without modification. In this paper, we develop an efficient coreset construction algorithm for Bayesian logistic regression models. We provide theoretical guarantees on the size and approximation quality of the coreset -- both for fixed, known datasets, and in expectation for a wide class of data generative models. Crucially, the proposed approach also permits efficient construction of the coreset in both streaming and parallel settings, with minimal additional effort. We demonstrate the efficacy of our approach on a number of synthetic and real-world datasets, and find that, in practice, the size of the coreset is independent of the original dataset size. Furthermore, constructing the coreset takes a negligible amount of time compared to that required to run MCMC on it.

Subfields: Artificial Intelligence, Statistics and Probability, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Variational Continual Learning (2017, methodology): “In addition, we extend VCL to include a small episodic memory by combining VI with the coreset data summarization method (Bachem et al., 2015; Huggins et al., 2016).”
GCR: Gradient Coreset based Replay Buffer Selection for Continual Learning (2021, neutral): “Coresets enable efﬁcient and scalable Bayesian inference [30, 31, 32, 33, 34] by approximating the model logit sum of the entire data.”
Coresets via Bilevel Optimization for Continual Learning and Streaming (2020, methodology): “, coresets for k-means [19], Gaussian mixture model [41] and logistic regression [29].”
BooVAE: Boosting Approach for Continual Learning of VAE (2019, background): “This procedure can also be seen as coreset selection (Huggins et al., 2016; Bachem et al., 2015).”
Facilitating Bayesian Continual Learning by Natural Gradients and Stein Gradients (2019, methodology): “There are some existing approaches to Bayesian coreset construction for scalable machine learning [7, 8], the idea is to find a sparse weighted subset of data to approximate the likelihood over the whole dataset.”
Subspace Inference for Bayesian Deep Learning (2019, methodology): “…a priori; on the other hand, we choose the constraints (e.g. the subspace) after performing unconstrained inference via SGD. Bayesian coresets (Huggins et al., 2016) use a weighted combination of the full dataset and Bayesian compressed regression (Guhaniyogi and Dunson, 2015) uses random…”
Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds (2018, methodology): “Coresets have been used successfully to accelerate various machine learning algorithms such as k-means clustering [20, 9], graphical model training [38], and logistic regression [23] (see the surveys of [6] and [39] for a complete list).”
Practical Coreset Constructions for Machine Learning (2017, methodology): “Boutsidis et al. [2012] Huggins et al. [2016] have shown that the upper bound on sensitivity computed by k -clustering is useful in the context of Bayesian Logistic Regression.”
Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications (2019, background): “Prototypes of a given task share some intuition with the notion of coresets (Agarwal et al., 2005; Huggins et al., 2016; Bachem et al., 2017; Tolochinsky & Feldman, 2018) because both prototypes and coresets describe the dataset in a more compact way—by returning a (potentially weighted) subset of t”
A Survey of Deep Active Learning (2020, methodology): “Inspired by the latest research on Bayesian core sets [33, 99], ACS-FW (Active Bayesian CoreSets with Frank-Wolfe optimization) [162] reconstructed the batch structure to optimize the sparse subset approximation of the log-posterior induced by the entire dataset.”
Metadata Archaeology: Unearthing Data Subsets by Leveraging Training Dynamics (2022, background): “…corpus of data (Zhang, 1992; Bien and Tibshirani, 2012; Kim et al., 2015; Kim et al., 2016), which can be used to speed up training (Sener and Savarese, 2018; Shim et al., 2021; Huggins et al., 2017; Sorscher et al., 2022) or aid in interpretability of the model predictions (Yoon et al., 2019).”
Selection Via Proxy: Efficient Data Selection For Deep Learning (2019, background): “…selection attempts to ﬁnd a representative subset of points to speed up learning or clustering; such as k -means and k -medians [Har-Peled and Kushal, 2007], SVM [Tsang et al., 2005], Bayesian logistic regression [Huggins et al., 2016], and Bayesian inference [Campbell and Broderick, 2017, 2018].”
On Last-Layer Algorithms for Classification: Decoupling Representation from Uncertainty Estimation (2020, methodology): “Third, to consider methods for variational inference and Laplace approximations, e.g. [RBB18], and alternative methods for Bayesian logistic regression on large datasets since the focus is on last layer Bayesian approaches using the features learned from deep neural networks, e.g. [HCB16, HB15].”
Distributed Weight Consolidation: A Brain Segmentation Case Study (2018, unknown): “VCL [21] performance was found to be improved by using coresets [1, 12], a small sample of data from the different training sets.”
Efficient posterior sampling for high-dimensional imbalanced logistic regression. (2019, background): “The Bayesian literature has largely overlooked the use of biased sub-sampling in efficient algorithm design, though recent coreset approaches address a related problem [18, 9].”
Bayesian Coresets: Revisiting the Nonconvex Optimization Perspective (2021, unknown): “Bayesian coresets have emerged as a promising approach for scalable Bayesian inference (Huggins et al., 2016; Campbell & Broderick, 2018, 2019; Campbell & Beronov, 2019).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, methodology): “Inspired from [111], the Bayesian batch AL [112], which selects a batch of samples at each AL iteration to perform posterior inference over the model parameters, was pro-”