Variational Relevance Vector Machines

By Charles M. Bishop, Michael E. Tipping. Conference on Uncertainty in Artificial Intelligence, 2000

434 citations recorded by the source corpus.

The Support Vector Machine (SVM) of Vapnik (1998) has become widely established as one of the leading approaches to pattern recognition and machine learning. It expresses predictions in terms of a linear combination of kernel functions centred on a subset of the training data, known as support vectors. Despite its widespread success, the SVM suffers from some important limitations, one of the most significant being that it makes point predictions rather than generating predictive distributions. Recently Tipping (1999) has formulated the Relevance Vector Machine (RVM), a probabilistic model whose functional form is equivalent to the SVM. It achieves comparable recognition accuracy to the SVM, yet provides a full predictive distribution, and also requires substantially fewer kernel functions. The original treatment of the RVM relied on the use of type II maximum likelihood (the `evidence framework') to provide point estimates of the hyperparameters which govern model sparsity. In this paper we show how the RVM can be formulated and solved within a completely Bayesian paradigm through the use of variational inference, thereby giving a posterior distribution over both parameters and hyperparameters. We demonstrate the practicality and performance of the variational RVM using both synthetic and real world examples.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1b4c906704cd125adf5c9a974e16c56e49925bf3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A physics-aware, probabilistic machine learning framework for coarse-graining high-dimensional systems in the Small Data regime (2019, methodology): “This hierarchical model has been widely studied and the capability of pruning irrelevant parameters θc,jm has been shown when small values for the hyperparameters are employed 2 (Bishop and Tipping, 2000; Tipping, 2001).”
Exploiting Structure in Wavelet-Based Bayesian Compressive Sensing (2009, methodology): “The BCS algorithm can be implemented via the relevance vector machines (RVM) [15]; we implemented it using a variational RVM [43].”
Analysis of Sparse Bayesian Learning (2001, methodology): “…can be plugged into (6) to give a posterior mean (most probabletpoint estimate for the parameters J.tMP and from that a mean final approximator: fMP = ()J.tMp· Empirically, the local maximisation of the marginal likelihood (8) with respect to a has been seen to work highly effectively [6] [1] [7].”
The variational approximation for Bayesian inference (2008, background): “The approximate posterior distributions in (48), (52), and (53) are then iteratively updated until convergence, since they depend on the statistics of each other, see [22] for details.”
Fast Marginal Likelihood Maximisation for Sparse Bayesian Models (2003, neutral): “More general hyperpriors are considered in [11, 1].”
A Bayesian Model for Simultaneous Image Clustering, Annotation and Object Segmentation (2009, methodology): “Such a Student-t prior is also applied in [4].”
Variational EM Algorithms for Non-Gaussian Latent Variable Models (2005, methodology): “In the ensemble learning approach (also Variational Bayes [4, 3, 6]) the idea is to ﬁnd the approximate separable posterior that minimizes the KL divergence from the true posterior, using the following decomposition of the log likelihood, log p ( y ) = q ( z | y ) log p ( z , y ) q ( z | y ) d z + D”
Sparse Bayesian Learning and the Relevance Vector Machine (2001, methodology): “An alternative approach is to iteratively maximise a variational lower bound on p(t), via a factorised approximation to p(w; ; (2)jt), the joint posterior distribution over all the model parameters (Bishop and Tipping, 2000).”
Machine Learning Techniques for Downscaling SMOS Satellite Soil Moisture Using MODIS Land Surface Temperature for Hydrological Application (2013, methodology): “used in this study which is found to have some advantages over SVM (Foody 2008; Bishop and Tipping 2000; Ghosh and Mujumdar 2008; Tipping 2001).”
A unified Bayesian framework for MEG/EEG source imaging (2009, positive): “(55) assuming that Σ s is parameterized via X s ¼ X ds i¼1 g i e i e i þ X ds j¼1 g dsþj ð Þ w j w T j ; ð62Þ and so dγ = 2ds. 16 When fMRI data is available, it is incorporated into a conjugate inverse Gamma hyperprior on γ, as is also commonly done with γ-MAP methods (Bishop and Tipping, 2000).”
Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns (2008, neutral): “One approach to overcome this computational problem is direct approximation of the logistic function using a variational parameter (Jaakkola and Jordan, 2000; Bishop and Tipping, 2000).”
Sparse Variational Analysis of Linear Mixed Models for Large Data Sets. (2011, methodology): “One possibility is to rely on a variational approximation to the posterior distribution (Jordan et al., 1999; Jaakkola and Jordan, 2000; Bishop and Tipping, 2000).”
Time-varying sparsity in dynamic regression models (2014, positive): “This interpretation of the variance of a normal prior distribution for a regression coefficient dates back to Tipping (2000) and Bishop and Tipping (2000). Since ψi,t and φi,t are independent and stationary, the process βi is also stationary and has a Normal-Gamma stationary distribution. The uncond”
Estimation of monthly evaporative loss using relevance vector machine, extreme learning machine and multivariate adaptive regression spline models (2016, methodology): “It is imperative to note that the RVM model enjoys the same benefits as the SVM model but in addition, the RVM model provides some estimates of uncertainty bounds which are crucial for performance assessment (Bishop and Tipping 2000; Samui and Dixon 2012).”
The Bayesian backfitting relevance vector machine (2004, positive): “…of sparse Bayesian learning and the introduction of prior distributions over the precisions (cid:11) i of each element of b : Approximate analytical solutions for the RVM can be obtained by the Laplace method (Tipping, 2001) or by using factorial variational approximations (Bishop & Tipping, 2000).”
Generalized Beta Mixtures of Gaussians (2011, background): “Using a similar approach to [5, 1], the approximate marginal posterior distributions of the parameters are given by β ∼ N (μβ ,Vβ), σ−2 ∼ G (c∗, d∗), τj ∼ GIG(a−1/2, 2〈λj〉, 〈σ−2〉〈β2 j 〉), λj ∼ G(a+b, 〈τj〉+〈φ〉), φ ∼ G(pb+1/2, 〈ω〉+ ∑p j=1〈λj〉), ω ∼ G(1, 〈φ〉 + 1), where μβ = 〈β〉 = (X′X + T−1)−1X′y, Vβ ”
Semi-Implicit Variational Inference (2018, methodology): “The optimization problem is further simpliﬁed if each latent variable’s distribution is in the same exponential family as its prior, which allows exploiting conditional conjugacy to derive closed-form coordinate-ascent update equations (Bishop & Tipping, 2000; Blei et al., 2017).”
Bayesian Compressive Sensing (2008, methodology): “For example, one may conveniently implement a Markov chain Monte Carlo (MCMC) [17] or, more efficiently and approximately, a variational Bayesian (VB) analysis [18].”