Adaptive Sparseness for Supervised Learning

By Mário A. T. Figueiredo. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003

599 citations recorded by the source corpus.

The goal of supervised learning is to infer a functional mapping based on a set of training examples. To achieve good generalization, it is necessary to control the "complexity" of the learned function. In Bayesian approaches, this is done by adopting a prior for the parameters of the function being learned. We propose a Bayesian approach to supervised learning, which leads to sparse solutions; that is, in which irrelevant parameters are automatically set exactly to zero. Other ways to obtain sparse classifiers (such as Laplacian priors, support vector machines) involve (hyper)parameters which control the degree of sparseness of the resulting classifiers; these parameters have to be somehow adjusted/estimated from the training data. In contrast, our approach does not involve any (hyper)parameters to be adjusted or estimated. This is achieved by a hierarchical-Bayes interpretation of the Laplacian prior, which is then modified by the adoption of a Jeffreys' noninformative hyperprior. Implementation is carried out by an expectation-maximization (EM) algorithm. Experiments with several benchmark data sets show that the proposed approach yields state-of-the-art performance. In particular, our method outperforms SVMs and performs competitively with the best alternative techniques, although it involves no tuning or adjustment of sparseness-controlling hyperparameters.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/3eaf21ba64b0775b68ebadbafe064bb619b75ca6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Penalized regression, standard errors, and Bayesian lassos (2010, methodology): “Figueiredo (2003) used the Laplace prior to obtain sparsity in supervised learning using an EM algorithm.”
Shrink Globally, Act Locally: Sparse Bayesian Regularization and Prediction (2012, background): “See, for example, West [1987], Carlin and Polson [1991], Pericchi and Smith [1992], Tibshirani [1996], Park and Casella [2008], and Hans [2009] Normal/Jeffreys, where p(βi) ∝ |βi| −1 [Figueiredo, 2003, Bae and Mallick, 2004].”
GENERALIZED DOUBLE PARETO SHRINKAGE. (2011, methodology): “The name emerges due to the fact that the hierarchy follows as θ ∼ N(0, τ), π(τ) ∝ 1/τ , where the latter is the Jeffreys’ prior on the prior variance of θ. Figueiredo (2003) proposed an expectation-maximization algorithm for maximum a posteriori estimation under Laplace and Normal-Jeffreys’…”
Multiclass Object Recognition with Sparse, Localized Features (2006, background): “Within machine learning, it has been found that increasing the sparsity of basis functions [7, 14] (equivalent to reducing the capacity of the classifier) plays an important role in improving generalization performance.”
Geometric ergodicity of the Bayesian lasso (2013, unknown): “Based on this observation, several authors proposed a Bayesian analysis using a Laplace-like prior for β (see for example [3, 7, 20]).”
Sparse multinomial logistic regression: fast algorithms and generalization bounds (2005, methodology): “This family of algorithms includes the relevance vector machine (RVM) [35], the sparse probit regression (SPR) algorithm [10], [11], sparse online Gaussian processes [6], the informative vector machine (IVM) [22], and the joint classifier and feature optimization (JCFO) algorithm [16], [17].”
Cross-Domain Multitask Learning with Latent Probit Models (2012, methodology): “…hierarchical Bayesian model, referred to as the latent probit model (LPM), which jointly represents the sparse domain transforms and a common sparse probit classifier (Albert & Chib, 1993) in the latent feature space, with the sparsity imposed by a hierarchical Laplacian prior (Figueiredo, 2003).”
BAYESIAN HYPER‐LASSOS WITH NON‐CONVEX PENALIZATION (2011, methodology): “Local posterior modes can be found using the EM algorithm (Dempster, Laird & Rubin 1977; Meng & van Dyk 1997), which has been suggested by both Figueiredo (2003) and Kiiveri (2003) as a means of fitting models using scale mixture of normal priors.”
Local shrinkage rules, Lévy processes and regularized regression (2010, unknown): “Address for correspondence: James G. Scott, Department of Information, Risk, and Operations Management, University of Texas at Austin, 1 University Station, B6500, Austin, TX 78712, USA.”
Large-Scale Bayesian Logistic Regression for Text Categorization (2007, methodology): “Since then, the use of constraints or penalties based on the absolute values of coefficients has been used to achieve sparseness in logistic regression and many other data-fitting tasks (Girosi 1998; Tipping 2001; Figueiredo and Jain 2001; Figueiredo 2003; Efron, Hastie, Johnstone, and Tibshirani 20”
Sparse Additive Generative Models of Text (2011, background): “The Laplace distribution L ( η ; m, σ ) is equivalent to a compound model, (cid:82) N ( η ; m, τ ) E ( τ ; σ ) dτ , where E ( τ ; σ ) indicates the Exponential distribution (Lange & Sinsheimer, 1993; Figueiredo, 2003).”
Bayesian Model and Dimension Reduction for Uncertainty Propagation: Applications in Random Media (2017, methodology): “he underlying physics of the problem. Several sparsity enforcing approaches were tested in this work, including the Laplacian prior (or LASSO regression [71,26]) as well as Student-ttype prior models [45,55,22]. We achieved best experimental results using a slightly modied version of the Relevance V”
Inference with normal-gamma prior distributions in regression problems (2010, methodology): “Bayesian estimation of regression models with other absolutely continuous priors has been investigated by several authors (MacKay 1996; Tipping 2001; Figueiredo and Jain 2001; Figueiredo 2003; Kiiveri 2008; Caron and Doucet 2008).”
Generalized Beta Mixtures of Gaussians (2011, methodology): “This is accomplished in a similar manner to [8] by obtaining the joint MAP estimates of the error variance and the regression coefficients having taken the expectation with respect to the conditional posterior distribution of τ−1 j using the second hierarchy given in Proposition 1.”
A New TwIST: Two-Step Iterative Shrinkage/Thresholding Algorithms for Image Restoration (2007, methodology): “Each iteration of IRS resembles a weighted ridge regression problem, with design matrixK; algorithms with a similar structure have been used for sparse regression [25], [31].”