Expectation propagation for neural networks with sparsity-promoting priors

By Pasi Jylänki, Aapo Nummenmaa, Aki Vehtari. 2013

38 citations recorded by the source corpus.

We propose a novel approach for nonlinear regression using a two-layer neural network (NN) model structure with sparsity-favoring hierarchical priors on the network weights. We present an expectation propagation (EP) approach for approximate integration over the posterior distribution of the weights, the hierarchical scale parameters of the priors, and the residual scale. Using a factorized posterior approximation we derive a computationally effcient algorithm, whose complexity scales similarly to an ensemble of independent sparse linear models. The approach enables flexible definition of weight priors with different sparseness properties such as independent Laplace priors with a common scale parameter or Gaussian automatic relevance determination (ARD) priors with different relevance parameters for all inputs. The approach can be extended beyond standard activation functions and NN model structures to form flexible nonlinear predictors from multiple sparse linear models. The effects of the hierarchical priors and the predictive performance of the algorithm are assessed using both simulated and real-world data. Comparisons are made to two alternative models with ARD priors: a Gaussian process with a NN covariance function and marginal maximum a posteriori estimates of the relevance parameters, and a NN with Markov chain Monte Carlo integration over all the unknown model parameters.

Subfields: Analytical Chemistry, Artificial Intelligence, Control and Systems Engineering, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Analytical Chemistry

In Artificial Intelligence

In Control and Systems Engineering

In Probabilistic & Bayesian Methods

Citations in the corpus

Assumed Density Filtering Methods for Learning Bayesian Neural Networks (2016, background): “…including Hamiltonian Monte Carlo, variational inference, Laplace approximation, and expectation propagation, have been studied for learning weight distributions in Bayesian neural networks (Neal 1995; Hinton and Van Camp 1993; Graves 2011; MacKay 1992; Jyl¨anki, Nummenmaa, and Vehtari 2014).”
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks (2015, methodology): “Several approaches have been proposed for Bayesian learn-a ing of neural networks, based on, e.g., the Laplace approximation (MacKay, 1992b), Hamiltonian Monte Carlo (Neal, 1995), expectation propagation (Jyl¨anki et al., 2014), and variational inference (Hinton & Camp, 1993).”