Ensemble Learning for Multi-Layer Networks

By David Barber, Charles M. Bishop. 1997

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Probabilistic Model-Agnostic Meta-Learning (2018, background): “Bayesian neural networks [27, 18, 29, 1] have been studied extensively as a way to incorporate uncertainty into deep networks.”
Expectation propagation for neural networks with sparsity-promoting priors (2013, methodology): “, Mackay, 1995; Williams, 1995), Gaussian approximations for NN models have been formed using variational Bayesian (VB) methods (Hinton and van Camp, 1993; Barber and Bishop, 1998; Honkela and Valpola, 2005), the extended Kalman filter (EKF) (Freitas, 1999), and the unscented Kalman filter (UKF) (Wa”
Expectation Backpropagation: Parameter-Free Training of Multilayer Neural Networks with Continuous or Discrete Weights (2014, methodology): “The Laplace approximation [17] and variational Bayes [10, 2, 9] based methods require real-value weights, tuning of the learning rate parameter, and stochastic neurons (to “smooth” the likelihood).”
Variational inference for count response semiparametric regression (2013, unknown): “It can yield tractable variational approximate inference in situations where ordinary MFVB is intractable and is a variant of earlier contributions of the same type by Barber and Bishop (1998) and Honkela et al. (2010). The Knowles and Minka (2011) approach yields particularly simple updates, based ”
Natural Conjugate Gradient in Variational Inference (2007, methodology): “Similar variational approximations have been applied for many such models (Barber and Bishop, 1998; Seeger, 2000; Lappalainen and Honkela, 2000; Valpola and Karhunen, 2002; Valpola et al., 2004; Honkela and Valpola, 2005).”
Approximate Riemannian Conjugate Gradient Learning for Fixed-Form Variational Bayes (2010, background): “…nevertheless been applied to a number of models that are not in the conjugate exponential family, such as multi-layer perceptron (MLP) networks (Barber and Bishop, 1998), kernel classifiers (Seeger, 2000), nonlinear factor analysis (Lappalainen and Honkela, 2000; Honkela and Valpola, 2005;…”
Ensemble learning in Bayesian neural networks (1998, background): “In this chapter we show that the ensemble learning approach can be extended to allow a Gaussian approximating distribution with a general covariance matrix, while still leading to a tractable algorithm (Barber and Bishop 1998).”
Pattern Recognition and Machine Learning (2006, unknown): “In this sense, it is analogous to the choice of a restrictive form f covariance matrix (for example, a diagonal matrix) in a multivariate Gaussian distribution.”
Explaining Variational Approximations (2010, methodology): “This approach has received less attention in the Computer Science literature (e.g. Barber & Bishop, 1998; Seeger 2000,2004; Honkela & Valpola 2005; Archambeau, Cornford, Opper & Shawe-Taylor 2007) but, nonetheless, is worthy of discussion.”
Variational Inference for Heteroscedastic Semiparametric Regression (2015, methodology): “…passing that we use involves approximating particular posterior density functions by Multivariate Normal density functions, which is an established practice within the variational approximation literature (e.g. Hinton & van Camp 1993; Barber & Bishop 1997; Raiko et al. 2007; Challis & Barber 2013).”
A family of algorithms for approximate Bayesian inference (2001, background): “Typically, q(x) is constrained to be Gaussian (Hinton & van Camp, 1993; Barber & Bishop, 1997; Seeger, 1999) but mixture distributions have also been suggested (Jaakkola & Jordan, 1999c).”
Overpruning in Variational Bayesian Neural Networks (2018, methodology): “Following initial work on using variational inference (VI) to fit neural networks [6, 1], a great deal of recent work has proposed new approaches for VI in these models [5, 2, 4, 8, 11].”
Uncertainty Estimation with Infinitesimal Jackknife, Its Distribution and Mean-Field Approximation (2020, methodology): “…models – an impractical task for modern learning models [10, 33]; the pseudo-ensemble distribution is in similar form as the Laplace approximation for Bayesian inference [4, 32] thus existing approaches in computational statistics such as Kronecker product factorization can be directly applied.”
Semiparametric Mean Field Variational Bayes: General Principles and Numerical Issues (2016, methodology): “Several articles concerned with deterministic approximate Bayesian inference, such as Barber and Bishop (1997), Honkela et al.”
A Scalable Laplace Approximation for Neural Networks (2018, methodology): “There has been previous interest in integrating neural networks into the Bayesian framework (MacKay, 1992; Hinton & Van Camp, 1993; Neal, 1993; Barber & Bishop, 1998), however these approaches were designed for small networks by current standards.”
Variational algorithms for approximate Bayesian inference (2003, methodology): “This work was later extended to show that tractable approximations were also possible with a full covariance Gaussian ( Barber and Bishop , 1998) (which in general will have the mode of the posterior at a different location than in the diagonal case).Neal and Hinton(1998) presented a generalisation ”
Bayesian Gaussian process models : PAC-Bayesian generalisation error bounds and sparse approximations (2003, methodology): “As a concrete example, it would be easy to apply it to Bayesian multi-layer perceptrons [110, 17], using inference approximations based on Laplace [110] or variational mean field [11].”
Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences (2018, background): “However, the exact Bayesian inference is not tractable in deep neural networks due to its high computational cost, and various approximate inference techniques—MCMC [17], Laplace approximation [14] and variational inference [1, 4, 8, 20]—have been proposed.”
The Variational Gaussian Approximation Revisited (2009, methodology): “Nevertheless, such a variational gaussian approximation has been applied, to our knowledge, only a few times to problems in machine learning (e.g., Barber & Bishop, 1998; Seeger, 2000; Honkela & Valpola, 2005).”
Affine Independent Variational Inference (2012, background): “It is therefore important to extend the class of tractable approximating distributions beyond standard forms such as the multivariate Gaussian [20, 12, 2, 13].”