Bayesian Regularization and Pruning Using a Laplace Prior

By Peter M. Williams. 1995

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Yoshua Bengio
2. Dacheng Tao
3. Yu Zhang
4. Xiaogang Wang
5. Sergey Levine
6. Qiang Yang
7. Yi Yang
8. Trevor Darrell

Citations in the corpus

• Expectation propagation for neural networks with sparsity-promoting priors (2013, methodology): “Implementing a successful algorithm requires some additional approximations as described by Williams (1995), whereas with EP the implementation is straightforward since it relies only on expectations of the prior terms with respect to a Gaussian measur .”
• Maximum entropy modeling of species geographic distributions (2006, background): “Maxent shares with other machine learning methods Regularization, which penalizes the use of large values of model parameters, can be interpreted as the prior (Williams, 1995).”
• Supervised probabilistic principal component analysis (2006, methodology): “Similarly, we add an L1 regularizer by introducing a Laplacian prior to each element, Wij , of the transformation matrix W, since Laplacian priors are equivalent to L1 regularization (Chen et al., 1998; Williams, 1995; Figueiredo, 2001).”
• Sparsifying Neural Network Connections for Face Recognition (2015, methodology): “We compare our neural correlation based weight pruning strategy with other pruning strategies proposed previously, including optimal brain damage (OBD) [7], weight magnitude based pruning (which is the bases of many weight pruning algorithms [6]), and Bayesian regularization and pruning (BRP) [27].”
• Constructive algorithms for structure learning in feedforward neural networks for regression problems (1997, methodology): “Regularization can then be accomplished by using appropriate priors that favor small network weights (such as the normal [6] or Laplace [9] distribution), and the regularization parameter can be automatically set.”
• Input-dependent estimation of generalization error under covariate shift (2005, background): “, learning with non-quadratic loss functions [16, 53, 49] or non-quadratic regularizers [10, 52, 48, 6].”
• Discriminative Random Fields (2006, unknown): “Another choice is to integrate out τ by choosing a noninformative hyperprior on τ as in Williams (1995) and Figueiredo (2001).”
• Adaptive Regularization in Neural Network Modeling (1996, background): “, the Laplacian [12], [42] and soft weight sharing [33] has been considered.”
• Simplifying Neural Nets by Discovering Flat Minima (1994, background): “Hinton and van Camp [3] and Williams [17] assume that pushing the posterior distribution (after learning) close to the prior leads to "good" generalization.”
• Modeling of species distributions with Maxent: new extensions and a comprehensive evaluation (2008, methodology): “First, Maxent uses l1 regularization, which tends to produce models with few non-zero coefficients (Williams 1995, Tibshirani 1996) and therefore encourages parsimony.”
• Bayesian methods for neural networks (2000, background): “It is possible to maximise the posterior density function by performing integrations over the hyper-parameters analytically (Buntine and Weigend, 1991; Mackay, 1996; Williams, 1995; Wolpert, 1993).”
• Efficient posterior sampling for high-dimensional imbalanced logistic regression. (2019, background): “Another common prior is the double exponential prior, also known as the Laplace prior [30, 24].”
• Dynamic modeling of genetic networks using genetic algorithm and S-system (2003, methodology): “Moreover, under the concept of regularization, the pruning term is equivalent to the Laplacian regularization term (Williams, 1995).”
• Sparse multinomial logistic regression: fast algorithms and generalization bounds (2005, background): “The sparsity-promoting nature of the Laplacian prior is theoretically well-justified (see [9], [12], [29], as well as references therein) and has been found to be practically and conceptually useful in several research areas [4], [23], [30], [41].”
• Flat Minima (1997, background): “…8 ij : v ij = w ij g. Flatness condition 1 is a \robustness condition" (or \fault tolerance condition", or \perturbation tolerance condition" { see, e.g., Minai & Williams, 1994; Murray & Edwards, 1993; Neti et al., 1992; Matsuoka, 1992; Bishop, 1993; Kerlirzin & Vallet, 1993; Carter et al., 1990).”