Expectation Backpropagation: Parameter-Free Training of Multilayer Neural Networks with Continuous or Discrete Weights

By Daniel Soudry, Itay Hubara, R. Meir. 2014

236 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices (2017, background): “Quantization [31, 27, 39, 45, 44] and factorization [22, 16, 18, 37] are proposed in literature to reduce redundancy in calculations to speed up inference.”
XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks (2016, background): “Expectation BackPropagation (EBP) in [36] showed high performance can be achieved by a network with binary weights and binary activations.”
BinaryConnect: Training Deep Neural Networks with binary weights during propagations (2015, background): “[37, 38] do not train their DNN with Backpropagation (BP) but with a variant called Expectation Backpropagation (EBP).”
A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, methodology): “A more scalable solution based on expectation propagation was proposed by Soudry [57] in 2014.”
Bayesian Hypernetworks (2017, methodology): “…learn an approximate posterior distribution using (stochastic) variational inference (Graves, 2011; Gal and Ghahramani, 2016; Salimans et al., 2015; Blundell et al., 2015), expectation propagation (Hernandez-Lobato and Adams, 2015; Soudry et al., 2014), or α -divergences (Li and Gal, 2017).”
Assumed Density Filtering Methods for Learning Bayesian Neural Networks (2016, methodology): “Probit likelihoods allow the analytical computation of ln Z, Our model is similar to those considered by (Soudry, Hubara, and Meir 2014) for binary classiﬁcation problems.”
Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks (2015, methodology): “A different scalable solution based on expectation propagation was proposed by Soudry et al. (2014).”
Binarized Neural Networks (2016, methodology): “Moreover, adding noise to weights and activations when computing the parameter gradients provide a form of regularization that can help to generalize better, as previously shown with variational weight noise (Graves, 2011), Dropout (Srivastava et al., 2014) and DropConnect (Wan et al.”
Task Agnostic Continual Learning Using Online Variational Bayes (2018, background): “also closely related to the assumed density filtering approach of [31, 9].”
Searching for MobileNetV3 (2019, background): “Quantization [23, 25, 47, 41, 51, 52, 37] is another important complementary effort to improve the network efficiency through reduced precision arithmetic.”
Task-Agnostic Continual Learning Using Online Variational Bayes With Fixed-Point Updates (2020, methodology): “Those methods include assumed density filtering (Soudry et al., 2014; Hernández-Lobato and Adams, 2015), approximate power Expectation Propagation (Hernández-Lobato et al.”
Bitwise Neural Networks (2016, methodology): “A more practical network was proposed in (Soudry et al., 2014) recently, where the posterior probabilities of the binary weights were sought using the Expectation Back Propagation (EBP) scheme, which is similar to backpropagation in its form, but has some advantages, such as parameter-free learning…”
Training Quantized Nets: A Deeper Understanding (2017, methodology): “Bayesian learning has also been applied to train binary networks [19, 20].”
Training Binary Multilayer Neural Networks for Image Classification using Expectation Backpropagation (2015, methodology): “As an extension of the previous work by Soudry et al. (2014), in this work, we study the performance of EBP algorithm on image classiﬁcation tasks with binary and real weights MNNs.”
Bayesian Recurrent Neural Networks (2017, background): “Nonethe-less, several papers explore applying expectation propagation to neural networks: Soudry et al. (2014) derive a closed form approximate online expectation propagation algorithm, whereas Hern´andez-Lobato & Adams (2015) proposed using multiple passes of assumed density ﬁltering (in…”
ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functions (2020, unknown): “tization, specically the extreme case of quantizing both weights and activations to only 1-bit, which is so-called network binarization or 1-bit CNNs. Neural network binarization originates from EBP [34] and BNN [7], which establish an end-to-end gradient back-propagation framework for training the ”
QKD: Quantization-aware Knowledge Distillation (2019, background): “Some researches consider quantizing both weights and activation maps [16, 41, 35, 52, 46].”
Deep Learning With Spiking Neurons: Opportunities and Challenges (2018, methodology): “A commonly used stochastic method is expectation backpropagation , where neuron activations and synaptic weights are represented by probability distributions updated by backpropagation (Soudry et al., 2014).”
Bayesian Gradient Descent: Online Variational Bayes Learning with Increased Robustness to Catastrophic Forgetting and Weight Pruning (2018, methodology): “Those methods include assumed density filtering (Soudry et al., 2014; Hernández-Lobato and Adams, 2015), approximate power Expectation Propagation (HernándezLobato et al., 2016), Stochastic Langevin Gradient Descent (Welling and Teh, 2011; Balan et al., 2015), incremental moment matching (Lee et…”