Techniques for Learning Binary Stochastic Feedforward Neural Networks

By Tapani Raiko, Mathias Berglund, Guillaume Alain, Laurent Dinh. 2014

129 citations recorded by the source corpus.

Stochastic binary hidden units in a multi-layer perceptron (MLP) network give at least three potential benefits when compared to deterministic MLP networks. (1) They allow to learn one-to-many type of mappings. (2) They can be used in structured prediction problems, where modeling the internal structure of the output is important. (3) Stochasticity has been shown to be an excellent regularizer, which makes generalization performance potentially better in general. However, training stochastic networks is considerably more difficult. We study training using M samples of hidden activations per input. We show that the case M=1 leads to a fundamentally different behavior where the network tries to avoid stochasticity. We propose two new estimators for the training gradient and propose benchmark tests for comparing training algorithms. Our experiments confirm that training stochastic networks is difficult and show that the proposed two estimators perform favorably among all the five known estimators.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Generating Sentences from a Continuous Space (2015, methodology): “If the vae were trained with a standard autoen-coder’s reconstruction objective, it would learn to encode its inputs deterministically by making the variances in q ( (cid:126)z | x ) vanishingly small (Raiko et al., 2015).”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “The gradients required for back-propagating through such a thresholding operation cannot be directly derived, so we use the straight-through estimation (STE) method [28,29].”
Categorical Reparameterization with Gumbel-Softmax (2016, background): “This is a common benchmark for training stochastic binary networks (SBN) (Raiko et al., 2014; Gu et al., 2016; Mnih & Rezende, 2016).”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “We will consider the task of predicting the bottom half x1 of an image of an MNIST digit given its top half x2, as introduced by [32].”
Semi-supervised Learning with Ladder Networks (2015, methodology): “One way to study this is to assign probabilistic labels q(y(n)) = P (y(n) | x(n)) to unlabeled inputs x(n) and try to train P (y | x) using those labels: it can be shown (see, e.g., Raiko et al., 2015, Eq.”
Deep Learning for Distributed Optimization: Applications to Wireless Resource Management (2019, methodology): “To resolve this, a gradient estimation idea [21], [22] is adopted such that the gradient of the stochastic operation in (21) is approximated by its expectation averaged over the quantization noise distribution, i.e., Here, the zero-mean quantization noise replaces the gradient of v i with a…”
MuProp: Unbiased Backpropagation for Stochastic Neural Networks (2015, methodology): “With these models, which are notoriously difficult to train, biased methods often significantly outperform the unbiased ones (Dayan et al., 1995; Gregor et al., 2014; Raiko et al., 2015), except in certain cases (Mnih & Gregor, 2014).”
Robust Convolutional Neural Networks under Adversarial Noise (2015, methodology): “To clarify, we adopted the artiﬁcial noise distribution in order to improve the robustness of CNNs and it is unrelated to natural image statistics.”
Variational Inference for Monte Carlo Objectives (2016, methodology): “This gives rise to the following local learning signal for sample j : For K = 1 , this estimator becomes essentially equivalent to the NVIL estimator, with log f ( x ) corresponding to the input-dependent baseline b ( x ) .”
Variable Rate Image Compression with Recurrent Neural Networks (2015, positive): “…and b can be replaced by b inf deﬁned as: The compression rate is determined by the number of bits generated in each stage, which corresponds to the number of rows in the W bin matrix, and by the number of stages, controlled by the number of repetitions of the residual autoencoder structure.”
Exponentially Increasing the Capacity-to-Computation Ratio for Conditional Computation in Deep Learning (2014, methodology): “Following Bengio et al. (2013a), Gregor et al. (2014) and Raiko et al. (2014), estimate a gradient using a heuristic that propagates the gradient on g (obtained by back-prop of the loss) backwards into the pre-threshold values x .”
Describing Multimedia Content Using Attention-Based Encoder-Decoder Networks (2015, background): “The question of training neural networks with stochastic discrete-valued hidden units has a long history, starting with Boltzmann machines [36], with recent work studying how to deal with such units in a system trained using back-propagated gradients [37]–[40].”
Discrete Variational Autoencoders (2016, background): “1 Moreover, it would be desirable to apply the efficient variational autoencoder framework to models with discrete values, but this has proven difficult, since backpropagation through discrete variables is generally not possible [12, 13].”
A Hybrid Convolutional Variational Autoencoder for Text Generation (2017, background): “The most straight-forward way to achieve a good reconstruction error in this case is to predict a very sharp probability distribution effectively corresponding to a single point in the latent space (Raiko et al., 2014).”
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models (2017, methodology): “We followed the structured prediction task described by Raiko et al. (2014), where we modeled the bottom half of an MNIST digit ( x ) conditional on the top half ( c ).”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, methodology): “In order to back-propagate through the approximating posterior of discrete binary variables, we employ a continuous relaxation estimator that trades bias for variance [46, 47], in particular the Gumbel-Softmax estimator [48, 49].”
Unbiased Gradient Estimation with Balanced Assignments for Mixtures of Experts (2021, background): “[33] Pekka Raiko, Mathias Berglund, Guillaume Alain, and Laurent Dinh.”
Iterative Refinement of Approximate Posterior for Training Directed Belief Networks (2015, background): “, 2015) and recent work on stochastic feed-forward networks (SFFN, Tang & Salakhutdinov, 2013; Raiko et al., 2014).”
Iterative Refinement of Approximate Posterior for Training Directed Belief Networks (2015, unknown): “AIR also shares similarities with RWS in the use of importance sampling, as with importance-weighted autoen-coders (IWAE, Burda et al., 2015) and recent work on stochastic feed-forward networks (SFFN, Tang & Salakhutdi-nov, 2013; Raiko et al., 2014).”
Undirected Graphical Models as Approximate Posteriors (2019, methodology): “For this, we follow the IW objective function proposed by (Raiko et al., 2015): Ez1:K⇠Qi q(zi|x1) log ⇣ 1 K KX i=1 p(x2|z i) ⌘ + H(q(z |x1)).”