Neural Networks for Machine Learning

By Richard F. Lyon. 2017

546 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

Citations in the corpus

Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, methodology): “He found that the fastest training was obtained when using the “straight-through estimator,” previously introduced in Hinton’s lectures (Hinton, 2012).”
Hierarchical Multiscale Recurrent Neural Networks (2016, methodology): “We ﬁnd that the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016) is efﬁcient for training this model containing discrete variables.”
Gated Feedback Recurrent Neural Networks (2015, methodology): “We used RMSProp (Hinton, 2012) and momentum to tune the model parameters (Graves, 2013).”
Binarized Neural Networks (2016, methodology): “He found in his experiments that the fastest training was obtained when using the “straight-through estimator,” previously introduced in Hinton’s lectures (Hinton, 2012).”
DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients (2016, methodology): “We adopt the “straight-through estimator” (STE) method (Hinton et al., 2012b; Bengio et al., 2013) to circumvent this problem.”
A Convolutional Attention Network for Extreme Summarization of Source Code (2016, methodology): “To train conv_attention and copy_attention we optimize the objective using stochastic gradient descent with RMSProp and Nesterov momentum (Sutskever et al., 2013; Hinton et al., 2012).”
Creating High Resolution Images with a Latent Adversarial Generator (2020, methodology): “To allow gradient propagation through the rounding operation, we use Hinton’s straight through estimator [7].”
Training Spiking Neural Networks Using Lessons From Deep Learning (2021, background): “Hinton overcame the challenge of thresholding activations and weights in binarized neural networks by simply ignoring them during the backward pass [175].”
Understanding Straight-Through Estimator in Training Activation Quantized Neural Nets (2019, methodology): “Hinton (2012) extended this idea to train multi-layer networks with binary activations (a.k.a. binary neuron), namely, to back-propagate as if the activation had been the identity function.”
Measuring the perceptual availability of phonological features during language acquisition using unsupervised binary stochastic autoencoders (2019, methodology): “We overcome this problem by approximating missing gradients using the straight-through estimator (Hinton, 2012; Bengio et al., 2013; Courbariaux et al., 2016) with slope-annealing (Chung et al.”
A Survey of Privacy Attacks in Machine Learning (2020, methodology): “While SGD is still quite popular, several improvements have been proposed to try to speed up convergence by adding momentum [91], by using adaptive learning rates as, for example, in the RMSprop algorithm [40], or by combining both improvements as in the Adam algorithm [56].”