Learning representations by back-propagating errors

By David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams. 1986

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Sequence to Sequence Learning with Neural Networks (2014, background): “In particular, while we were unable to train a standard RNN on the non-reversed translation problem (shown in fig.”
Distributed Representations of Sentences and Documents (2014, background): “Distributed representations for words were ﬁrst proposed in (Rumelhart et al., 1986) and have become a successful paradigm, especially for statistical language modeling (El-man, 1990; Bengio et al., 2006; Mikolov, 2012).”
ADADELTA: An Adaptive Learning Rate Method (2012, methodology): “One method of speeding up training per-dimension is the momentum method [2].”
Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013, background): “This was what motivated the move from neural networks based on so-called formal neurons, with a hard threshold output, to neural networks whose units are based on a sigmoidal non-linearity, and the well-known back-propagation algorithm to compute the gradients (Rumelhart et al. , 1986).”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Variants of SGD commonly used with neural networks include SGD with momentum (Polyak, 1964; Rumelhart et al., 1986; Sutskever et al., 2013), Nesterov momentum (Nes-terov, 1983; Sutskever et al., 2013), RMSProp (Hinton et al., 2012), and Adam (Kingma and Ba, 2015).”
Character-level Convolutional Networks for Text Classification (2015, methodology): “The design is modular, where the gradients are obtained by back-propagation [27] to perform optimization.”
Gradient Descent Happens in a Tiny Subspace (2018, background): “Its ubiquity in deep learning is connected to the efﬁciency at which gradients can be computed (Rumelhart et al., 1985; 1986), though its success remains somewhat of a mystery due to the highly nonlinear and nonconvex nature of typical deep learning loss landscapes (Bottou et al., 2016).”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “Recurrent neural networks (RNNs, Rumelhart et al. (1986)) learn to represent the history of observations in a compressed state vector.”
Enriching Word Vectors with Subword Information (2016, background): “Learning continuous representations of words has a long history in natural language processing (Rumelhart et al., 1988).”
Group Normalization (2018, methodology): “These methods are effective for training sequential models (RNN/LSTM [49, 22]) or generative models (GANs [15, 27]).”
Non-local Neural Networks (2017, background): “, in speech, language), recurrent operations [38, 23] are the dominant solution to long-range dependency modeling.”
TensorFlow: A system for large-scale machine learning (2016, neutral): “A user can, for example, deﬁne a neural network as a composition of layers and a loss function, and the library will derive the backpropagation [64].”
FFJORD: Free-form Continuous Dynamics for Scalable Reversible Generative Models (2018, methodology): “This approach is a continuous-time analog to the backpropgation algorithm (Rumelhart et al., 1986; Andersson, 2013) and can be combined with gradient-based optimization methods to ﬁt the parameters θ .”
Density estimation using Real NVP (2016, background): “Recent work in this line of research have successfully taken advantage of recent advances in recurrent networks [44], in particular long-short term memory [21], and residual networks [20, 19] in order to learn state-of-the-art generative image models [54, 39] and language models [26].”