Learning internal representations by error propagation

By David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams. 1986

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Fast R-CNN (2015, methodology): “The use of deep ConvNets [15, 17] for image classiﬁ-cation [13] and object detection [9, 18] has led to signiﬁ-cant gains in accuracy.”
Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation (2013, neutral): “[30], LeCun et al.”
Pylearn2: a machine learning research library (2013, methodology): “This class supports extensions including Polyak averaging [41], momentum [44], and early stopping.”
Theano: new features and speed improvements (2012, methodology): “The specific algorithm used is backpropagation through time Rumelhart et al. (1986), which optimizes for speed but not memory consumption.”
Gradient Descent Happens in a Tiny Subspace (2018, background): “Its ubiquity in deep learning is connected to the efficiency at which gradients can be computed (Rumelhart et al., 1985; 1986), though its success remains somewhat of a mystery due to the highly nonlinear and nonconvex nature of typical deep learning loss landscapes (Bottou et al., 2016).”
Inferring Algorithmic Patterns with Stack-Augmented Recurrent Nets (2015, methodology): “The model presented above is continuous and can thus be trained with stochastic gradient descent (SGD) method and back-propagation through time (Rumelhart et al., 1985; Williams & Zipser, 1995; Werbos, 1988).”
Environmental drivers of systematicity and generalization in a situated agent (2019, background): “Rumelhart et al. (1986) showed that a two-layer feedforward network with sufﬁcient hidden units can learn to predict the negation of binary patterns.”
Towards Understanding and Mitigating Social Biases in Language Models (2021, methodology): “We begin with a standard deﬁnition of language modeling: given some context c and a target vocabulary V consisting of a discrete set of word tokens, a model p θ with parameters θ aims to predict a distribution over the next candidates V over multiple time steps until a maximum step T is reached: In ”
Scalable agent alignment via reward modeling: a research direction (2018, background): “The currently most successful function approximators are deep neural networks trained with back-propagation (Rumelhart et al., 1986).”
Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks (2015, background): “…a rich history of the use of synthetic tasks in machine learning, from the XOR problem which helped motivate neural networks (Minsky & Papert, 1969; Rumelhart et al., 1985), to circle, spiral and ring datasets that helped motivate some of the most well-known clustering and semi-supervised learning…”
Long-term recurrent convolutional networks for visual recognition and description (2014, background): “Such “deep” models predated deep spatial convolution models in the literature [31, 45].”
Pointer Networks (2015, neutral): “Recurrent Neural Networks (RNNs) have been used for learning functions over sequences from examples for more than three decades [3].”
Bag of Tricks for Efficient Text Classification (2016, methodology): “, we use stochastic gradient descent and backpropagation (Rumelhart et al., 1986) with a linearly decaying learning rate.”
Multi-column deep neural networks for image classification (2012, background): “From then on, only trivial 1-dimensional winner-take-all regions are possible, that is, the top part of the hierarchy becomes a standard multi-layer perceptron (MLP) [36, 18, 28].”
Conditional Random Fields as Recurrent Neural Networks (2015, methodology): “these parameters in a single iteration can be learnt in the RNN setting using the standard backpropagation through time algorithm [10, 14].”
Analyzing the Performance of Multilayer Neural Networks for Object Recognition (2014, background): “Over the last two years, a sequence of results on benchmark visual recognition tasks has demonstrated that convolutional neural networks (CNNs) [6,14,18] will likely replace engineered features, such as SIFT [15] and HOG [2], for a wide variety of problems.”
Pretrained Transformers as Universal Computation Engines (2021, methodology): “Classical approaches to sequence processing used recurrent neural network (RNN) approaches (Rumelhart et al., 1985; Hochreiter & Schmidhuber, 1997).”
Visualizing and Understanding Recurrent Networks (2015, background): “erence paper at ICLR 2016 2 RELATED WORK Recurrent Networks. Recurrent Neural Networks (RNNs) have a long history of applications in various sequence learning tasks Werbos (1988); Schmidhuber (2015); Rumelhart et al. (1985). Despite their early successes, the difﬁculty of training simple recurrent n”
Optimization Methods for Large-Scale Machine Learning (2016, methodology): “ respect to the parameter vector wcan be computed by the chain rule using algorithmic dierentiation [71]. This dierentiation technique is known in the machine learning community as back propagation [134,135]. The number of layers in a DNN and the size of each layer are usually determined by performi”