Training Recurrent Neural Networks

By Geoffrey E. Hinton, Ilya Sutskever. 2013

263 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Learning to Execute (2014, methodology): “rom a methodological perspective, we formulate the program evaluation task as a sequenceto-sequence learning problem with a recurrent neural network (Sutskever et al., 2014) (see also (Mikolov, 2012; Sutskever, 2013; Pascanu et al., 2013)). Other interesting applications of recurrent neural networks”
On the difficulty of training recurrent neural networks (2012, background): “It was previously shown (Sutskever, 2012) that initialization plays an important role for training RNNs.”
Representation Learning: A Review and New Perspectives (2012, neutral): “e the training difﬁculty of deep networks (Glorot and Bengio, 2010) and this is coherent with the success of the initialization procedure of Echo State Networks (Jaeger, 2007), as recently studied by Sutskever (2012). There are also several experimental results (Glorot and Bengio, 2010; Glorot et al”
A Deep and Tractable Density Estimator (2013, methodology): “r units (Nair & Hinton, 2010) in all experiments. We found that this type of unit allow us to use higher learning rates and made training converge faster. We used Nesterov’s accelerated gradient (Sutskever, 2013) with momentum value 0.9. No weight decay was applied. To avoid overﬁtting, we early”
Advances in optimizing recurrent networks (2012, methodology): “ients effect. 3.5. Simpliﬁed Nesterov Momentum Nesterov accelerated gradient (NAG) [30] is a ﬁrst-order optimization method to improve stability and convergence of regular gradient descent. Recently, [6] showed that NAG could be computed by the following update rules: vt = µt−1vt−1 −ǫt−1∇f(θt−1 + µt”
An overview of gradient descent optimization algorithms (2016, background): “m performs much worse. 9Refer to http://cs231n.github.io/neural-networks-3/ for another explanation of the intuitions behind NAG, while Ilya Sutskever gives a more detailed overview in his PhD thesis [19]. 10http://www.wired.com/2012/06/google-x-neural-network/ 11Duchi et al. [8] give this matrix as”
Deep Learning of Representations: Looking Forward (2013, background): “it was discovered that one could train deep supervised nets by proper initialization, just large enough for gradients to ﬂow well and activations to convey useful information (Glorot and Bengio, 2010;Sutskever, 2012).3 Another interesting ingredient in the 3 and potentially with the use of momentum ”
High-quality Speech Coding with Sample RNN (2018, methodology): “ layers and fully connected layers contain 1024 hidden units each. During training, we use a batch size of 24 and a sequence length of 6400 samples for truncated back propagation through time (TBPTT, [25, 26]) on a single GPU. We use the ADAM optimizer [27] ( 1 = 0:9, 2 = 0:999, and = 1e-8) with an ”
Deep learning for computational biology (2016, neutral): “These include convolutional neural networks, which are widely used for modelling images (Box 2), recurrent neural networks for sequential data (Sutskever, 2013; Lipton, 2015), or restricted Boltzmann machines (Salakhutdinov & Larochelle, 2010; Hinton, 2012) and autoencoders (Hinton & Salakhutdinov, ”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, methodology): “improve the current optimization techniques for learning deep architectures (Martens, 2010; Le et al., 2011; Martens and Sutskever, 2011; Dean et al., 2012; Sutskever, 2013).”
Foundations and Advances in Deep Learning (2014, methodology): “A similar idea was also proposed by Tang and Sutskever (2011), but they applied the transformation only to visible units.”
Long Short-Term Memory Networks for Accurate State-of-Charge Estimation of Li-ion Batteries (2018, background): “Aside from the latter applications, RNNs can be useful for just about any type of time series data [26].”
Hierarchical Control Using Networks Trained with Higher-Level Forward Models (2014, background): “Neural networks have also received renewed attention in recent years as generic substrates for feedback controllers (Huh & Todorov, 2009; Sutskever, 2013).”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, methodology): “d Salakhutdinov, 2014) and to model sequential structure through recursive application (Bottou, 2011) of learned representation-to-representation transformations, e.g. with recurrent neural networks (Sutskever, 2012) or recursive neural networks (Socher et al., 2011). Deep learning methods have esse”