On the training of recurrent neural networks

By Mounir Ben Nasr, Mohamed Chtourou. 2011

374 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Control and Systems Engineering, Information Systems, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Information Systems

In Optimization & Training

Citations in the corpus

Learning to Execute (2014, methodology): “From a methodological perspective, we formulate the program evaluation task as a language modeling problem on a sequence (Mikolov, 2012; Sutskever, 2013; Pascanu et al., 2013).”
On the difficulty of training recurrent neural networks (2012, background): “It was previously shown (Sutskever, 2012) that initialization plays an important role for training RNNs.”
Representation Learning: A Review and New Perspectives (2012, methodology): “…each layer closer to 1 across all its singular values) could substantially reduce the training difficulty of deep networks (Glorot and Bengio, 2010) and this is coherent with the success of the initialization procedure of Echo State Networks (Jaeger, 2007), as recently studied by Sutskever (2012).”
A Deep and Tractable Density Estimator (2013, methodology): “We used Nesterov’s accelerated gradient (Sutskever, 2013) with momentum value 0.9.”
Deep Learning: Methods and Applications (2014, methodology): “To make deep learning techniques scalable to very large training data, theoretically sound parallel learning and optimization algorithms together with novel architectures need to be further developed [31, 39, 49, 69, 181, 322, 356].”
Advances in optimizing recurrent networks (2012, background): “…hyperparameters, such as the number of hidden unitsnh, regularization coefficientsλL1, the choice of non-linearity function, or the momentum scheduleµt, learning rateǫt, number of leaky unitsnleaky or leaky factorsα according to log-likelihood on a validation set and we report the final…”
An overview of gradient descent optimization algorithms (2016, background): “io/neural-networks-3/ for another explanation of the intuitions behind NAG, while Ilya Sutskever gives a more detailed overview in his PhD thesis [19].”
Deep Learning of Representations: Looking Forward (2013, background): “…useful information (Glorot and Bengio, 2010; Sutskever, 2012).3 Another interesting ingredient in the 3 and potentially with the use of momentum (Sutskever, 2012) success of training the deep supervised networks of Glorot and Bengio (2010) (and later of Krizhevsky et al. (2012)) is the presence…”
TopicRNN: A Recurrent Neural Network with Long-Range Semantic Dependency (2016, background): “While in principle RNN-based models can “remember” arbitrarily long histories if provided enough capacity, in practice such large-scale neural networks can easily encounter difficulties during optimization (Bengio et al., 1994; Pascanu et al., 2013; Sutskever, 2013) or overfitting issues (Srivastava”
Deep learning for computational biology (2016, methodology): “To validate the learning process, the loss should be monitored as a function of the number of training epochs, that is the number of times the full training set has been traversed (Fig 5D).”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, background): “More recently, Bengio et al. [49] and Sutskever [50] have explored new optimization methods in training generative RNNs that modify stochastic gradient descent and show these modifications can out-perform Hessian-free optimization methods.”
rnn : Recurrent Library for Torch (2015, methodology): “So the rnn package started out as a single Recurrent module that internally implemented a general interface for implementing variations of Simple RNNs as described in [15, section 2.5-2.8], [10, section 3.2-3.3] and [1].”
Connectionist multivariate density-estimation and its application to speech synthesis (2016, methodology): “, 2013), better data-independent initialization techniques (Sutskever, 2013) and training algorithms (Martens, 2010).”
A Primal-Dual Method for Training Recurrent Neural Networks Constrained by the Echo-State Property (2013, methodology): “Our work also differs from the earlier work reported in (Sutskever, 2013; Martens & Sutskever, 2011), which adopted a special way of initialization using echo state networks (Jaeger, 2001a) when carrying out the standard BPTT procedure, without using a formal constrained optimization framework.”
Persistent RNNs: Stashing Recurrent Weights On-Chip (2016, background): “A common solution to this problem is to use truncated back-propagation through time (Sutskever, 2013) (BPTT).”