On the difficulty of training recurrent neural networks

By Razvan Pascanu, Tomas Mikolov, Yoshua Bengio. 2012

6,009 citations recorded by the source corpus.

There are two widely known issues with properly training recurrent neural networks, the vanishing and the exploding gradient problems detailed in Bengio et al. (1994). In this paper we attempt to improve the understanding of the underlying issues by exploring these problems from an analytical, a geometric and a dynamical systems perspective. Our analysis is used to justify a simple yet effective solution. We propose a gradient norm clipping strategy to deal with exploding gradients and a soft constraint for the vanishing gradients problem. We validate empirically our hypothesis and proposed solutions in the experimental section.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Yoshua Bengio
  2. Mohit Bansal
  3. Jianfeng Gao
  4. Quoc V. Le
  5. Percy Liang
  6. Luke S. Zettlemoyer
  7. Sergey Levine
  8. Kyunghyun Cho

In Statistical and Nonlinear Physics

  1. Yoshua Bengio
  2. Arash Vahdat
  3. Stefano Ermon
  4. Greg Yang
  5. Aaron C. Courville
  6. Kyunghyun Cho
  7. Sam Schoenholz
  8. Yang Song

In Model Architectures

  1. Yoshua Bengio
  2. Quoc V. Le
  3. Kyunghyun Cho
  4. Ruslan Salakhutdinov
  5. Richard Socher
  6. Mohit Bansal
  7. Vladlen Koltun
  8. Yoav Goldberg

Citations in the corpus