Neural Networks with Adaptive Learning Rate and Momentum Terms

By M. Moreira, E. Fiesler. 1995

Subfields: Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

1. Yoshua Bengio
2. Razvan Pascanu
3. Kyunghyun Cho
4. Tomas Mikolov
5. Quoc V. Le
6. J. Urgen Schmidhuber
7. Li Deng
8. Richard Socher

In Optimization & Training

1. Yoshua Bengio
2. Razvan Pascanu
3. Tomas Mikolov
4. Kyunghyun Cho
5. James Martens
6. Taskin Kavzoglu
7. Aaron C. Courville
8. Caglar Gulcehre

Citations in the corpus

• On the difficulty of training recurrent neural networks (2012, background): “Compared to other learning rate adaptation strategies, which focus on improving convergence by collecting statistics on the gradient (as for example in Duchi et al. (2011), or Moreira and Fiesler (1995) for an overview), we rely on the instantaneous gradient.”
• The use of backpropagating artificial neural networks in land cover classification (2003, background): “Such strategies are known as adaptive learning strategies (Heermann and Khazenie 1992, Moreira and Fiesler 1995).”
• Understanding the exploding gradient problem (2012, background): “Compared to other learning rate adaptation strategies , which focus on improving convergence by collecting statistics on the gradient (as for example in Duchi et al. (2011), or Moreira and Fiesler (1995) for an overview), we rely on the instantaneous gradient .”