Equilibrated adaptive learning rates for non-convex optimization

By Yann Dauphin, Harm de Vries, Yoshua Bengio. 2015

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Frank Hutter
2. Leslie N. Smith
3. Ilya Loshchilov
4. Yoshua Bengio
5. Aliaksei Severyn
6. Jianfeng Gao
7. Chunyuan Li
8. Quoc V. Le

Citations in the corpus

• Session-based Recommendations with Recurrent Neural Networks (2015, neutral): “We experimented with both rmsprop (Dauphin et al., 2015) and adagrad (Duchi et al., 2011).”
• Cyclical Learning Rates for Training Neural Networks (2015, background): “[4] argue that the difficulty in minimizing the loss arises from saddle points rather than poor local minima.”
• Recurrent Dropout without Memory Loss (2016, methodology): “For training we use the RMSProp algorithm (Dauphin et al., 2015) with ρ”
• Online Batch Selection for Faster Training of Neural Networks (2015, background): “…some recent progress in understanding and addressing the latter problems (Bordes et al., 2009; Dauphin et al., 2014; Choromanska et al., 2014; Dauphin et al., 2015), state-of-the-art optimization techniques tend to approximate the inverse Hessian in a reduced way, e.g., by considering only…”