Deep learning in neural networks: An overview
By J. Urgen Schmidhuber. 2014
17,623 citations recorded by the source corpus.
In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.
Subfields: Artificial Intelligence, Scaling & Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
- Ruslan Salakhutdinov
- Dacheng Tao
In Scaling & Learning Theory
- Behnam Neyshabur
- Andrew M. Saxe
- Surya Ganguli
- Madhu S. Advani
- Razvan Pascanu
- Zhenyu A. Liao
- Romain Couillet
- Philip M. Long
Citations in the corpus
- What can the brain teach us about building artificial intelligence? (2016, background): “These models have achieved remarkable gains in many domains spanning object recognition, speech recognition, and control (LeCun, Bengio, & Hinton, 2015; Schmidhuber, 2015).”
- End-to-End Training of Deep Visuomotor Policies (2015, background): “CNNs have a long history in computer vision and deep learning (Fukushima, 1980; LeCun et al., 1989; Schmidhuber, 2015), and have recently gained prominence due to excellent results on a number of vision benchmarks (Ciresan et al.”
- Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, background): “1 Introduction Over the years, deep learning approaches (see [5, 26] for survey) have shown great success in many visual perception problems (e.”
- Recurrent Highway Networks (2016, background): “Network depth is of central importance in the resurgence of neural networks as a powerful machine learning paradigm [1].”
- Visualizing and Understanding Recurrent Networks (2015, background): “Recurrent Neural Networks (RNNs) have a long history of applications in various sequence learning tasks (31; 26; 25).”
- Interaction Networks for Learning about Objects, Relations and Physics (2016, background): “Deep learning [23, 16] couples generic architectures with efficient optimization 30 algorithms to provide highly scalable learning and inference in challenging real-world settings.”
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Algorithmic breakthroughs, the feasibility of collecting large amounts of data, and increasing computational power have contributed to the current popularity of neural networks, in particular with multiple (deep) hidden layers, that indeed have started to outperform previous state-of-the-art machine”
- Financial Time Series Forecasting with Deep Learning : A Systematic Literature Review: 2005-2019 (2019, methodology): “In literature, there are different kinds of DL models: Deep Multilayer Perceptron (DMLP), RNN, LSTM, CNN, Restricted Boltzmann Machines (RBMs), DBN, Autoencoder (AE), and DRL [37, 38].”
- Meta-Learning with Latent Embedding Optimization (2018, background): “In contrast, traditional deep learning approaches (LeCun et al., 2015; Schmidhuber, 2015) treat each task independently and hence are often data inefficient – despite providing significant performance improvements across the board, such as for image classification (Simonyan & Zisserman, 2014; He et…”
- Error Feedback Fixes SignSGD and other Gradient Compression Schemes (2019, background): “…optimization algorithms [Bottou, 2010] which are amenable to large-scale parallelization, taking advantage of massive computational resources [Krizhevsky et al., 2012, Dean et al., 2012] have been at the core of significant recent progress in deep learning [Schmidhuber, 2015, LeCun et al., 2015].”
- Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…2017; Rajani and Mooney, 2017; Reed, Akata, Lee, and Schiele, 2016; Sadeghi, Kumar Divvala, and Farhadi, 2015; Samek et al., 2017; Schank, 2013; Schmidhuber, 2015; Shwartz-Ziv and Tishby, 2017; Simonyan, Vedaldi, and Zisserman, 2013; Springenberg, Dosovitskiy, Brox, and Riedmiller, 2014;…”
- A practical tutorial on autoencoders for nonlinear feature fusion: Taxonomy, models, software and guidelines (2018, methodology): “The gradient descent algorithm, firstly used for convolutional neural networks [7] and later for unsupervised learning [8], was one of the foundations of modern deep learning [9] methods.”