On the Origin of Deep Learning

By Haohan Wang, Bhiksha Raj, Eric P. Xing. 2017

243 citations recorded by the source corpus.

This paper is a review of the evolutionary history of deep learning models. It covers from the genesis of neural networks when associationism modeling of the brain is studied, to the models that dominate the last decade of research in deep learning like convolutional neural networks, deep belief networks, and recurrent neural networks. In addition to a review of these models, this paper primarily focuses on the precedents of the models above, examining how the initial ideas are assembled to construct the early models and how these preliminary models are developed into their current forms. Many of these evolutionary paths last more than half a century and have a diversity of directions. For example, CNN is built on prior knowledge of biological vision system; DBN is evolved from a trade-off of modeling power and computation complexity of graphical models and many nowadays models are neural counterparts of ancient linear models. This paper reviews these evolutionary paths and offers a concise thought flow of how these models are developed, and aims to provide a thorough background for deep learning. More importantly, along with the path, this paper summarizes the gist behind these milestones and proposes many directions to guide the future research of deep learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Demystifying Parallel and Distributed Deep Learning (2018, methodology): “gins of deep learning methodologies from a historical perspective, as well as discuss the potential capabilities of DNNs w.r.t. learnable functions and representational power. Two of the three surveys[209,236] also describe optimization methods and applied regularization techniques in detail. Bengio”
A survey and critique of multiagent deep reinforcement learning (2018, background): “o highlight that existent literature contain pertinent ideas and algorithms that should not be ignored, on the contrary, they should be examined and cited [268, 269] to understand recent developments [270]. Dealing with non-stationarity in independent learners.. It is well known that using independe”
Bearing Fault Detection and Diagnosis Using Case Western Reserve University Dataset With Deep Learning Approaches: A Review (2020, background): “On working with both structured and unstructured data, DL networks perform automatic feature extraction without human involvement [66], [67].”
High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks (2019, background): “In other words, the almost-100-year development process of neural networks functions like a “nat-ural selection” of technologies [60].”
A survey of the recent architectures of deep convolutional neural networks (2019, background): “In this regard, Bengio and Delalleau (Delalleau and Bengio 2011) suggested that deeper networks can maintain the expressive power of the network at a reduced cost (Wang and Raj 2017).”
A Comprehensive Review of Speech Emotion Recognition Systems (2021, background): “Deep Belief Networks (DBN) is an unsupervised generative model that mixes the directed and undirected connections between the variables that constitute either the visible layer or all hidden layers [63], as shown in Fig.”
Review on Convolutional Neural Network (CNN) Applied to Plant Leaf Disease Classification (2021, background): “Kunihiko Fukushima introduced the Neocognitron in 1980, which inspired CNNs [29].”
Deep Reinforcement Learning: An Overview (2017, background): “, 2017) 3 • deep learning (LeCun et al., 2015; Schmidhuber, 2015; Bengio, 2009; Wang and Raj, 2017) • efficient processing of deep neural networks (Sze et al.”
Reinforcement Learning for Selective Key Applications in Power Systems: Recent Advances and Future Challenges (2021, background): “The history of ANNs dates back to 1940s [39], and it has received tremendous interests in the recent decade due to the booming of data technology and computing power, which allows efﬁcient training of wider and deeper ANNs.”
Deep Learning Techniques: An Overview (2020, unknown): “Then other networks are evolved eventually [15,24].”
Select-additive learning: Improving generalization in multimodal sentiment analysis (2016, background): “The same loss function is often used in discriminative neural networks [27].”