A tutorial survey of architectures, algorithms, and applications for deep learning

By Li Deng. 2014

772 citations recorded by the source corpus.

In this invited paper, my overview material on the same topic as presented in the plenary overview session of APSIPA-2011 and the tutorial material presented in the same conference [1] are expanded and updated to include more recent developments in deep learning. The previous and the updated materials cover both theory and applications, and analyze its future directions. The goal of this tutorial survey is to introduce the emerging area of deep learning or hierarchical learning to the APSIPA community. Deep learning refers to a class of machine learning techniques, developed largely since 2006, where many stages of non-linear information processing in hierarchical architectures are exploited for pattern classification and for feature learning. In the more recent literature, it is also connected to representation learning, which involves a hierarchy of features or concepts where higher-level concepts are defined from lower-level ones and where the same lower-level concepts help to define higher-level ones. In this tutorial survey, a brief history of deep learning research is discussed first. Then, a classificatory scheme is developed to analyze and summarize major work reported in the recent deep learning literature. Using this scheme, I provide a taxonomy-oriented survey on the existing deep architectures and algorithms in the literature, and categorize them into three classes: generative, discriminative, and hybrid. Three representative deep architectures – deep autoencoders, deep stacking networks with their generalization to the temporal domain (recurrent networks), and deep neural networks (pretrained with deep belief networks) – one in each of the three classes, are presented in more detail. Next, selected applications of deep learning are reviewed in broad areas of signal and information processing including audio/speech, image/vision, multimodality, language modeling, natural language processing, and information retrieval. Finally, future directions of deep learning are discussed and analyzed.

Subfields: Artificial Intelligence, Model Architectures, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Signal Processing

Citations in the corpus

A Survey of Deep Network Solutions for Learning Control in Robotics: From Reinforcement to Imitation (2016, neutral): “We refer readers to Deng (2014) and Guo et al. (2016) for a comprehensive overview of deep learning techniques for perception.”
Deep learning for visual understanding: A review (2016, background): “Deep networks have been shown to be successful for computer vision tasks because they can extract appropriate features while jointly performing discrimination [9,13].”
A Survey on Deep Learning (2018, background): “Among various machine-learning algorithms, “deep learning,” also known as representation learning [29], is widely used in these applications.”
All Weather Perception: Joint Data Association, Tracking, and Classification for Autonomous Ground Vehicles (2016, methodology): “The ﬁeld of deep learning and deformable parts models (Deng, 2014) and (Felzenszwalb et al., 2010b) has advanced from hand-designed feature detection methods to learn hierarchies of features in an unsupervised manner directly from data, vastly improving algorithm detection and classiﬁcation…”
Efficient Emotion Recognition from Speech Using Deep Learning on Spectrograms (2017, background): “Deep learning has contributed to breakthroughs across multiple areas of multimedia, including speech processing [16-17].”
Deep Learning Advances in Computer Vision with 3D Data (2017, methodology): “Deep Belief Networks (DBNs) are probabilistic generative models consisting of multiple layers of stochastic hidden variables [Deng 2014].”
AI-Based Modeling: Techniques, Applications and Research Issues Towards Automation, Intelligent and Smart Systems (2022, methodology): “Techniques in this category are mostly employed for unsupervised learning, as they are commonly used for feature learning or data generation and representation [19, 20].”
Identifying Top Sellers In Underground Economy Using Deep Learning-Based Sentiment Analysis (2014, methodology): “Deep learning algorithms are based on two building blocks: word embedding and the recursive neural network [12].”
A Survey of Deep Active Learning (2020, neutral): “DL is also called representation learning [50].”
Pooling Methods in Deep Neural Networks, a Review (2020, background): “There are mainly three important reasons for the booming of deep learning today: the dramatically increased chip processing abilities, the significantly lowered cost of computing hardware, and the considerable advances in the machine learning algorithms [7].”
Traffic Light Control Using Deep Policy-Gradient and Value-Function Based Reinforcement Learning (2017, background): “By utilising advances in deep learning and its application to different domains [11, 26, 27], deep learning has gained attention in the area of traffic management systems.”
Privacy-preserving deep learning (2015, background): “Surveys of deep-learning architectures, algorithms, and applications can be found in [5, 16].”