A State-of-the-Art Survey on Deep Learning Theory and Architectures

By Md. Zahangir Alom, Tarek M. Taha, C. Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Mahmudul Hasan, Brian Van Essen, Abdul A. S. Awwal, Vijayan K. Asari. Electronics, 2019

1,450 citations recorded by the source corpus.

In recent years, deep learning has garnered tremendous success in a variety of application domains. This new field of machine learning has been growing rapidly and has been applied to most traditional application domains, as well as some new areas that present more opportunities. Different methods have been proposed based on different categories of learning, including supervised, semi-supervised, and un-supervised learning. Experimental results show state-of-the-art performance using deep learning when compared to traditional machine learning approaches in the fields of image processing, computer vision, speech recognition, machine translation, art, medical imaging, medical information processing, robotics and control, bioinformatics, natural language processing, cybersecurity, and many others. This survey presents a brief survey on the advances that have occurred in the area of Deep Learning (DL), starting with the Deep Neural Network (DNN). The survey goes on to cover Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), including Long Short-Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). Additionally, we have discussed recent developments, such as advanced variant DL techniques based on these DL approaches. This work considers most of the papers published after 2012 from when the history of deep learning began. Furthermore, DL approaches that have been explored and evaluated in different application domains are also included in this survey. We also included recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys that have been published on DL using neural networks and a survey on Reinforcement Learning (RL). However, those papers have not discussed individual advanced techniques for training large-scale deep learning models and the recently developed method of generative models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8dd53f10ca5fa14faeed2bd2951d247f1ac60f40

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multivariate Temporal Convolutional Network: A Deep Neural Networks Approach for Multivariate Time Series Forecasting (2019, background): “As an important part of the ﬁeld of artiﬁcial intelligence, deep neural networks (DNNs) provide state-of-the-art accuracy on many tasks [10] and has been developed intensively in natural language processing (NLP), computer vision (CV), time series classiﬁcations and time series forecasting.”
A survey on deep learning tools dealing with data scarcity: definitions, challenges, solutions, tips, and applications (2023, background): “This paper compares the varied architectures of DL models [1, 144, 145].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, background): “Other more recent architectures encompass Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Bayesian Deep Learning (BDL) (Alom et al, 2019; Berman et al, 2019).”
Deepfake Video Detection Using Convolutional Vision Transformer (2021, background): “With the rapid advancement of the CNNs [4, 20], Generative Adversarial Networks (GANs) [18], and its variants [22], it is now possible to create hyper-realistic images [32], videos [61] and audio signals [53, 15] that are much harder to detect and distinguish from real untampered audiovisuals.”
Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages (2020, methodology): “Model-based TTS research has been dominated by statistical parametric speech synthesis (SPSS) [5]–[9] until recent years in which deep learning has delivered extraordinary achievements in various fields [10]–[12].”
Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer Interface: A Comprehensive Review (2020, background): “Conversely, in the decoding phase, actual features are regenerated from the lower-dimensional features with reverse processing (Alom et al., 2019).”
Epigenetics Analysis and Integrated Analysis of Multiomics Data, Including Epigenetic Data, Using Artificial Intelligence in the Era of Precision Medicine (2019, background): “AlexNet achieved state-of-the-art recognition accuracy against all the traditional machine learning and computer vision approaches, which was a significant breakthrough in the field of machine learning and computer vision for visual recognition and classification tasks and the point in history where”
COVID_MTNet: COVID-19 Detection with Multi-Task Deep Learning Approaches (2020, methodology): “The network is initialized with the He initialization method [21].”
Hyperspectral Image Classification—Traditional to Deep Models: A Survey for Future Prospects (2021, background): “• High Computational Burden: One of the main challenges of DNN is dealing with a big amount of data that involves increased memory bandwidth, high computational cost, and storage consumption [129].”
Efficient Acceleration of Deep Learning Inference on Resource-Constrained Edge Devices: A Review (2023, background): “There are available reviews of different DL techniques [5], DNNs on edge [2], [16], DL compression techniques for hardware acceleration [10], DL inference on FPGA-based platforms [17], edge intelligence [18], and edge computing [19].”