Review of Deep Learning Algorithms and Architectures

By A. Shrestha, Ausif Mahmood. IEEE Access, 2019

1,518 citations recorded by the source corpus.

Deep learning (DL) is playing an increasingly important role in our lives. It has already made a huge impact in areas, such as cancer diagnosis, precision medicine, self-driving cars, predictive forecasting, and speech recognition. The painstakingly handcrafted feature extractors used in traditional learning, classification, and pattern recognition systems are not scalable for large-sized data sets. In many cases, depending on the problem complexity, DL can also overcome the limitations of earlier shallow networks that prevented efficient training and abstractions of hierarchical representations of multi-dimensional training data. Deep neural network (DNN) uses multiple (deep) layers of units with highly optimized algorithms and architectures. This paper reviews several optimization methods to improve the accuracy of the training and to reduce training time. We delve into the math behind training algorithms used in recent deep networks. We describe current shortcomings, enhancements, and implementations. The review also covers different types of deep architectures, such as deep convolution networks, deep residual networks, recurrent neural networks, reinforcement learning, variational autoencoders, and others.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f49f37c23158f8b698545e6f0a25877b39601ce8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning for Land Use and Land Cover Classification Based on Hyperspectral and Multispectral Earth Observation Data: A Review (2020, background): “The authors of [136] provide a review discussing the optimisation methods vs.”
Within the Lack of Chest COVID-19 X-ray Dataset: A Novel Detection Model Based on GAN and Deep Transfer Learning (2020, background): “Table 2 shows a series of major contributions in the ﬁeld of the neural network to deep learning [21].”
Deep Learning for Change Detection in Remote Sensing Images: Comprehensive Review and Meta-Analysis (2020, background): “Hence, this function D(s, a) is represented as the summation of current reward R and future reward α · max[D(s, a)] in the following way [120]:”
A Comprehensive Overview and Comparative Analysis on Deep Learning Models: CNN, RNN, LSTM, GRU (2023, background): “The units sum the inputs after multiplying them by their corresponding weights [4].”
Hierarchical Transfer Learning for Multilingual, Multi-Speaker, and Style Transfer DNN-Based TTS on Low-Resource Languages (2020, methodology): “Model-based TTS research has been dominated by statistical parametric speech synthesis (SPSS) [5]–[9] until recent years in which deep learning has delivered extraordinary achievements in various ﬁelds [10]–[12].”
Machine Learning Applied to Electrified Vehicle Battery State of Charge and State of Health Estimation: State-of-the-Art (2020, methodology): “For a more detailed explanation about the methods and its basic math, interested readers are referred to [9], [10].”
A review of machine learning in processing remote sensing data for mineral exploration (2021, background): “Deep learning methods have the ability to model complex and large datasets and are loosely grouped into feedforward and recurrent neural network architectures (Shrestha and Mahmood, 2019).”
Object Detection and Image Segmentation with Deep Learning on Earth Observation Data: A Review-Part I: Evolution and Recent Trends (2020, background): “Such a network then transports values from the input layer via the weights over all hidden layers to the output layer and can be seen as a non-linear function of higher order, mapping input values to outputs [5,25].”
Scientific Machine Learning Through Physics–Informed Neural Networks: Where we are and What’s Next (2022, background): “The main principle of reinforcement learning is to have an agent learn from its surroundings through exploration and by defining a reward (Shrestha and Mahmood, 2019).”
Understanding of Machine Learning with Deep Learning: Architectures, Workflow, Applications and Future Directions (2023, background): “…rules to function; rather, it leverages a massive amount of data to map the provided input to particular labels. is created utilising multiple layers of algorithms (arti ﬁ cial neural networks, or ANNs), each of which delivers a unique interpretation of the data provided to it [25,26].”