The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches

By Md. Zahangir Alom, Tarek M. Taha, C. Yakopcic, Stefan Westberg, Paheding Sidike, Mst Shamima Nasrin, Brian Van Essen, Abdul A. S. Awwal, Vijayan K. Asari. 2018

1,014 citations recorded by the source corpus.

Deep learning has demonstrated tremendous success in variety of application domains in the past few years. This new field of machine learning has been growing rapidly and applied in most of the application domains with some new modalities of applications, which helps to open new opportunity. There are different methods have been proposed on different category of learning approaches, which includes supervised, semi-supervised and un-supervised learning. The experimental results show state-of-the-art performance of deep learning over traditional machine learning approaches in the field of Image Processing, Computer Vision, Speech Recognition, Machine Translation, Art, Medical imaging, Medical information processing, Robotics and control, Bio-informatics, Natural Language Processing (NLP), Cyber security, and many more. This report presents a brief survey on development of DL approaches, including Deep Neural Network (DNN), Convolutional Neural Network (CNN), Recurrent Neural Network (RNN) including Long Short Term Memory (LSTM) and Gated Recurrent Units (GRU), Auto-Encoder (AE), Deep Belief Network (DBN), Generative Adversarial Network (GAN), and Deep Reinforcement Learning (DRL). In addition, we have included recent development of proposed advanced variant DL techniques based on the mentioned DL approaches. Furthermore, DL approaches have explored and evaluated in different application domains are also included in this survey. We have also comprised recently developed frameworks, SDKs, and benchmark datasets that are used for implementing and evaluating deep learning approaches. There are some surveys have published on Deep Learning in Neural Networks [1, 38] and a survey on RL [234]. However, those papers have not discussed the individual advanced techniques for training large scale deep learning models and the recently developed method of generative models [1].

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “The current deep learning movement would not have been possible without ImageNet [Alom et al., 2018].”
Large image datasets: A pyrrhic win for computer vision? (2020, methodology): “The vastness of this dataset allowed a Convolutional Neural Network (CNN) with 60 million parameters [49] trained by the SuperVision team from University of Toronto to usher in the rebirth of the CNN-era (see [3]), which is now widely dubbed the AlexNet moment in AI.”
Deep learning tools for the measurement of animal behavior in neuroscience (2019, background): “The current deep learning revolution started with achieving human-level accuracy for object recognition on the ImageNet challenge, a popular benchmark with many categories and millions of images [16 ,17].”
Machine Learning Algorithms in Civil Structural Health Monitoring: A Systematic Review (2020, background): “It is still evolving to address difficulties such as the random shapes and [115], AlexNet [6], ResNet [131], VGG-16 [1], YOLO object detection [102] are frequently reported.”
The AI gambit: leveraging artificial intelligence to combat climate change—opportunities, challenges, and recommendations (2021, background): “As described by Thompson et al. (2020, 8), during the “ multicore era ” , DL was “ ported to GPUs, initially yielding a 5 − 15 × speed-up which by 2012 had grown to more than 35 × ” and which led to the AlexNet breakthrough in 2012 (Alom et al. 2018).”
Breast Cancer Classification from Histopathological Images with Inception Recurrent Residual Convolutional Neural Network (2018, methodology): “We used only 1×1 and 3×3 convolution filters in this implementation, as inspired by the NiN [6] and Squeeze Net [6] models.”
Convolutional neural network: a review of models, methodologies and applications to object detection (2019, background): “Though DL has advancements over traditional machine learning (ML), it faces various challenges [33] including: 1.”
Nuclei Segmentation with Recurrent Residual Convolutional Neural Networks based U-Net (R2U-Net) (2018, background): “Therefore, the computational pathology [2] and microscopy images play a big role in decision making for disease diagnosis, since these images able to provide a large amount of information for computer-aided diagnosis (CAD).”
Compute Trends Across Three Eras of Machine Learning (2022, background): “Many authors decide to start the Deep Learning Era with the release of AlexNet in 2012 [4, 23], but there is some room for debate regarding this, and we instead believe that 2010 is most inline with the available evidence:”
A review of convolutional neural networks in computer vision (2024, neutral): “Meanwhile, AlexNet became the dividing line between traditional and deep learning algorithms and was the first deep CNN model in modern times.”
3D Deep Learning on Medical Images: A Review (2020, background): “Further, this journey can be traced with some of the most important development phases such as continuous Back Propagation Model (1960), convolutional neural networks (1979), LSTM (long short-term memory) (1997), ImageNet (2009), AlexNet (2011) [20], [21].”
Activation Functions: Comparison of trends in Practice and Research for Deep Learning (2018, methodology): “The learning process is the process of estimating the models parameters such that the model can perform the specified task [4].”
SSDMNV2: A real time DNN-based face mask detection system using single shot multibox detector and MobileNetV2 (2020, neutral): “For this purpose, LeNet-5 (LeCun, Bottou, Bengio, & Haffner, 1998), AlexNet (Alom et al., 2018), VGG-16 (Simonyan & Zisserman, 2014), and ResNet-50 (He, Zhang, Ren, & Sun, 2016) were chosen.”
Recurrent Residual Convolutional Neural Network based on U-Net (R2U-Net) for Medical Image Segmentation (2018, methodology): “Convolutional Neural Network (DCNN) models have been proposed such as AlexNet [1], VGG [5], GoogleNet [6], Residual Net [7], DenseNet [8], and CapsuleNet [9][65].”
The De-democratization of AI: Deep Learning and the Compute Divide in Artificial Intelligence Research (2020, background): “Other computer scientists (Alom et al., 2018; Russakovsky et al., 2015) and journalists (Gershgorn, 2018) also supported this claim.”
Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks (2020, background): “Furthermore, Alom et al.[9] suggest that it should be easy to calculate an activation function spare completing”
A review of machine learning in processing remote sensing data for mineral exploration (2021, background): “Furthermore, pre-trained CNN architectures are widely shared openly that address the limitations of computational resource needs for large datasets and models (Tahmasebi et al., 2020; Alom et al., 2018).”
Thin MobileNet: An Enhanced MobileNet Architecture (2019, background): “The advent of deeper and complicated models [1, 2, 3, 4, 6] has led to the development of a wide number of hardware architectures like GPUs, Bluebox 2.0, S32V234 MCU, etc to increase the speed of the training process and deploy the models for various computer vision applications.”