The unreasonable effectiveness of deep learning in artificial intelligence

By Terrence J. Sejnowski. Proceedings of the National Academy of Sciences of the United States of America, 2020

331 citations recorded by the source corpus.

Deep learning networks have been trained to recognize speech, caption photographs, and translate text between languages at high levels of performance. Although applications of deep learning networks to real-world problems have become ubiquitous, our understanding of why they are so effective is lacking. These empirical results should not be possible according to sample complexity in statistics and nonconvex optimization theory. However, paradoxes in the training and effectiveness of deep learning networks are being investigated and insights are being found in the geometry of high-dimensional spaces. A mathematical theory of deep learning would illuminate how they function, allow us to assess the strengths and weaknesses of different network architectures, and lead to major improvements. Deep learning has provided natural ways for humans to communicate with digital devices and is foundational for building artificial general intelligence. Deep learning was inspired by the architecture of the cerebral cortex and insights into autonomy and general intelligence may be found in other brain regions that are essential for planning and survival, but major breakthroughs will be needed to achieve these goals.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Computational Theory and Mathematics

View Paper: https://www.semanticscholar.org/paper/340712391e245e245a76cd74523c1d6443035b62

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Computational Theory and Mathematics

Citations in the corpus

A critical evaluation, challenges, and future perspectives of using artificial intelligence and emerging technologies in smart classrooms (2023, background): “The last few years, the topic of AI has been empowered by the groundbreaking technology of deep learning (Sejnowski, 2020) that allowed the successful application of AI to several complex machine learning tasks.”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, background): “The modern deep learning (DL) “revolution” was ushered in on the back of computer vision and image recognition [14, 15].”
Machine learning and deep learning—A review for ecologists (2022, unknown): “The question of why deep neural networks do not suffer from overparameterization, but instead even depend on it for making accurate predictions is still heavily debated (Sejnowski, 2020 ), and we will comment on this later.”
A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data (2021, background): “While there have been rapid developments in machine learning in the last 20 years and there is currently plenty of euphoria and admiration about the so-called “unreasonable effectiveness of deep learning in artificial intelligence” [5], the development of physics-informed machine learning and its ap”