A Survey of the Usages of Deep Learning for Natural Language Processing

By Dan Otter, Julian R. Medina, Jugal Kumar Kalita. 2020

1,660 citations recorded by the source corpus.

Over the last several years, the field of natural language processing has been propelled forward by an explosion in the use of deep learning models. This article provides a brief introduction to the field and a quick overview of deep learning architectures and methods. It then sifts through the plethora of recent studies and summarizes a large assortment of relevant contributions. Analyzed research areas include several core linguistic processing issues in addition to many applications of computational linguistics. A discussion of the current state of the art is then provided along with recommendations for future research in the field.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Xiangnan He
3. Xiang Wang
4. Meng Wang
5. Le Wu
6. Kun Zhang
7. Bo Du
8. Zhijin Qin

Citations in the corpus

• Self-Polish: Enhance Reasoning in Large Language Models via Problem Refinement (2023, background): “Large language models (LLMs) have achieved impressive performance on a variety of NLP tasks (Brown et al., 2020; Otter et al., 2021; Chowdhery et al., 2022), but their capability to perform multi-step reasoning is considered a limitation, which can not be tackled solely by scaling up the model size…”
• Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, background): “We suggest readers refer to detailed reviews of neural networks in NLP in [101, 152].”
• Large-Scale Generative Data-Free Distillation (2020, background): “Recent advances in deep learning have dramatically accelerated machine learning progress in a wide variety of artificial cognition tasks, including vision [33, 36, 45, 50], speech [17] and natural language processing [44].”
• GPT-4 passes the bar exam (2024, neutral): “20,21 37 The Uniform Bar Exam”
• Comparing BERT against traditional machine learning text classification (2020, background): “On the other hand, Machine Learning (ML) [15] and deep learning based approaches [18] that classically have analyzed annotated corpora of texts inferring which features of the text, typically in a bag of words fashion [30] or by n-grams, are relevant for the classification automatically.”
• An Empirical Study of Remote Sensing Pretraining (2022, methodology): “Transformer is first proposed in [41], and has been widely used in the natural language processing (NLP) field [42],”
• Sample-Efficient Optimization in the Latent Space of Deep Generative Models via Weighted Retraining (2020, background): “Recently, machine learning has shown promising results in many problems that can be framed as optimization, such as conditional image [57, 40] and text [44] generation, molecular and materials design [11, 47], and neural architecture search [10].”
• A Survey of Visual Transformers (2021, background): “The most recent dominant models are the selfsupervised Transformers, which are pre-trained over sufficient datasets and then fine-tuned over a small sample set for a given downstream task [2]–[9].”
• Machine Learning: Algorithms, Real-World Applications and Research Directions (2021, background): “– NLP and sentiment analysis: Natural language processing (NLP) involves the reading and understanding of spoken or written language through the medium of a computer [79, 103].”
• Semantic Communications: Principles and Challenges (2021, positive): “The advancement on NLP [49] enables text coding to consider the semantic meaning of text, which motivates us to re-design the transceiver for achieving successful semantic information transmission.”
• Universal Differential Equations for Scientific Machine Learning (2020, methodology): “Recent advances in machine learning have been dominated by deep learning which utilizes readily available “big data” to solve previously difficult problems such as image recognition [1, 2, 3] and natural language processing [4, 5, 6].”
• esCorpius: A Massive Spanish Crawling Corpus (2022, background): “As pointed out in [5], most NLP research is conducted in English, followed by Mandarin Chinese, and at a great distance to other languages, including Spanish despite its large number of speakers around the world.”
• A Survey on Accuracy-Oriented Neural Recommendation: From Collaborative Filtering to Information-Rich Recommendation (2021, methodology): “Given the above neural based NLP models, we discuss some typical textual enhanced recommendation models based on the above techniques.”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “To the best of our knowledge, this survey is the first to review the recent progress of VLA models, a rapidly emerging research area.”
• A Survey on Generative Diffusion Models (2022, background): “Autoregressive language models generate text sequentially, ensuring high semantic coherence but slower generation speed [211].”
• Natural language processing: state of the art, current trends and challenges (2017, methodology): “that even though a great amount of work on natural language processing is available in literature surveys (one may refer to [15, 32, 63, 98, 133, 151] focusing on one domain such as usage of deep-learning techniques in NLP, techniques used for email spam filtering, medication safety, management rese”