A Primer on Neural Network Models for Natural Language Processing

By Yoav Goldberg. 2015

1,239 citations recorded by the source corpus.

Over the past few years, neural networks have re-emerged as powerful machine-learning models, yielding state-of-the-art results in fields such as image recognition and speech processing. More recently, neural network models started to be applied also to textual natural language signals, again with very promising results. This tutorial surveys neural network models from the perspective of natural language processing research, in an attempt to bring natural-language researchers up to speed with the neural techniques. The tutorial covers input encoding for natural language tasks, feed-forward networks, convolutional networks, recurrent networks and recursive networks, as well as the computation graph abstraction for automatic gradient computation.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

A Sensitivity Analysis of (and Practitioners’ Guide to) Convolutional Neural Networks for Sentence Classification (2015, methodology): “Convolutional Neural Networks (CNNs) have recently been shown to achieve impressive results on the practically important task of sentence categorization (Kim, 2014; Kalchbrenner et al., 2014; Wang et al., 2015; Goldberg, 2015; Iyyer et al., 2015).”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “The work by [6] only presented the basic principles for applying neural networks to NLP in a tutorial manner.”
Decision support from financial disclosures with deep neural networks and transfer learning (2017, methodology): “2 The utilized deep learning techniques are described in detail in [29, 17].”
Deep multi-task learning with low level tasks supervised at lower layers (2016, background): “For further details on RNNs and LSTMs, see (Goldberg, 2015; Cho, 2015).”
Automatic differentiation in machine learning: a survey (2015, positive): “Natural language processing (NLP) constitutes one of the areas where rapid progress is being made by applying deep learning techniques (Goldberg, 2016), with applications in tasks including machine translation (Bahdanau et al.”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, methodology): “Goyal et al. (2016) found an improvement in the quality of generated dialogue acts when using a character-based, rather than a word-based rnn.”
Understanding Convolutional Neural Networks for Text Classification (2018, background): “Specifically, current common wisdom suggests that CNNs classify text by working through the following steps (Goldberg, 2016):”
Rationale-Augmented Convolutional Neural Networks for Text Classification (2016, background): “Neural models that exploit word embeddings have recently achieved impressive results on text classification tasks (Goldberg, 2015).”
Are Distributional Representations Ready for the Real World? Evaluating Word Vectors for Grounded Perceptual Meaning (2017, neutral): “large-scale natural language processing applications (e.g. Turney and Pantel, 2010; Turian et al., 2010; Collobert and Weston, 2008; Socher et al., 2013; Goldberg, 2016).”
Easy-First Dependency Parsing with Hierarchical Tree LSTMs (2016, methodology): “1997) which is shown to be a very capable sequence learner. However, our algorithm and encoding method do not rely on any speciﬁc property of the LSTM architecture, and the 1We follow the notation of Goldberg (2015), with the exception of taking the output of the RNN to be a single vector rather tha”
Text as Data (2017, background): “They form the basis for many deep learning applications involving textual data (see, e.g., Chen and Manning 2014; Goldberg 2016).”
Neural information retrieval: at the end of the early years (2017, background): “For introductions to deep learning approaches to other domains, see Hinton et al. (2012) for ASR, Goldberg (2016) for NLP, and Wu et al. (2016) for MT. Goldberg (2016) covers details on training neural networks and a broader set of architectures including feedforward NN, convolutional neural…”
An End-to-End Automatic Cloud Database Tuning System Using Deep Reinforcement Learning (2019, methodology): “Although deep learning has successfully been applied to solving computationally intensive learning tasks in many domains [15, 17, 18, 24, 48], there are few studies that have used deep learning techniques to solve database tuning problems[49].”
DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning (2017, methodology): “Due to space constraints, we refer the reader to NLP primers (e.g., [12]) for a formal characterization of LSTMs. Thetrainingstepentailsfindingproper assignmentstotheweights so that the final output of the sequence of LSTMs produces the desired label (output) that comes with inputs in the training…”