Neural Network Methods for Natural Language Processing

By Yoav Goldberg. 2017

832 citations recorded by the source corpus.

Neural networks are a family of powerful machine learning models. This book focuses on the application of neural network models to natural language data. The first half of the book (Parts I and II) covers the basics of supervised machine learning and feed-forward neural networks, the basics of working with machine learning over language data, and the use of vector-based rather than symbolic representations for words. It also covers the computation-graph abstraction, which allows to easily define and train arbitrary neural networks, and is the basis behind the design of contemporary neural network software libraries. The second part of the book (Parts III and IV) introduces more specialized neural network architectures, including 1D convolutional neural networks, recurrent neural networks, conditioned-generation models, and attention-based models. These architectures and techniques are the driving force behind state-of-the-art algorithms for machine translation, syntactic parsing, and many other applications. Finally, we also discuss tree-shaped networks, structured prediction, and the prospects of multi-task learning.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, background): “ith an eye to transparency, we took a closer look at the analogy structure. In the original proportional analogy implementation, all terms of the equation A:B::C:D are distinct (Mikolov et al. 2013a; Goldberg 2017, p. 138). In other words, the model is forced to return a different concept than the o”
Hypothesis Only Baselines in Natural Language Inference (2018, background): “Goldberg (2017) suggests that “solving [NLI] perfectly entails human level understanding of language”, and Nangia et al.”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “…have produced visualizations ranging from tasks like NLI (Rocktäschel et al., 2016; Yin et al., 2016), summarization (Rush et al., 2015), MT post-editing (Jauregi Unanue et al., 2018), and morphological inﬂection (Aharoni and Goldberg, 2017), to matching users on social media (Tay et al., 2018).”
Evaluation of Text Generation: A Survey (2020, background): “stance between two sequences (e.g., sentences, paragraphs, etc.), each represented with relative word frequencies. It combines item similarity4 on bag-of-word (BOW) histogram representations of text (Goldberg et al., 2018) with word embedding similarity. In short, WMD has several intriguing properti”
What do RNN Language Models Learn about Filler–Gap Dependencies? (2018, background): “Many recent advancements in Natural Language Processing have come from the introduction of Recurrent Neural Networks (RNN) (Elman, 1990; Goldberg, 2017).”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, background): “in NLP demonstrating impressive improvements over traditional approaches on many tasks (Goldberg, 2017).”
A survey of deep learning techniques for autonomous driving (2019, background): “cial Intelligence (AI) became the main technologies behind many breakthroughs in computer vision (Krizhevsky et al., 2012), robotics (Andrychowicz et al., 2018) and Natural Language Processing (NLP) (Goldberg, 2017). They also have a major impact in the autonomous driving revolution seen today both ”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, background): “Machine translation (MT) has come a long way, steadily improving the quality of automatic systems, relying mostly on the advent of neural techniques (Goldberg, 2017) and availability of large data collections.”
RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency (2018, methodology): “Progress in natural language processing has recently come from deriving sentence representations using recurrent neural networks (RNNs) (Elman, 1990; Sutskever et al., 2014; Goldberg, 2017).”
Understanding Natural Language Understanding Systems. A Critical Analysis (2023, methodology): “7 Thus, a neural language model is an artificial neural network trained on a task of string prediction (Goldberg 2017).”
Linguistic generalization and compositionality in modern artificial neural networks (2019, background): “Gates have proven empirically extremely effective, and are standard in modern language-processing networks [38].”
Evaluating Text GANs as Language Models (2018, neutral): “, 2016) and many other applications (Goldberg, 2017).”
Deep Reinforcement Learning: An Overview (2017, neutral): “1 We refer readers to materials for further study: reinforcement learning (Sutton and Barto, 2017; Szepesv´ari, 2010; Bertsekas, 2012; Powell, 2011; Bertsekas and Tsitsiklis, 1996; Puterman, 2005; Littman, 2015; Kaelbling et al., 1996); deep learning (LeCun et al., 2015; Goodfellow et al., 2016; Ben”
Scheduled Multi-Task Learning: From Syntax to Translation (2018, methodology): “Søgaard and Goldberg (2016) demonstrated the potential of such an approach.”
Rearranging the Familiar: Testing Compositional Generalization in Recurrent Networks (2018, background): “Neural networks have recently been successfully applied to many tasks requiring considerable generalization abilities (LeCun et al., 2015), including applications in the domain of natural language (Goldberg, 2017).”