Grammar as a Foreign Language

By Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, Geoffrey E. Hinton. 2014

942 citations recorded by the source corpus.

Syntactic constituency parsing is a fundamental problem in natural language processing and has been the subject of intensive research and engineering for decades. As a result, the most accurate parsers are domain specific, complex, and inefficient. In this paper we show that the domain agnostic attention-enhanced sequence-to-sequence model achieves state-of-the-art results on the most widely used syntactic constituency parsing dataset, when trained on a large synthetic corpus that was annotated using existing parsers. It also matches the performance of standard parsers when trained only on a small human-annotated dataset, which shows that this model is highly data-efficient, in contrast to sequence-to-sequence models without the attention mechanism. Our parser is also fast, processing over a hundred sentences per second with an unoptimized CPU implementation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, methodology): “In contrast to RNN sequence-to-sequence models [37], the Transformer outperforms the BerkeleyParser [29] even when training only on the WSJ training set of 40K sentences.”
Can Active Memory Replace Attention? (2016, methodology): “This model is identical as the one in [7], except that is uses GRU cells instead of LSTM cells.”
Neural GPUs Learn Algorithms (2015, background): “…networks with such cells can learn very complex tasks in an end-to-end manner, such as translation (Sutskever et al., 2014; Bahdanau et al., 2014; Cho et al., 2014), parsing (Vinyals & Kaiser et al., 2015), speech recognition (Chan et al., 2016) or image caption generation (Vinyals et al., 2014).”
Multi-task Sequence to Sequence Learning (2015, methodology): “The early seq2seq work (Sutskever et al., 2014; Cho et al., 2014; Luong et al., 2015b; Vinyals et al., 2015b) uses only the last encoder state to initialize the decoder and sets s = [ ] in Eq.”
Semi-supervised Sequence Learning (2015, methodology): “[32], which has been successfully used for machine translation [21, 11], text parsing [33], image captioning [35], video analysis [31], speech recognition [4] and conversational modeling [28, 34].”
Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning (2018, background): “(2016) also present evidence that sequence-to-sequence parsers (Vinyals et al., 2015) more strongly encode source language syntax.”
Creating Training Corpora for NLG Micro-Planners (2017, methodology): “pare the performance of a sequence-to-sequence model (Vinyals et al., 2015) on both datasets to estimate the complexity of the learning task induced by each dataset.”
A Diversity-Promoting Objective Function for Neural Conversation Models (2015, methodology): “Recent research has shown that deep LSTMs work better than single-layer LSTMs for S EQ 2S EQ tasks (Vinyals et al., 2015; Sutskever et al., 2014).”
Efficient Natural Language Response Suggestion for Smart Reply (2017, methodology): “For example, neural network models have been used to learn more robust parsers [14, 24, 29].”
TensorFlow: A system for large-scale machine learning (2016, neutral): “, and demonstrate the compelling performance that TensorFlow achieves for several real-world applications. 1 Introduction In recent years, machine learning has driven advances in many different ﬁelds [3, 5, 23, 24, 30, 27, 40, 45, 48, 50, 55, 68, 69, 73, 76]. We attribute this success to the inventi”
Order Matters: Sequence to sequence for sets (2015, neutral): “• A write block, which is an LSTM pointer network (Vinyals et al., 2015a) that takes in q ∗ T (as the context it needs from which to produce the output from the input set), and points at elements of m i (implicitly, x i ), one step at a time.”
TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems (2016, neutral): “We and others at Google have performed a wide variety of research using DistBelief including work on unsupervised learning [31], language representation [35, 52], models for image classification and object detection [16, 48], video classification [27], speech recognition [56, 21, 20],”
Adversarial Feature Learning (2016, background): “In other perceptual domains such as natural language processing or speech recognition, deep networks have proven highly effective as well [2, 11, 27, 29].”
Pix2seq: A Language Modeling Framework for Object Detection (2021, background): “…shown capable of modeling various sequential data, such as machine translation (Sutskever et al., 2014; Bahdanau et al., 2014), image captioning (Vinyals et al., 2015b; Karpathy & Fei-Fei, 2015; Xu et al., 2015), and many others (Vinyals et al., 2015a; Huang et al., 2018; Ramesh et al., 2021;…”
What you can cram into a single $&!#* vector: Probing sentence embeddings for linguistic properties (2018, methodology): “Following Vinyals et al. (2015), we train a seq2seq architecture to generate linearized grammatical parse trees (see Table 1) from source sentences ( Seq2Tree ).”
Pointer Networks (2015, background): “These developments have made it possible to apply RNNs to new domains, achieving state-of-the-art results in core problems in natural language processing such as translation [1, 5] and parsing [7], image and video captioning [8, 9], and even learning to execute small programs [2, 10].”
A Neural Conversational Model (2015, methodology): “It has also been used for other tasks such as parsing (Vinyals et al., 2014a) and image captioning (Vinyals et al., 2014b).”