Graph Convolutional Encoders for Syntax-aware Neural Machine Translation
By Jasmijn Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, Khalil Sima'an. 2017
519 citations recorded by the source corpus.
We present a simple and effective approach to incorporating syntactic structure into neural attention-based encoder-decoder models for machine translation. We rely on graph-convolutional networks (GCNs), a recent class of neural networks developed for modeling graph-structured data. Our GCNs use predicted syntactic dependency trees of source sentences to produce representations of words (i.e. hidden states of the encoder) that are sensitive to their syntactic neighborhoods. GCNs take word representations as input and produce word representations as output, so they can easily be incorporated as layers into standard encoders (e.g., on top of bidirectional RNNs or convolutional neural networks). We evaluate their effectiveness with English-German and English-Czech translation experiments for different types of encoders and observe substantial improvements over their syntax-agnostic versions in all the considered setups.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Graph Learning, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Noah A. Smith
- Caiming Xiong
- Yoav Goldberg
In Computer Vision and Pattern Recognition
- Dacheng Tao
- Luc Van Gool
- Mohit Bansal
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Bernt Schiele
- Yi Yang
In Graph Learning
- Shirui Pan
- Nicola De Cao
- Marco Damonte
- Shay B. Cohen
- Maosong Sun
- Michael Sejr Schlichtkrull
- Ivan Titov
- Philip S. Yu
In Machine Translation
- Daniel Beck
- Iryna Gurevych
- Ivan Vulic
- Goran Glavas
- Rui Wang
- Trevor Cohn
- Miguel Ballesteros
- Xiaodong He
Citations in the corpus
- Graph Convolutional Networks for Text Classification (2018, methodology): “GCN was also explored in several NLP tasks such as semantic role labeling (Marcheggiani and Titov 2017), relation classification (Li, Jin, and Luo 2018) and machine translation (Bastings et al. 2017), where GCN is used to encode syntactic structure of sentences.”
- Sentence-State LSTM for Text Representation (2018, background): “The closest previous work in NLP includes the use of convolutional neural networks (Bastings et al., 2017; Marcheggiani and Titov, 2017) and DAG LSTMs (Peng et al.”
- A Comprehensive Survey on Graph Neural Networks (2019, methodology): “Marcheggiani et al. [154] further adopt the same model as Bastings et al. [153] to handle the semantic dependency graph of a sentence.”
- A Survey of Knowledge-enhanced Text Generation (2020, background): “Syntactic dependency graph is a directed acyclic graph representing syntactic relations between words [6].”
- DeepGCNs: Can GCNs Go As Deep As CNNs? (2019, background): “Graphs are also popular modes of representation in natural language processing [2, 23], where they are used to represent complex relations between large text units.”
- Long-tail Relation Extraction via Knowledge Graph Embeddings and Graph Convolution Networks (2019, background): “GCNs have also had success in other NLP tasks such as semantic role labeling (Marcheggiani and Titov, 2017), dependency parsing (Strubell and McCallum, 2017), and machine translation (Bastings et al., 2017).”
- Scheduled Multi-Task Learning: From Syntax to Translation (2018, result): “2016; Kuncoro et al., 2017; Niehues and Cho, 2017; Sennrich and Haddow, 2016; Eriguchi et al., 2017; Aharoni and Goldberg, 2017; Nadejde et al., 2017; Bastings et al., 2017; Matthews et al., 2018) has proven to be beneficial, achieving higher results in language modeling and machine translation.”
- Higher-Order Explanations of Graph Neural Networks via Relevant Walks (2020, methodology): “Graph neural networks (GNNs) [1], [2] have been proposed as a method to learn from observations in general graph structures and have found use in an ever growing number of applications [3], [4], [5], [6], [7], [8].”
- Explainability Techniques for Graph Convolutional Networks (2019, background): “…Zitnik et al., 2018), physics (Battaglia et al., 2016; Chang et al., 2017; Gilmer et al., 2017; Watters et al., 2017; Sanchez-Gonzalez et al., 2018), visual recognition (Qi et al., 2017; 2018; Narasimhan et al., 2018), and natural language processing (Bastings et al., 2017; Beck et al., 2018).”
- Line Graph Enhanced AMR-to-Text Generation with Mix-Order Graph Attention Networks (2020, methodology): “Furthermore, our framework can be efficiently applied to other graph-to-sequence tasks such as WebNLG (Gardent et al., 2017) and syntax-based neural machine translation (Bastings et al., 2017).”
- Graph-to-Tree Neural Networks for Learning Structured Input-Output Translation with Applications to Semantic Parsing and Math Word Problem (2020, methodology): “Researchers recently attempted to apply the Graph Neural Networks (GNNs) in various NLP tasks, including the neural machine translation (Bastings et al., 2017; Beck et al., 2018), conversational machine reading comprehension (Chen et al., 2019), and AMR-to-text (Song et al., 2018).”
- Text Level Graph Neural Network for Text Classification (2019, methodology): “, 2018a), neural machine translation (Bastings et al., 2017), and relational reasoning (Battaglia et al.”
- Deep Learning on Graphs: A Survey (2018, unknown): “For NLP problems, GCNs can be adopted as syntactic constraints [141].”
- Structural Neural Encoders for AMR-to-text Generation (2019, methodology): “All systems are implemented in PyTorch (Paszke et al., 2017) using the framework OpenNMT-py (Klein et al., 2017).”
- Towards String-To-Tree Neural Machine Translation (2017, methodology): “…side: Eriguchi et al. (2016a,b) replace the encoder in an attention-based model with a Tree-LSTM (Tai et al., 2015) over a constituency parse tree; Bastings et al. (2017) encoded sentences using graph-convolutional networks over dependency trees; Sennrich and Haddow (2016) proposed a factored NMT…”