Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling

By Diego Marcheggiani, Ivan Titov. 2017

885 citations recorded by the source corpus.

Semantic role labeling (SRL) is the task of identifying the predicate-argument structure of a sentence. It is typically regarded as an important step in the standard NLP pipeline. As the semantic representations are closely related to syntactic ones, we exploit syntactic information in our model. We propose a version of graph convolutional networks (GCNs), a recent class of neural networks operating on graphs, suited to model syntactic dependency graphs. GCNs over syntactic dependency trees are used as sentence encoders, producing latent feature representations of words in a sentence. We observe that GCN layers are complementary to LSTM ones: when we stack both GCN and LSTM layers, we obtain a substantial improvement over an already state-of-the-art LSTM SRL model, resulting in the best reported scores on the standard benchmark (CoNLL-2009) both for Chinese and English.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Graph Learning

Citations in the corpus

Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, methodology): “Largely inspired by Graph Convolution Networks (GCN) (Kipf and Welling, 2017; Marcheggiani and Titov, 2017), we investigate two methods to obtain”
Graph Convolutional Networks for Text Classification (2018, background): “One group of studies focused on models based on word embeddings (Mikolov et al. 2013; Pennington, Socher, and Manning 2014).”
Sentence-State LSTM for Text Representation (2018, background): “The closest previous work in NLP includes the use of convolutional neural networks (Bastings et al., 2017; Marcheggiani and Titov, 2017) and DAG LSTMs (Peng et al.”
Graph Convolutional Encoders for Syntax-aware Neural Machine Translation (2017, methodology): “Since syntactic GCNs produce representations at word level, it is straightforward to use them as encoders within the attention-based encoderdecoder framework.”
Learning to Represent Programs with Graphs (2017, methodology): “They have been used in a series of applications, such as link prediction and classiﬁcation (Grover & Leskovec, 2016) and semantic role labeling in NLP (Marcheggiani & Titov, 2017).”
Exploring Visual Relationship for Image Captioning (2018, methodology): “We devise our Graph Convolutional Networks plus Long Short-Term Memory (GCN-LSTM) architecture to generate image descriptions by additionally incorporating both semantic and spatial object relationships.”
Auto-Encoding Scene Graphs for Image Captioning (2018, background): “Thanks to the recent advances in spatial Graph Convolutional Networks (GCNs) [31, 24], we can embed the graph structure into vector representations, which can be seamlessly integrated into the encoder-decoder.”
Linguistically-Informed Self-Attention for Semantic Role Labeling (2018, background): “Still, recent work (Roth and Lapata, 2016; He et al., 2017; Marcheggiani and Titov, 2017) indicates that neural network models could see even higher performance gains by leveraging syntactic information rather than ignoring it.”
A Comprehensive Survey on Graph Neural Networks (2019, background): “Marcheggiani et al. [152] propose the Syntactic GCN which runs on top of a CNN/RNN sentence encoder.”
Graph Convolution over Pruned Dependency Trees Improves Relation Extraction (2018, methodology): “In this work, we propose a novel extension of the graph convolutional network (Kipf and Welling, 2017; Marcheggiani and Titov, 2017) that is tailored for relation extraction.”
How Do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking (2020, background): “We learn which edges are relevant in multi-hop question answering and graph-based semantic role labeling (Marcheggiani and Titov, 2017; De Cao et al., 2019).”
Graph convolutional networks: a comprehensive review (2019, methodology): “develop a syntactic graph convolutional network model that can be used on top of syntactic dependence trees, which is suitable for various NLP applications such as semantic role labeling [104], and neural machine translation [105].”
Graph Learning: A Survey (2021, methodology): “Syntactic GCN was proposed to solve semantic role labeling and neural machine translation [162].”
Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution (2019, methodology): “Marcheggiani and Titov (2017) introduces a way of incorporating syntactic information into sequential neural networks by using GCN.”
Dual Graph Convolutional Networks for Aspect-based Sentiment Analysis (2021, methodology): “Additionally, following (Marcheggiani and Titov, 2017), we add a self-loop for each node in the SynGCN and SemGCN modules.”
Interpreting Graph Neural Networks for NLP With Differentiable Edge Masking (2020, methodology): “For those that do (Marcheggiani & Titov, 2017; Veli ˇ ckovi ´ c et al., 2018; Neil et al., 2018; Xie & Grossman, 2018), such scores are, as we demonstrate in Section 6, not necessarily informative, as gates can function to scale rather than ﬁlter messages.”
DeepGCNs: Can GCNs Go As Deep As CNNs? (2019, background): “Graphs are also popular modes of representation in natural language processing [2, 23], where they are used to represent complex relations between large text units.”
Improving Robustness by Augmenting Training Sentences with Predicate-Argument Structures (2020, background): “Marcheggiani and Titov (2017) build a graph representation from the input text using their corresponding dependency relations and use graph convolutional networks (GCNs) to process the resulting graph for SRL.”
Reasoning with Latent Structure Refinement for Document-Level Relation Extraction (2020, positive): “(2019) also utilize co-reference links to construct the dependency graph and use labelled edge GCNs (Marcheggiani and Titov, 2017) for document-level relation extraction.”