Modeling Graph Structure in Transformer for Better AMR-to-Text Generation

By Jie Zhu, Jiehan Zhu, Junhui Li, Muhua Zhu, Longhua Qian, Guodong Zhou. 2019

105 citations recorded by the source corpus.

Recent studies on AMR-to-text generation often formalize the task as a sequence-to-sequence (seq2seq) learning problem by converting an Abstract Meaning Representation (AMR) graph into a word sequences. Graph structures are further modeled into the seq2seq framework in order to utilize the structural information in the AMR graphs. However, previous approaches only consider the relations between directly connected concepts while ignoring the rich structure in AMR graphs. In this paper we eliminate such a strong limitation and propose a novel structure-aware self-attention approach to better model the relations between indirectly connected concepts in the state-of-the-art seq2seq model, i.e. the Transformer. In particular, a few different methods are explored to learn structural representations between two concepts. Experimental results on English AMR benchmark datasets show that our approach significantly outperforms the state-of-the-art with 29.66 and 31.82 BLEU scores on LDC2015E86 and LDC2017T10, respectively. To the best of our knowledge, these are the best results achieved so far by supervised models on the benchmarks.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing, Model Architectures, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

In Model Architectures

In Natural Language Generation

Citations in the corpus

DART: Open-Domain Structured Data Record to Text Generation (2020, background): “…2018; Gehrmann et al., 2018; Puduppully et al., 2018, 2019; Iso et al., 2019; Castro Ferreira et al., 2019; Shen et al., 2019; Roberti et al., 2019; Zhu et al., 2019b; Koncel-Kedziorski et al., 2019; Ye et al., 2020; Harkous et al., 2020; Wang et al., 2020b; Song et al., 2020; Shahidi et al.,…”
Structural Information Preserving for Graph-to-Text Generation (2020, methodology): “The second group reports our systems, where the RA-Trans-F-ours baseline is our implementation of the feature-based model of Zhu et al. (2019).”
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020, background): “All signiﬁcance tests refer to a comparison with (Zhu et al., 2019). with the latter ( 65 .”
Have Your Text and Use It Too! End-to-End Neural Data-to-Text Generation with Semantic Fidelity (2020, methodology): “For a fair comparison, we lower-case our generated texts for the LDC2017T10 dataset to match the outputs of Zhu et al. (2019).”
A Human Evaluation of AMR-to-English Generation Systems (2020, methodology): “…approaches include using structure-aware self-attention to capture relations between concepts within a sequence-to-sequence transformer model (Zhu et al., 2019), and generating syntactic constituency trees as an intermediate step before generating surface structure (Cao and Clark, 2019).”
Modeling Global and Local Node Contexts for Text Generation from Knowledge Graphs (2020, methodology): “Systems based on the global encoding strategy are typically based on Transformer architectures (Zhu et al., 2019; Cai and Lam, 2020), using self-attention to compute a node representation based on all nodes in the graph.”
Line Graph Enhanced AMR-to-Text Generation with Mix-Order Graph Attention Networks (2020, methodology): “Comparing to previous studies, our approach with K = 4 order neighborhood information reaches the best BLEU scores, improving over the state-of-the-art model (Zhu et al., 2019) by 0.92 on both datasets.”