Extracting Syntactic Trees from Transformer Encoder Self-Attentions

By David Marecek, Rudolf Rosa. 2018

37 citations recorded by the source corpus.

This is a work in progress about extracting the sentence tree structures from the encoder’s self-attention weights, when translating into another language using the Transformer neural network architecture. We visualize the structures and discuss their characteristics with respect to the existing syntactic theories and annotations.

Subfields: Artificial Intelligence, Information Extraction, Computational Linguistics & Semantics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Extraction

In Computational Linguistics & Semantics

Citations in the corpus

What Does BERT Look at? An Analysis of BERT’s Attention (2019, background): “Marecek and Rosa (2018) propose heuristic ways of converting attention scores to syntactic trees, but do not quantitatively evaluate their approach.”
Adaptively Sparse Transformers (2019, background): “As suggested by recent works on interpreting attention head roles, separate attention heads may learn to look for various relationships between tokens (Tang et al., 2018; Ra-ganato and Tiedemann, 2018; Mareˇcek and Rosa, 2018; Tenney et al., 2019; Voita et al., 2019).”
Do Attention Heads in BERT Track Syntactic Dependencies? (2019, background): “In their preliminary work, Mareček and Rosa (2018) aggregate the attention weights across the self-attention layers and heads to form a single attention weight”
On Identifiability in Transformers (2019, neutral): “Our results suggest that some of the conclusions in prior work (Vaswani et al., 2017; Vig, 2019; Marecek & Rosa, 2018; Clark et al., 2019; Raganato & Tiedemann, 2018; Voita et al., 2019; Tang et al., 2018; Wangperawong, 2018; Padigela et al., 2019; Baan et al., 2019; Lin et al., 2019; Dehghani et…”
From Balustrades to Pierre Vinken: Looking for Syntax in Transformer Self-Attentions (2019, unknown): “A similar approach was already suggested (but not evaluated) in (Mareček and Rosa, 2018).”
exBERT: A Visual Analysis Tool to Explore Learned Representations in Transformer Models (2020, background): “Some showed that Trans-former models, similar to recurrent models, can effectively encode syntactic properties in their representations (Raganato and Tiedemann, 2018b; Mareˇcek and Rosa, 2018).”