Novel positional encodings to enable tree-based transformers

By Vighnesh Leonardo Shiv, Chris Quirk. 2019

164 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Code Llama: Open Foundation Models for Code (2023, neutral): “…of programs, with specialized objectives for code deobfuscation (Lachaux et al., 2021), contrastive learning through semantic-preserving code transformations (Jain et al., 2021), leveraging Abstract Syntax Trees to learn tree-aware positional encodings (Shiv & Quirk, 2019; Peng et al., 2021).”
Mathematical Reasoning via Self-supervised Skip-tree Training (2020, methodology): “Transformer models for program understanding have focused on providing inductive biases in the architecture [Shiv and Quirk, 2019, Hellendoorn et al., 2020], whereas this work suggests to use a modiﬁed language modeling proxy task.”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “A novel positional encoding scheme is used to extend the Transformer to treestructured data [115].”
TUTA: Tree-based Transformers for Generally Structured Table Pre-training (2020, methodology): “Later work proposed a hybrid neural network by combining bidirectional RNNs and CNNs in the task of column type prediction [5].”
Rethinking Positional Encoding in Language Pre-training (2020, background): “For example, Shiv & Quirk (2019) extended the sequence positional encoding to tree-based positional encoding in Transformer; Wang et al. (2019) extended positional encoding to complex-valued domain; Liu et al. (2020) modeled the positional encoding by dynamical systems.”
A Transformer-based Approach for Source Code Summarization (2020, methodology): “To learn the order of tokens in a sequence or to model the relationship between tokens, Transformer requires to be injected with positional encodings (Vaswani et al., 2017; Shaw et al., 2018; Shiv and Quirk, 2019).”
Language-Agnostic Representation Learning of Source Code from Structure and Context (2021, background): “Shiv & Quirk (2019) propose positional encodings for nodes on trees; however, their approach assumes regular trees, which is an unrealistic assumption when working with Ab-stract Syntax Trees, as an AST node can have arbitrarily many children, e.g., the arguments of a function.”
Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Tree-based PE was proposed both for APE (Shiv & Quirk, 2019; Xiao et al., 2019; Ma et al., 2019) and RPE (Omote et al., 2019).”
DreamCoder: bootstrapping inductive program synthesis with wake-sleep library learning (2021, unknown): “recurrent networks [13] and transformers [48]).”
Code Prediction by Feeding Trees to Transformers (2020, methodology): “Recently, [26] tried to introduce tree structures to Transformer models via positional encoding.”
A Survey on Pretrained Language Models for Neural Code Intelligence (2022, background): “Although prior work makes a few attempts in exploiting these knowledge (Peng et al., 2021a; Shiv and Quirk, 2019; Guo et al., 2021; Wang et al., 2021a), there is still a lack of principled way to inject structure bias in language models designed for sequential data.”
Language Modeling for Formal Mathematics (2020, methodology): “Transformer models for program understanding have focused on providing inductive biases in the architecture [Shiv and Quirk, 2019, Hellendoorn et al., 2020], whereas this work suggests to use a modiﬁed language modeling proxy task.”
Integrating Tree Path in Transformer for Code Representation (2021, background): “Several works [Shen et al., 2018, Shiv and Quirk, 2019] proposed new architectures for modeling tree structures, including in the domain of natural languages processing [Wang et al., 2019] and code representation learning [Hellendoorn et al., 2019, Zügner et al., 2021].”