Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data

By Jacob A. Harer, Christopher P. Reale, Peter Chin. 2019

48 citations recorded by the source corpus.

Many common sequential data sources, such as source code and natural language, have a natural tree-structured representation. These trees can be generated by fitting a sequence to a grammar, yielding a hierarchical ordering of the tokens in the sequence. This structure encodes a high degree of syntactic information, making it ideal for problems such as grammar correction. However, little work has been done to develop neural networks that can operate on and exploit tree-structured data. In this paper we present the Tree-Transformer \textemdash{} a novel neural network architecture designed to translate between arbitrary input and output trees. We applied this architecture to correction tasks in both the source code and natural language domains. On source code, our model achieved an improvement of $25\%$ $\text{F}0.5$ over the best sequential method. On natural language, we achieved comparable results to the most complex state of the art systems, obtaining a $10\%$ improvement in recall on the CoNLL 2014 benchmark and the highest to date $\text{F}0.5$ score on the AESW benchmark of $50.43$.

Subfields: Artificial Intelligence, Information Systems, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Model Architectures

Citations in the corpus

Unified Pre-training for Program Understanding and Generation (2021, background): “…colossal effort to pretrain models on a massive amount of unlabeled data (e.g., text, images, videos) (Devlin et al., 2019; Liu et al., 2019; Conneau and Lample, 2019; Conneau et al., 2020; Li et al., 2019; Sun et al., 2019) to transfer representation encoders across a wide variety of applications.”
A Transformer-based Approach for Source Code Summarization (2020, methodology): “Recent works in code summarization utilize structural information of a program in the form of Abstract Syntax Tree (AST) that can be encoded using tree structure encoders such as Tree-LSTM (Shido et al., 2019), Tree-Transformer (Harer et al., 2019), and Graph Neural Network (LeClair et al., 2020).”
Code Prediction by Feeding Trees to Transformers (2020, methodology): “As for learning source code, it has been showed that taking tree structured helped code correction [19] and code translation [37].”