Structured Prediction as Translation between Augmented Natural Languages

By Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, C. D. Santos, Bing Xiang, Stefano Soatto. 2021

364 citations recorded by the source corpus.

We propose a new framework, Translation between Augmented Natural Languages (TANL), to solve many structured prediction language tasks including joint entity and relation extraction, nested named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, and dialogue state tracking. Instead of tackling the problem by training task-specific discriminative classifiers, we frame it as a translation task between augmented natural languages, from which the task-relevant information can be easily extracted. Our approach can match or outperform task-specific models on all tasks, and in particular, achieves new state-of-the-art results on joint entity and relation extraction (CoNLL04, ADE, NYT, and ACE2005 datasets), relation classification (FewRel and TACRED), and semantic role labeling (CoNLL-2005 and CoNLL-2012). We accomplish this while using the same architecture and hyperparameters for all tasks and even when training a single model to solve all tasks at the same time (multi-task learning). Finally, we show that our framework can also significantly improve the performance in a low-resource regime, thanks to better use of label semantics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “We map original semantic roles in PropBank to human-readable labels ( i.e., ARG0 → AGENT ) in order to leverage knowledge learned by pretrained models about roles’ meanings (Paolini et al., 2021).”
GLM: General Language Model Pretraining with Autoregressive Blank Infilling (2021, neutral): “Similarly, Athiwaratkun et al. (2020 2020) also study blanking inﬁlling models.”
Unified Structure Generation for Universal Information Extraction (2022, methodology): “1-base is an initial model of UIEbase; 2) Fine-tuned T5-base is fine-tuned with sequence generation tasks such as summarization, which have been shown effective in many lowresource NLP tasks (Paolini et al., 2021); 3) UIEbase w/o SSI is the distant supervised version of UIE without SSI in the pre-tr”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “We use a preprocessed version of this dataset from (Yu et al., 2020a) and use the same type names with TANL (Paolini et al., 2021).”
Text2Event: Controllable Sequence-to-Structure Generation for End-to-end Event Extraction (2021, methodology): “…using token annotation: TANL is the SOTA sequence generation-based method that models event extraction as a trigger-argument pipeline manner (Paolini et al., 2021); Multi-task TANL extends TANL by transferring structure knowledge from other tasks; EEQA (Du and Cardie, 2020) and MQAEE (Li et…”
Aspect Sentiment Quad Prediction as Paraphrase Generation (2021, methodology): “Inspired by recent success in formulating various NLP tasks as text generation problems (Athi-waratkun et al., 2020; Paolini et al., 2021; Liu et al., 2021), we propose to tackle ASQP in a sequence-to-sequence (S2S) manner in this paper.”
All NLP Tasks Are Generation Tasks: A General Pretraining Framework (2021, neutral): “Athiwaratkun et al. (2020); Paolini et al. (2021) propose augmented natural language for structured prediction tasks such as sequence tagging and relation extraction.”
Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “Another example is the CoNLL-2009 dataset [58] for dependency-based Semantic Role Labeling [55], which includes texts taken mostly from the Wall Street Journal and is skewed towards finance-related news.”
Large language models for generative information extraction: a survey (2023, neutral): “Therefore, there has been a recent surge of interest in generative IE methods [28] that adopt LLMs to generate structural information rather than extracting structural information from plain text.”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “Second, the direct generation of labels in output sequences allows the PLMs to exploit the semantics of the labels to improve the performance and data eiciency, a beneit that cannot be achieved in discriminative models [131].”