DeepStruct: Pretraining of Language Models for Structure Prediction

By Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, Dawn Song. 2022

98 citations recorded by the source corpus.

We introduce a method for improving the structural understanding abilities of language models. Unlike previous approaches that finetune the models with task-specific augmentation, we pretrain language models to generate structures from the text on a collection of task-agnostic corpora. Our structure pretraining enables zero-shot transfer of the learned knowledge that models have about the structure tasks. We study the performance of this approach on 28 datasets, spanning 10 structure prediction tasks including open information extraction, joint entity and relation extraction, named entity recognition, relation classification, semantic role labeling, event extraction, coreference resolution, factual probe, intent detection, and dialogue state tracking. We further enhance the pretraining with the task-specific training sets. We show that a 10B parameter language model transfers non-trivially to most tasks and obtains state-of-the-art performance on 21 of 28 datasets that we evaluate. Our code and datasets will be made publicly available.

Subfields: Artificial Intelligence, Large Language Models, Information Extraction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Yuxiao Dong
3. Xiao Liu
4. Ming Ding
5. Zhengxiao Du
6. W. Tam
7. Wenguang Chen
8. Hanyu Lai

In Large Language Models

1. Jie Tang
2. Ming Ding
3. Xiao Liu
4. Yuxiao Dong
5. Aohan Zeng
6. Yufei Xue
7. Wendi Zheng
8. Xiao Xia

In Information Extraction

1. Wenguang Chen
2. Jie Tang
3. Xiao Liu
4. Zhengxiao Du

Citations in the corpus

• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Fortunately, one of the attempts on Post-LN initialized with the newly-proposed DeepNorm (Wang et al., 2022b) generates promising training stability.”
• Large language models for generative information extraction: a survey (2023, background): “For example, DEEPSTRUCT [151] introduced structure pre-training on a collection of task-agnostic corpora to enhance the structural understanding of language models.”