StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding

By Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, Luo Si. 2019

301 citations recorded by the source corpus.

Recently, the pre-trained language model, BERT (and its robustly optimized version RoBERTa), has attracted a lot of attention in natural language understanding (NLU), and achieved state-of-the-art accuracy in various NLU tasks, such as sentiment classification, natural language inference, semantic textual similarity and question answering. Inspired by the linearization exploration work of Elman [8], we extend BERT to a new model, StructBERT, by incorporating language structures into pre-training. Specifically, we pre-train StructBERT with two auxiliary tasks to make the most of the sequential order of words and sentences, which leverage language structures at the word and sentence levels, respectively. As a result, the new model is adapted to different levels of language understanding required by downstream tasks. The StructBERT with structural pre-training gives surprisingly good empirical results on a variety of downstream tasks, including pushing the state-of-the-art on the GLUE benchmark to 89.0 (outperforming all published models), the F1 score on SQuAD v1.1 question answering to 93.0, the accuracy on SNLI to 91.7.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “This has led to the development of a wide variety of pre-training objectives [Dai and Le, 2015; Ramachandran et al., 2016; Radford et al., 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019b; Wang et al., 2019a; Song et al., 2019; Dong et al., 2019; Joshi et al., 2019].”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, background): “Since 2018, we have seen the rise of a set of large-scale Transformer-based Pre-trained Language Models (PLMs), such as GPT [2, 3], BERT [4], RoBERTa [5], XLNet [6], UniLM [7], ELECTRA [8], T5 [9], ALUM [10], StructBERT [11] and ERINE [12] .”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “See also the n-gram word order reconstruction task (Wang et al., 2019a).”
M6: A Chinese Multimodal Pretrainer (2021, background): “Pretraining has become a focus in the research in natural language processing (NLP) [1, 2, 7, 16, 18, 19, 27, 31, 37, 44, 49].”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, methodology): “Two other knowledge-enhanced MLMs are KEP-LER (Wang et al., 2019c) and K-Adapter (Wang et al., 2020), which are based on Roberta (Liu et al., 2019) rather than BERT.”
COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “For example, next sentence prediction [11], sentence ordering [27] and previous sentence prediction [56] concatenate two sentences (either correlated or random), and train the Transformer to classify the pair.”
SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levels (2021, unknown): “For the model architecture, we pre-train a 12-layer SemVLP-base model with hidden size of 768, where we initialize it with the parameters from StructBERT base model (Wang et al., 2019).”
Attention mechanism in neural networks: where it comes and where it goes (2022, unknown): “StructBERT proposes two novel linearization strategies [155].”
Out of Order: How important is the sequential order of words in a sentence in Natural Language Understanding tasks? (2020, methodology): “2019) that outperform humans on GLUE, making an important case for studying their behaviors; (3) it is unknown how sensitive GLUE models are to word order and whether GLUE tasks require models to be wordorder sensitive (Wang et al., 2020).”
SLM: Learning a Discourse Language Representation with Sentence Unshuffling (2020, background): “, 2020), reordering of local permutations (Wang et al., 2019), detecting incorrectly replaced tokens (Clark et al.”
E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learning (2021, background): “Self-supervised pre-training has achieved great success in a wide range of natural language understanding (Devlin et al., 2018; Liu et al., 2019; Wang et al., 2019; Lan et al., 2019) and generation tasks (Song et al., 2019; Lewis et al., 2019; Bi et al., 2020).”