SpanBERT: Improving Pre-training by Representing and Predicting Spans

By Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke S. Zettlemoyer, Omer Levy. 2019

2,196 citations recorded by the source corpus.

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.1

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “4, which was loosely inspired by SpanBERT [Joshi et al., 2019].”
RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “However, some recent work has questioned the necessity of the NSP loss (Lample and Conneau, 2019; Yang et al., 2019; Joshi et al., 2019).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “While such salient span masking has not been shown to be impactful in previous work with standard BERT training (Joshi et al., 2019), it is crucial for REALM.”
BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension (2019, methodology): “Recent work has shown that very strong performance can be achieved by training for longer (Liu et al., 2019), by tying parameters across layers (Lan et al., 2019), and by masking spans instead of words (Joshi et al., 2019).”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “Pre-trained language models (PLMs), such as BERT (Devlin et al., 2018), XLNet (Yang et al., 2019), RoBERTa (Liu et al., 2019) and SpanBERT (Joshi et al., 2019), have achieved great success in many NLP tasks (e.g., the GLUE benchmark (Wang et al., 2018) and the challenging multi-hop reasoning task (D”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “In addition, we replace the next sentence prediction head with sentence order prediction as suggested by (Lan et al., 2019) and use whole word n-gram masking of (Joshi et al., 2019).”
StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding (2019, neutral): “0 SpanBERT [12] 64.”
Reasoning Like Program Executors (2022, methodology): “The language model baselines consist of BERT (Devlin et al., 2019), SpanBERT (Joshi et al., 2020) and Rea-sonBERT (Deng et al., 2021).”
Big Bird: Transformers for Longer Sequences (2020, methodology): “Most prominently, SpanBERT [42], ORQA [54], REALM [34], RAG [57] have achieved strong performance for different tasks.”
PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization (2019, background): “Transformer-based sequence models has led to a dramatic improvement in performance when finetuned for both natural language understanding and text generation tasks (Radford et al., 2018a; Devlin et al., 2019; Rothe et al., 2019; Yang et al., 2019; Joshi et al., 2019; Song et al., 2019; Dong et al., ”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “Contiguous masking was explored in Joshi et al. 2020 and masking mention spans speciﬁcally is similar to the salient span masking used in Guu et al. 2020.”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, methodology): “Among existing pre-training methods, BERT-like approaches, such as BERT (Devlin et al., 2018), RoBERTa (Liu et al., 2019) and ALBERT(Lan et al., 2019), are the most prominent by pre-training the bidirectional Transformer (Vaswani et al., 2017) encoder on a large text corpus through masked language…”
HuggingFace's Transformers: State-of-the-art Natural Language Processing (2019, neutral): “, 2019) up to co-reference resolution (Joshi et al., 2019) and commonsense inference (Bosselut et al.”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “The extracted phrases are represented by their start and end to-ken vectors from the ﬁnal layer of a transformer initialized from SpanBERT [Joshi et al., 2020].”
Learning Dense Representations of Phrases at Scale (2020, positive): “In our pilot experiments, we found that Span-BERT (Joshi et al., 2020) leads to superior performance compared to BERT (Devlin et al., 2019).”
ExpBERT: Representation Engineering with Natural Language Explanations (2020, methodology): “…(Table 2).1 On TACRED, which is the most well-established of our benchmarks and on which there is significant prior work, ExpBERT (which uses a smaller BERT-base model that is not fine-tuned on our task) outperforms the standard, fine-tuned BERT-large model by +1.5 F1 points (Joshi et al., 2019).”