BERT for Coreference Resolution: Baselines and Analysis

By Mandar Joshi, Omer Levy, Daniel S. Weld, Luke S. Zettlemoyer. 2019

351 citations recorded by the source corpus.

We apply BERT to coreference resolution, achieving a new state of the art on the GAP (+11.5 F1) and OntoNotes (+3.9 F1) benchmarks. A qualitative analysis of model predictions indicates that, compared to ELMo and BERT-base, BERT-large is particularly better at distinguishing between related but distinct entities (e.g., President and CEO), but that there is still room for improvement in modeling document-level context, conversations, and mention paraphrasing. We will release all code and trained models upon publication.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “We use the independent version of the Joshi et al. (2019b) implementation of the higher-order coref-4 erence model (Lee et al., 2018).”
Longformer: The Long-Document Transformer (2020, methodology): “Coreference Resolution We use OntoNotes (Pradhan et al., 2012), and the model from Joshi et al. (2019), a modiﬁcation of the system from Lee et al. (2018) to replace ELMo with BERT.”
ETC: Encoding Long and Structured Inputs in Transformers (2020, background): “While 512 might be enough for some tasks (e.g., co-reference resolution seems to beneﬁt from even smaller input lengths (Joshi et al., 2019)), this is problematic in others.”
Blockwise Self-Attention for Long Document Understanding (2019, methodology): “, 2019), coreference resolution (Joshi et al., 2019b), relation extraction (Soares et al.”
Picking BERT’s Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis (2020, methodology): “(2019) through analysis of BERT’s attention heads—may explain BERT’s strong performance on coreference resolution, a task that relies on identifying pronoun-antecedent relationships (Joshi et al., 2019).”
Coreferential Reasoning Learning for Language Representation (2020, methodology): “Similar to previous language representation models (Devlin et al., 2019; Yang et al., 2019; Joshi et al., 2019a; Liu et al., 2019a), we adopt English Wikipeida 4 as our training corpus, which contains about 3,000M tokens.”
Paired Representation Learning for Event and Entity Coreference (2020, methodology): “The whole trigger-based event pair ( i , j ) is denoted by entity span (Joshi et al., 2019, 2020).”
Recognition of They/Them as Singular Personal Pronouns in Coreference Resolution (2022, methodology): “End-to-End (Lee et al., 2018), BERT (Joshi et al., 2019), and Span-BERT (Joshi et al., 2020) are released under Apache License 2.0.”
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021, methodology): “For the coreference resolution model on OntoNotes 5.0, we adapt existing code implementation 4 (Joshi et al., 2019) to support loading RoBERTa-base as the base model.”
Attention mechanism in neural networks: where it comes and where it goes (2022, methodology): “Also, BERT model has been considered for different tasks such as coreference resolution [146].”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, methodology): “SpanBERT introduces contextual span representation to the the e2e-coreference model (Lee et al., 2018; Joshi et al., 2019) to achieve state-of-the-art results on the English portion of the popular CoNLL-2012 shared task coreference benchmark (Pradhan et al., 2012).”
CorefQA: Coreference Resolution as Query-based Span Prediction (2019, methodology): “This veriﬁes the importance of span-level pre-training for coreference resolution and is consistent with previous ﬁndings (Joshi et al., 2019a).”
Pre-training of Mention Representations in Coreference Models (2020, methodology): “As a baseline model, we use the work of Joshi et al. (2019b), which builds on top of Lee et al. (2018) and uses SpanBERT (Joshi et al., 2019a).”
CIDER: Commonsense Inference for Dialogue Explanation and Reasoning (2021, background): “Explicit triplets can be relatively easily parsed out using semantic parsing (Speer et al., 2017) and simple co-reference resolution (Joshi et al., 2019).”
Unsupervised Distillation of Syntactic Information from Contextualized Word Representations (2020, background): “…RoBERTA (Liu et al., 2019b), were shown to produce representations that excel in recovering both structure-related information (Gulordava et al., 2018; van Schijndel and Linzen, 2018; Wilcox et al., 2018; Goldberg, 2019) as well as in semantic information (Yang et al., 2019; Joshi et al., 2019).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “[77, 78] adds the substantial e2e-coref algorithm [91], which transforms ratings of pairs of spans produced by the LM into valid mention clusters.”
The NLP Task Effectiveness of Long-Range Transformers (2022, methodology): “In overall, we have similar observations as Joshi et al. (2019) that overlapped segments do not offer improvements on the performance.”