End-to-end Neural Coreference Resolution

By Kenton C. T. Lee, Luheng He, Michael Lewis, Luke S. Zettlemoyer. 2017

966 citations recorded by the source corpus.

We introduce the first end-to-end coreference resolution model and show that it significantly outperforms all previous work without using a syntactic parser or hand-engineered mention detector. The key idea is to directly consider all spans in a document as potential mentions and learn distributions over possible antecedents for each. The model computes span embeddings that combine context-dependent boundary representations with a head-finding attention mechanism. It is trained to maximize the marginal likelihood of gold antecedent spans from coreference clusters and is factored to enable aggressive pruning of potential mentions. Experiments demonstrate state-of-the-art performance, with a gain of 1.5 F1 on the OntoNotes benchmark and by 3.1 F1 using a 5-model ensemble, despite the fact that this is the first approach to be successfully trained with no external resources.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “As an upper bound on performance, the state-of-the-art coreference model from Lee et al. (2017)9 finds an antecedent span with the head word 64% of the time.”
Deep Contextualized Word Representations (2018, methodology): “Our baseline coreference model is the end-to-end neural model from Lee et al. (2017) with all hy-Model”
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems (2019, methodology): “Given the BERT representation for each word in the original sentence, we get span representations of the pronoun and noun phrase via a self-attention span-pooling operator (Lee et al., 2017), before feeding it into a logistic regression classiﬁer.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, methodology): “We then ran the state-of-the-art named entity recognition model (Peters et al., 2018) and the end-to-end coreference resolution model (Lee et al., 2017) provided by AllenNLP (Gardner et al., 2018) to annotate the crawled articles.”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, methodology): “Span selection models (Lee et al., 2016, 2017; He et al., 2018) typically create a ﬁxed-length representation of a span using its boundary tokens (start and end).”
Knowledge Enhanced Contextual Word Representations (2019, positive): “4.1) with a linear projection, (1) Then, the KAR computes C mention-span representations s m ∈ R E , one for each candidate mention, by pooling over all word pieces in a mention-span using the self-attentive span pooling from Lee et al. (2017).”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, neutral): “In fact, this disparity is 30% higher than a similar result based only on GloVe embeddings (Lee et al., 2017).”
Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction (2022, methodology): “Similar to Lee et al. (2017), we can obtain the representation of each pair by concatenating the two quantity representations and the element-wise product between them.”
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods (2018, methodology): “In this section we evaluate of three representative systems: rule based, Rule , (Raghunathan et al., 2010), feature-rich, Feature , (Durrett and Klein, 2013), and end-to-end neural (the current state-of-the-art), E2E , (Lee et al., 2017).”
AllenNLP: A Deep Semantic Natural Language Processing Platform (2018, background): “…state-of-the-art for a wide range of tasks such as text classiﬁ-cation (Howard and Ruder, 2018), machine translation (Vaswani et al., 2017), semantic role labeling (Zhou and Xu, 2015; He et al., 2017), coreference resolution (Lee et al., 2017a), and semantic parsing (Krishnamurthy et al., 2017).”
Learning Gender-Neutral Word Embeddings (2018, methodology): “We train the end-to-end coreference resolution model (Lee et al., 2017) with different word em-”
Gender Bias in Contextualized Word Embeddings (2019, neutral): “Figure 1 shows there are two principal components for gender in ELMo, in contrast to GloVe which only has one (Bolukbasi et al., 2016).”
What do you learn from context? Probing for sentence structure in contextualized word representations (2019, methodology): “We take a list of contextual vectors [ e 0 , e 1 , . . . , e n ] and integer spans s (1) = [ i (1) , j (1) ) and (optionally) ) as inputs, and use a projection layer followed by the self-attention pooling operator of Lee et al. (2017) to compute ﬁxed-length span representations.”
Gender Bias in Neural Natural Language Processing (2018, methodology): “NCR Model I The model of Lee et al. [2017] uses pretrained word embeddings, thus all features and mention representations are learned from these pretrained embeddings.”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, methodology): “When applied to a neural network based coreference resolution model (Lee et al., 2017, 2018) originally trained on OntoNotes 5.0 which was tested on WinoBias, gender augmentation lowered the difference be-tween F 1 scores on pro-stereotypical and anti-stereotypical test sets signiﬁcantly, which…”
Deep RNNs Encode Soft Hierarchical Syntax (2018, background): “parts of speech, dependencies) in state-of-the-art NLP models (He et al., 2017; Lee et al., 2017; Klein et al., 2017).”
Higher-Order Coreference Resolution with Coarse-to-Fine Inference (2018, methodology): “Recent coreference resolution systems have heavily relied on ﬁrst order models (Clark and Manning, 2016a; Lee et al., 2017), where only pairs of entity mentions are scored by the model.”
Information-Theoretic Probing with Minimum Description Length (2020, methodology): “It takes a list of contextual vectors [ e 0 , e 1 , . . . , e n ] and integer spans s 1 = [ i 1 , j 1 ) and (optionally) s 2 = [ i 2 , j 2 ) as inputs, and uses a projection layer followed by the self-attention pooling operator of Lee et al. (2017) to compute ﬁxed-length span representations.”