Deep Reinforcement Learning for Mention-Ranking Coreference Models

By Kevin Clark, Christopher D. Manning. 2016

378 citations recorded by the source corpus.

Coreference resolution systems are typically trained with heuristic loss functions that require careful tuning. In this paper we instead apply reinforcement learning to directly optimize a neural mention-ranking model for coreference evaluation metrics. We experiment with two approaches: the REINFORCE policy gradient algorithm and a reward-rescaled max-margin objective. We find the latter to be more effective, resulting in significant improvements over the current state-of-the-art on the English and Chinese portions of the CoNLL 2012 Shared Task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Gender Bias in Coreference Resolution (2018, methodology): “Neural The move to deep neural models led to more powerful antecedent scoring functions, and the subsequent learned feature combinations resulted in new state-of-the-art performance (Wise-man et al., 2015; Clark and Manning, 2016b).”
Toward Gender-Inclusive Coreference Resolution (2019, background): “…resolution systems on our ablated data: the AI2 system [AI2; Gardner et al. 2017], hugging face, which is a neural system based on spacy, and the Stanford deterministic [SfdD; Raghunathan et al. 2010], statistical [SfdS; Clark and Manning 2015] and neural [SfdN; Clark and Manning 2016] systems.”
Coarse-to-Fine Question Answering for Long Documents (2016, background): “Reinforcement learning recently gained popularity in tasks such as co-reference resolution (Clark and Manning, 2016), information extraction (Narasimhan et al., 2016), semantic parsing (Andreas et al., 2016) and textual games (Narasimhan et al., 2015; He et al., 2016).”
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods (2018, background): “, 2015a), and neuralnetwork based (Clark and Manning, 2016; Lee et al., 2017) have been proposed.”
Gender Bias in Neural Natural Language Processing (2018, methodology): “Neural coreference resolution systems typically employ a mention-ranking model [Clark and Manning, 2016a] in which a feed-forward neural network produces a coreference score assigning to every pair of mentions an indicator of their coreference likelihood.”
End-to-end Neural Coreference Resolution (2017, methodology): “, 2012; Björkelund and Kuhn, 2014; Martschat and Strube, 2015), or (4) mention-ranking models (Durrett and Klein, 2013; Wiseman et al., 2015; Clark and Manning, 2016a).”
A Simple, Fast Diverse Decoding Algorithm for Neural Generation (2016, methodology): “We ﬁrst go over the vanilla beam search method and then detail the proposed algorithm which fosters diversity during decoding.”
Higher-Order Coreference Resolution with Coarse-to-Fine Inference (2018, background): “, 2017) that our proposed model builds upon, there is a large body of literature on coreference resolvers that fundamentally rely on scoring span pairs (Ng and Cardie, 2002; Bengtson and Roth, 2008; Denis and Baldridge, 2008; Fernandes et al., 2012; Durrett and Klein, 2013; Wiseman et al., 2015; Cla”
Mapping Instructions and Visual Observations to Actions with Reinforcement Learning (2017, methodology): “, 2016), co-reference resolution (Clark and Manning, 2016), and dialog (Li et al.”
From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood (2017, background): “, 2015), coreference resolution (Clark and Manning, 2016), machine translation (Norouzi et al.”
Dynamic Entity Representations in Neural Language Models (2017, background): “We suspect that a stronger baseline system (such as that of Clark and Manning, 2016a) could give greater improvements, if it can be adapted to provide k -best lists.”
CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialog (2019, background): “Coreference Resolution is a well studied problem in the NLP community (Ng, 2010; Lee et al., 2017; Wiseman et al., 2016; Clark and Manning, 2016a,b).”
Unsupervised Deep Structured Semantic Models for Commonsense Reasoning (2019, methodology): “Finally, we also compare to the pre-trained Coreference Resolution Tool (Clark and Manning, 2016a,b) 12 , and we can see that it doesn’t adapt to our commonsense reasoning tasks and can’t tell the difference between each pair of sentences from WSC.”
Learning Dynamic Context Augmentation for Global Entity Linking (2019, background): “Agent: The Agent is a candidate ranking model that has a similar architecture to (Clark and Manning, 2016), aiming to output the action preference Hθ(St−1, S′ t−1, A j t ) of each linking action Ajt = (mt → e j t ).”
Automated Storytelling via Causal, Commonsense Plot Ordering (2020, methodology): “First, we extract all the coreference clusters using a pre-trained neural coreference resolution model (Clark and Manning 2016).”
Visual Reference Resolution using Attention Memory for Visual Dialog (2017, methodology): “…attention α mem t and key k mem t are then retrieved from the attention memory using the computed addressing vector β t by This relevant attention retrieval allows the proposed model to resolve the visual reference by indirectly resolving coreferences [34–36] through the memory addressing process.”
Look Again at the Syntax: Relational Graph Convolutional Network for Gendered Ambiguous Pronoun Resolution (2019, methodology): “For the ﬁnal submission for stage 2 of the shared task, we averaged our result with a BERT-score-layer (Zhang et al., 2018; Clark and Manning, 2016) result.”