Coreferential Reasoning Learning for Language Representation

By Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Maosong Sun, Zhiyuan Liu. 2020

191 citations recorded by the source corpus.

Language representation models such as BERT could effectively capture contextual semantic information from plain text, and have been proved to achieve promising results in lots of downstream NLP tasks with appropriate fine-tuning. However, existing language representation models seldom consider coreference explicitly, the relationship between noun phrases referring to the same entity, which is essential to a coherent understanding of the whole discourse. To address this issue, we present CorefBERT, a novel language representation model designed to capture the relations between noun phrases that co-refer to each other. According to the experimental results, compared with existing baseline models, the CorefBERT model has made significant progress on several downstream NLP tasks that require coreferential reasoning, while maintaining comparable performance to previous models on other common NLP tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning to Retrieve Passages without Supervision (2021, background): “coreference resolution and coreferential reasoning (Kocijan et al., 2019; Varkel and Globerson, 2020; Ye et al., 2020) and question answering (Ram et al.”
ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “ProoFVer outperforms current state-of-theart (SOTA) fact verification models (Liu et al., 2020; Ye et al., 2020; Zhong et al., 2020) in both performance and robustness.”
Few-Shot Question Answering by Pretraining Span Selection (2021, methodology): “Taking advantage of recurring words in a passage (restricted to nouns or named entities) was proposed in past work as a signal for coreference (Kocijan et al., 2019; Ye et al., 2020).”
Retrieval-Augmented Generation for Large Language Models: A Survey (2023, background): “The overarching optimization goal is to train the language model to accurately restore the obscured spans, thereby enriching its understanding of entity semantics [ Ye et al. , 2020 ] .”
Fine-grained Fact Verification with Kernel Graph Attention Network (2019, methodology): “CorefBERT (Ye et al., 2020) extends our KGAT architecture and explicitly models co-referring relationship in context for better performance.”
Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling (2020, methodology): “aph structures. They can be further divided into pipeline models (BERTTS (Wang et al.,2019b)), hierarchical models (HINBERT (Tang et al.,2020a)), and pre-training methods (CorefBERT and CorefRoBERTa (Ye et al., 2020)). We also include BERT baseline (Wang et al.,2019b) in our comparison. We ﬁnd that ”
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2020, methodology): “We choose the following baselines: (1) FastQA (Weis-senborn et al., 2017) and BiDAF (Seo et al., 2016), which are widely used question answering systems; (2) BERT , RoBERTa , CorefBERT , SpanBERT , MTB and CP , which are introduced in previous sections.”
Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts (2020, methodology): “We manually categorize the test instances by the reasoning type, which can be described as follows: • Coreferential Reasoning over Long Texts Coreferential reasoning (Ye et al. 2020) is a form of reasoning over multiple mentions.”
DialFact: A Benchmark for Fact-Checking in Dialogue (2021, methodology): “CorefBert-Colloquial (Ye et al., 2020) is one of the best performing models on FEVER and is designed to better capture and represent the corefer-”
A Multi-Level Attention Model for Evidence-Based Fact Checking (2021, result): “This allows us to fairly compare our model with a series of previous methods (Soleimani et al., 2019; Zhou et al., 2019; Liu et al., 2020; Ye et al., 2020) that also rely on Hanselowski et al. (2018)’s results.”
Pre-training of Mention Representations in Coreference Models (2020, methodology): “Ye et al. (2020) use an approach similar to Ko-cijan et al. (2019), but include a Language Modeling objective.”