ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning

By Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, Jie Zhou. 2020

143 citations recorded by the source corpus.

Pre-trained Language Models (PLMs) have shown superior performance on various downstream Natural Language Processing (NLP) tasks. However, conventional pre-training objectives do not explicitly model relational facts in text, which are crucial for textual understanding. To address this issue, we propose a novel contrastive learning framework ERICA to obtain a deep understanding of the entities and their relations in text. Specifically, we define two novel pre-training tasks to better understand entities and relations: (1) the entity discrimination task to distinguish which tail entity can be inferred by the given head entity and relation; (2) the relation discrimination task to distinguish whether two relations are close or not semantically, which involves complex relational reasoning. Experimental results demonstrate that ERICA can improve typical PLMs (BERT and RoBERTa) on several language understanding tasks, including relation extraction, entity typing and question answering, especially under low-resource settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Huang
2. Shumin Deng
3. Yaojie Lu
4. Luo Si
5. Xinyan Xiao
6. Le Sun
7. Chuanqi Tan
8. Hongyu Lin

In Computer Vision and Pattern Recognition

1. Fei Huang
2. Shumin Deng
3. Yaojie Lu
4. Luo Si
5. Xinyan Xiao
6. Le Sun
7. Chuanqi Tan
8. Hongyu Lin

Citations in the corpus

• A Review on Language Models as Knowledge Bases (2022, unknown): “…et al., 2020; Verga et al., 2020; Sun et al., 2021; de Jong et al., 2021), objectives focusing on semantic understanding of relations and entities (Qin et al., 2021), or knowledge embedding objective where entities and relations are encoded in KGs as distributed representations KE (Wang et al.,…”
• Unified Structure Generation for Universal Information Extraction (2022, background): “And several task-specific pre-training techniques are proposed on these architectures (Mengge et al., 2020; Wang et al., 2021b; Qin et al., 2021).”
• AdaPrompt: Adaptive Prompt-based Finetuning for Relation Extraction (2021, methodology): “Together with inspired by ERICA (Qin et al., 2020), to maintain the ability of language understanding inherited from PLMs and avoid catastrophic forgetting , AdaPrompt-tuning adds Entity Discrimination objective (ED) to train PLMs for understanding an entity via considering its relations with other…”
• Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “This approach, which proved highly successful for relation classification, has been applied to the same task in languages beyond English (Köksal and Özgür, 2020; Ananthram et al., 2020), and as an additional LM pretraining objective (Qin et al., 2021).”