Prompt-Learning for Fine-Grained Entity Typing

By Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juan-Zi Li, Hong-Gee Kim. 2021

176 citations recorded by the source corpus.

As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{cloze}-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We first develop a simple and effective prompt-learning pipeline by constructing entity-oriented verbalizers and templates and conducting masked language modeling. Further, to tackle the zero-shot regime, we propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types. Extensive experiments on three fine-grained entity typing benchmarks (with up to 86 classes) under fully supervised, few-shot and zero-shot settings show that prompt-learning methods significantly outperform fine-tuning baselines, especially when the training data is insufficient.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Jie Tang
4. Jie Zhou
5. Yankai Lin
6. Shirui Pan
7. Peng Li
8. Xiao Liu

In Computer Vision and Pattern Recognition

1. Zhiyuan Liu
2. Maosong Sun
3. Jie Zhou
4. Peng Li
5. Yankai Lin
6. Dacheng Tao
7. Jie Tang
8. Shafiq R. Joty

Citations in the corpus

• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5 (2021, background): “…al., 2020; Tam et al., 2021) and continuous or soft prompts (Li & Liang, 2021; Liu et al., 2021c; Lester et al., 2021), ( ii ) applications of PL (Han et al., 2021; Ben-David et al., 2021; Ding et al., 2021), and analysis of prompt-based learning (Liu et al., 2021a; Le Scao & Rush, 2021; Zhong…”
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, background): “Prompt tuning and P-tuning have been proved quite effective in many NLU tasks, including knowledge probing, sentimental analysis (Li et al., 2021), relation extraction (Han et al., 2021), entity typing (Ding et al., 2021) and so on.”
• Exploring Low-dimensional Intrinsic Task Subspace via Prompt Tuning (2021, background): “…tasks and adding some prompt tokens into the input, including human-designed explainable prompts (Brown et al., 2020; Schick and Schütze, 2021a,b; Han et al., 2021b; Ding et al., 2021; Hu et al., 2021) and automatically searched prompts (Jiang et al., 2020; Shin et al., 2020; Gao et al., 2021).”
• No More Manual Tests? Evaluating and Improving ChatGPT for Unit Test Generation (2023, background): “To date, LLMs have been applied to various domains and achieved a great success [39], [40], [41], [42], [43], [44], [45], [46], [47].”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “CR-RL [158] proposes an actor-critic deep reinforcement learning-based coreference resolver for cross-document CR.”
• OpenPrompt: An Open-source Framework for Prompt-learning (2021, background): “For example, with appropriate template, zero-shot prompt-learning could even outperform 32-shot fine-tuning (Ding et al., 2021a).”
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners (2021, background): “However, GPT-3 is not designed for fine-tuning; it mainly relies on the handcraft prompt (in-context learning [24, 51, 11, 29]).”
• Is GPT-3 a Good Data Annotator? (2022, background): “For instance, with an appropriate template, zero-shot prompt-learning can even outperform 32-shot fine-tuning (Ding et al., 2021a).”
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction (2021, methodology): “(Ding et al. 2021) apply prompt-tuning to entity typing with prompt-learning by constructing entity-oriented verbalizer and templates.”