Zero-Shot Information Extraction via Chatting with ChatGPT

By Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang and 2 more. arXiv.org, 2023

511 citations recorded by the source corpus.

Zero-shot information extraction (IE) aims to build IE systems from the unannotated text. It is challenging due to involving little human intervention. Challenging but worthwhile, zero-shot IE reduces the time and effort that data labeling takes. Recent efforts on large language models (LLMs, e.g., GPT-3, ChatGPT) show promising performance on zero-shot settings, thus inspiring us to explore prompt-based methods. In this work, we ask whether strong IE models can be constructed by directly prompting LLMs. Specifically, we transform the zero-shot IE task into a multi-turn question-answering problem with a two-stage framework (ChatIE). With the power of ChatGPT, we extensively evaluate our framework on three IE tasks: entity-relation triple extract, named entity recognition, and event extraction. Empirical results on six datasets across two languages show that ChatIE achieves impressive performance and even surpasses some full-shot models on several datasets (e.g., NYT11-HRL). We believe that our work could shed light on building IE models with limited resources.

Subfields: Artificial Intelligence, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/f4cba0db34aa0c389cec267ca1f3ba5255ea2645

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Diyi Yang
2. Jiaao Chen
3. Jie Zhou
4. Zhixu Li
5. Omar Shaikh
6. Caleb Ziems
7. Viet Dac Lai
8. Siyuan Ma

In Management Science and Operations Research

1. Siyuan Ma
2. Trung Bui
3. Thien Huu Nguyen
4. Diyi Yang
5. Jinan Xu
6. Franck Dernoncourt
7. Fandong Meng
8. Zihao Wu

Citations in the corpus

• Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, unknown): “[51] proposed ChatIE, a ChatGPTbased multi-round question-and-answer framework for information extraction.”
• Can Large Language Models Transform Computational Social Science? (2023, methodology): “We can also achieve additional performance gains by prompting models iteratively across multiple rounds of refinement as in Wei et al. (2023).”
• ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning (2023, background): “, 2023), information extraction (Wei et al., 2023; Gao et al., 2023), text classification (Kuzman et al.”
• Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, methodology): “[97] proposed ChatIE, a ChatGPT-based multi-round question-and-answer framework for information extraction.”
• Instruction Tuning for Large Language Models: A Survey (2023, unknown): “One of the major issues with LLMs is the mismatch between the training objective and users’ objective: LLMs are typically trained on minimizing the contextual word prediction error on large corpora; while users want the model to “follow their instructions helpfully and safely” [13, 39, 93, 94, 111].”
• Is ChatGPT a Good NLG Evaluator? A Preliminary Study (2023, background): “Wei et al. (2023) prove that ChatGPT can achieve good performance and even surpasses some full-shot models on several datasets through a multi-turn question-answering manner.”
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023, background): “There are several preprints that evaluate ChatGPT on CoNLL03 (Wei et al., 2023; Li et al., 2023a; Han et al., 2023) and at least one conference paper published on ACL 2023 that evaluates GPT-3 (Brown et al., 2020) and Codex (Chen et al., 2021) on the same benchmark (Li et al., 2023b).”