Is ChatGPT a Good Recommender? A Preliminary Study

By Junling Liu, Chaoyong Liu, Renjie Lv, Kangdi Zhou, Y. Zhang. arXiv.org, 2023

409 citations recorded by the source corpus.

Recommendation systems have witnessed significant advancements and have been widely used over the past decades. However, most traditional recommendation methods are task-specific and therefore lack efficient generalization ability. Recently, the emergence of ChatGPT has significantly advanced NLP tasks by enhancing the capabilities of conversational models. Nonetheless, the application of ChatGPT in the recommendation domain has not been thoroughly investigated. In this paper, we employ ChatGPT as a general-purpose recommendation model to explore its potential for transferring extensive linguistic and world knowledge acquired from large-scale corpora to recommendation scenarios. Specifically, we design a set of prompts and evaluate ChatGPT's performance on five recommendation scenarios. Unlike traditional recommendation methods, we do not fine-tune ChatGPT during the entire evaluation process, relying only on the prompts themselves to convert recommendation tasks into natural language tasks. Further, we explore the use of few-shot prompting to inject interaction information that contains user potential interest to help ChatGPT better understand user needs and interests. Comprehensive experimental results on Amazon Beauty dataset show that ChatGPT has achieved promising results in certain tasks and is capable of reaching the baseline level in others. We conduct human evaluations on two explainability-oriented tasks to more accurately evaluate the quality of contents generated by different models. And the human evaluations show ChatGPT can truly understand the provided information and generate clearer and more reasonable results. We hope that our study can inspire researchers to further explore the potential of language models like ChatGPT to improve recommendation performance and contribute to the advancement of the recommendation systems field.

Subfields: Artificial Intelligence, Health Informatics

View Paper: https://www.semanticscholar.org/paper/ca7bd64d372e3bcb3f4633ca4a20291ff57de3c3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wayne Xin Zhao
2. Jiliang Tang
3. Yupeng Hou
4. Hongyu Lu
5. Ruobing Xie
6. Dawei Yin
7. Shirui Pan
8. Junjie Zhang

Citations in the corpus

• When large language models meet personalization: perspectives of challenges and opportunities (2023, neutral): “For example, Liu’s work [143] proposes a crowdsourcing task to assess the quality of generated explanations and review summaries, providing a way to evaluate”
• Recommender Systems in the Era of Large Language Models (LLMs) (2023, positive): “…of LLMs) and down-stream recommendation tasks, these conventional prompting methods have only shown limited applications in specific recommendation tasks that have similar nature to language generation tasks, such as the review summary of users [38] and the relation labeling between items [4].”
• Representation Learning with Large Language Models for Recommendation (2023, background): “LLMs have the potential to generate text answers that may include recommendations for non-existent items due to hallucination issues [20].”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Advanced LLMs like ChatGPT 2 and PaLM2 3 , with billions of parameters, exhibit great potential in many complex practical tasks, such as education [8], code generation [9] and recommendation [10].”
• Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach (2023, neutral): “We make two major technical contributions: (1) A formal introduction of instructions for recommendation .”
• Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, methodology): “In addition to the knowledge learned via pre-training, recent studies suggest that LLMs present pre-liminary success on tasks with implicit graph structures such as recommendation [35; 14], ranking [26], and multi-hop reasoning [7], in which LLMs are adopted to make the final predictions.”
• Large Language Models are Zero-Shot Rankers for Recommender Systems (2023, methodology): “We follow the experimental setting in Section 3.1 and replace the base LLM with LLaMA of different sizes (7B, 13B, 35B, and 65B) to investigate the effect of model scaling on the zero-shot recommendation task.”