Large Language Models are Zero-Shot Rankers for Recommender Systems

By Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, Wayne Xin Zhao. European Conference on Information Retrieval, 2023

576 citations recorded by the source corpus.

Recently, large language models (LLMs) (e.g., GPT-4) have demonstrated impressive general-purpose task-solving abilities, including the potential to approach recommendation tasks. Along this line of research, this work aims to investigate the capacity of LLMs that act as the ranking model for recommender systems. We first formalize the recommendation problem as a conditional ranking task, considering sequential interaction histories as conditions and the items retrieved by other candidate generation models as candidates. To solve the ranking task by LLMs, we carefully design the prompting template and conduct extensive experiments on two widely-used datasets. We show that LLMs have promising zero-shot ranking abilities but (1) struggle to perceive the order of historical interactions, and (2) can be biased by popularity or item positions in the prompts. We demonstrate that these issues can be alleviated using specially designed prompting and bootstrapping strategies. Equipped with these insights, zero-shot LLMs can even challenge conventional recommendation models when ranking candidates are retrieved by multiple candidate generators. The code and processed datasets are available at https://github.com/RUCAIBox/LLMRank.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Systems

View Paper: https://www.semanticscholar.org/paper/f4e723958a93762befb4d4a039b44a7d752f9917

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Information Systems

Citations in the corpus

Adapting Large Language Models by Integrating Collaborative Semantics for Recommendation (2023, background): “The first approach [18]–[21] verbalizes the user behaviors into text sequences ( e.g., concatenating the titles and category labels of the interacted items), and designs special prompts to instruct LLMs for fulfilling the recommendation tasks.”
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, methodology): “Supervised fine-tuning in Large Language Models Large language models (LLMs) undergo the SFT stage to further unlock the performance in task solving and aligning human instruction.”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, background): “, 2022) have demonstrated impressive performance on a wide range of natural language tasks, achieving comparable or better performance when compared with their supervised counterparts that are potentially trained with millions of labeled examples, even in the zero-shot setting (Kojima et al., 2022; ”
Representation Learning with Large Language Models for Recommendation (2023, background): “In recent times, there have been several endeavors to leverage diverse data modalities in order to enhance traditional ID-based recommenders [9, 17, 46].”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “Besides, LLM-as-a-judge has also been explored in many traditional retrieval applications for relevance assessment, such as search (Thomas et al., 2024), retrieval (Ma et al., 2024) and recommendation (Hou et al., 2024).”
When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “In particular, for ranking tasks that involve providing a list of items in order of preference, a candidate generation module is employed to narrow down the pool of items [145] and [146].”
Recommender Systems in the Era of Large Language Models (LLMs) (2023, unknown): “ChatGPT [4] In-context Learning (ICL) Top-K Recommendation ChatGPT [3], [38], [82], [83] [84], [85], [86], [”
Recommendation as Instruction Following: A Large Language Model Empowered Recommendation Approach (2023, unknown): “Despite the similar efforts in prior studies [2, 13], we discuss the expression of user needs in the context of LLMs, and treat recommendation as a specific task approached by instruction following.”