Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent

By Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Shuaiqiang Wang, Dawei Yin, Zhumin Chen, Zhaochun Ren, Da-Wei Yin. 2023

576 citations recorded by the source corpus.

Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks, including search engines. However, existing work utilizes the generative ability of LLMs for Information Retrieval (IR) rather than direct passage ranking. The discrepancy between the pre-training objectives of LLMs and the ranking objective poses another challenge. In this paper, we first investigate generative LLMs such as ChatGPT and GPT-4 for relevance ranking in IR. Surprisingly, our experiments reveal that properly instructed LLMs can deliver competitive, even superior results to state-of-the-art supervised methods on popular IR benchmarks. Furthermore, to address concerns about data contamination of LLMs, we collect a new test set called NovelEval, based on the latest knowledge and aiming to verify the model's ability to rank unknown knowledge. Finally, to improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models using a permutation distillation scheme. Our evaluation results turn out that a distilled 440M model outperforms a 3B supervised model on the BEIR benchmark. The code to reproduce our results is available at www.github.com/sunnweiwei/RankGPT.

Subfields: Artificial Intelligence, Information Systems, Information Retrieval & Search, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Information Retrieval & Search

In Large Language Models

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, background): “A surge of recent publications have attempted to both locally imitate proprietary models for specific tasks (Sun et al., 2023; Hsieh et al., 2023; Honovich et al., 2022) and broadly imitate models, e.g., Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), GPT4ALL…”
Fine-Tuning LLaMA for Multi-Stage Text Retrieval (2023, methodology): “Compared to RankGPT 4 [32], which prompts GPT-4 to perform passage reranking through permutation generation within a multi-stage retrieval pipeline, our repLLaMA– rankLLaMA combination achieves higher nDCG@10 scores on both DL19 and DL20.”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “Many works have demonstrated that LLMs could be directly used as recommenders without fine-tuning (Wang et al., 2023i; Dai et al., 2023c) or generate auxiliary textual features to benefit recommender systems (Xi et al., 2023; Ren et al., 2023; Wei et al., 2024).”
Large Language Models for Information Retrieval: A Survey (2023, unknown): “Recently, listwise methods [130, 131] that directly rank a list of documents have been proposed as a solution to this problem.”
Representation Learning with Large Language Models for Recommendation (2023, background): “In recent times, there have been several endeavors to leverage diverse data modalities in order to enhance traditional ID-based recommenders [9, 17, 46].”
A Survey on Evaluation of Large Language Models (2023, background): “, 2023) X (Sun et al., 2023) X (Lanzi and Loiacono, 2023) Game design (Zhang et al.”
Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting (2023, methodology): “• RankGPT (Sun et al., 2023): The listwise prompting based approach using various GPT based LLMs.”
Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Tasks (2023, methodology): “…professional exams (Kung et al., 2023; Nori et al., 2023; Terwi-esch, 2023; i.a. ), exceed state-of-the-art methods on many traditional benchmarks (Sun et al., 2023; Sobania et al., 2023; Zhang et al., 2023a; Dhingra et al., 2023; i.a. ), and surpass human performance on tasks that require…”
CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “Furthermore, Sun et al. (2023) show that instruction-tuned LLMs can outperform traditional supervised cross-encoders (Nogueira et al., 2020; Zhuang et al., 2023b) in zero-shot reranking settings.”
What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models (2025, unknown): “Listwise approaches prompt the model with multiple candidate passages simultaneously to estimate a ranked order [53, 54].”
BRIDGE: Multimodal-to-Text Retrieval via Reinforcement-Learned Query Alignment (2026, unknown): “Existing approaches to this problem focus on the re-triever side: training larger vision-language encoders [12, 15, 45], fine-tuning on multimodal contrastive data [44], or applying LLM-based reranking [3, 4, 26, 39].”
MARVEL: Multimodal Adaptive Reasoning-intensiVe Expand-rerank and retrievaL (2026, unknown): “LLM-based rerankers [2, 3, 23, 31, 34] reason over candidates but depend entirely on the quality of the upstream retriever.”