Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting

By Zhen Qin, R. Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu and 3 more. NAACL-HLT, 2023

461 citations recorded by the source corpus.

Ranking documents using Large Language Models (LLMs) by directly feeding the query and candidate documents into the prompt is an interesting and practical problem. However, researchers have found it difficult to outperform fine-tuned baseline rankers on benchmark datasets. We analyze pointwise and listwise ranking prompts used by existing methods and argue that off-the-shelf LLMs do not fully understand these challenging ranking formulations. In this paper, we propose to significantly reduce the burden on LLMs by using a new technique called Pairwise Ranking Prompting (PRP). Our results are the first in the literature to achieve state-of-the-art ranking performance on standard benchmarks using moderate-sized open-sourced LLMs. On TREC-DL 2019&2020, PRP based on the Flan-UL2 model with 20B parameters performs favorably with the previous best approach in the literature, which is based on the blackbox commercial GPT-4 that has 50x (estimated) model size, while outperforming other LLM-based solutions, such as InstructGPT which has 175B parameters, by over 10% for all ranking metrics. By using the same prompt template on seven BEIR tasks, PRP outperforms supervised baselines and outperforms the blackbox commercial ChatGPT solution by 4.2% and pointwise LLM-based solutions by more than 10% on average NDCG@10. Furthermore, we propose several variants of PRP to improve efficiency and show that it is possible to achieve competitive results even with linear complexity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/2d3bc530d8f1ed36932a70bc362ea94d988adec9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei Xiong
2. Hanze Dong
3. Hui Xiong
4. Hengshu Zhu
5. Ying-Bo Zhou
6. Hao Wang
7. Han Zhao
8. Enhong Chen

In Computer Vision and Pattern Recognition

1. Hui Xiong
2. Hengshu Zhu
3. Hao Wang
4. Enhong Chen
5. Hanze Dong
6. Doyen Sahoo
7. Qi Liu

Citations in the corpus

• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, unknown): “Finally, Qin et al. (2024) critique the limitations of pointwise and listwise ranking prompts in existing methods, noting that typical LLMs often lack the depth to grasp complex ranking tasks.”
• A survey on large language models for recommendation (2023, background): “PRP (Pairwise Ranking Prompting) [ Qin et al. , 2023 ] proposes pairwise ranking for listwise tasks with LLM, which enumerates all pairs and performs a global aggregation to generate a score for each item.”
• ChatDev: Communicative Agents for Software Development (2023, unknown): “…knowledge (Brown et al., 2020; Bubeck et al., 2023; Vaswani et al., 2017; Radford et al.; Touvron et al., 2023; Wei et al., 2022a; Shanahan et al., 2023; Chen et al., 2021; Brants et al., 2007; Chen et al., 2021; Ouyang et al., 2022; Yang et al., 2023a; Qin et al., 2023b; Kaplan et al., 2020).”
• Large Language Models for Information Retrieval: A Survey (2023, unknown): “We choose representative methods [146, 148, 152, 155] in pointwise, listwise and pairwise ranking, and include several supervised methods [13, 140, 143] mentioned in Section 5.1 for performance comparison.”
• CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Reranking (2024, unknown): “Recently, listwise reranking approaches (Pradeep et al., 2023; Reddy et al., 2024) have gained popularity for their ability to score multiple passages simultaneously, as opposed to pointwise Zhuang et al. (2023a;c) or pairwise Qin et al. (2023) reranking, where scoring is performed in isolation.”
• SemRAG: Semantic Knowledge-Augmented RAG for Improved Question-Answering (2025, unknown): “The retrieved chunks are then ranked, often using re-ranking techniques to ensure relevance and reduce noise [14, 15].”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “Recent advances in multimodal retrieval (Ma et al., 2024a; Wei et al., 2025) and structured data processing (Li et al., 2023b, 2024a) show promising directions.”
• RLHF Workflow: From Reward Modeling to Online RLHF (2024, background): “We mention in passing that it is possible to include detailed rubrics in the data format to further improve the preference dataset, which we leave for future work (Qin et al., 2023).”
• LLM-confidence reranker: A training-free approach for enhancing retrieval-augmented generation systems (2026, unknown): “• YesNo (Qin et al., 2024): A point-wise reranking technique that prompts pre-trained language models to generate “yes” or “no” indicating document relevance to the query, then ranks based on normalized “yes” likelihood, without fine-tuning.”