Fine-Tuning LLaMA for Multi-Stage Text Retrieval

By Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, Jimmy J. Lin. Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023

431 citations recorded by the source corpus.

While large language models (LLMs) have shown impressive NLP capabilities, existing IR applications mainly focus on prompting LLMs to generate query expansions or generating permutations for listwise reranking. In this study, we leverage LLMs directly to serve as components in the widely used multi-stage text ranking pipeline. Specifically, we fine-tune the open-source LLaMA-2 model as a dense retriever (repLLaMA) and a pointwise reranker (rankLLaMA). This is performed for both passage and document retrieval tasks using the MS MARCO training data. Our study shows that finetuned LLM retrieval models outperform smaller models. They are more effective and exhibit greater generalizability, requiring only a straightforward training strategy. Moreover, our pipeline allows for the fine-tuning of LLMs at each stage of a multi-stage retrieval pipeline. This demonstrates the strong potential for optimizing LLMs to enhance a variety of retrieval tasks. Furthermore, as LLMs are naturally pre-trained with longer contexts, they can directly represent longer documents. This eliminates the need for heuristic segmenting and pooling strategies to rank long documents. On the MS MARCO and BEIR datasets, our repLLaMA-rankLLaMA pipeline demonstrates a high level of effectiveness.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a531e9a328ad1488567fa68c15d5bf30bfb90c78

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Furu Wei
2. Hong-Liu Cao
3. Siva Reddy
4. Nan Yang
5. Mohit Bansal
6. Jianfeng Gao
7. Percy Liang
8. Luke S. Zettlemoyer

Citations in the corpus

• LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, methodology): “In contrast, decoder-only models are much more sample-efficient and do not require large-scale contrastive pre-training (Wang et al., 2023).”
• Improving Text Embeddings with Large Language Models (2023, methodology): “RepLLaMA [23] proposes to fine-tune LLaMA-2 with bi-encoder architecture for ad-hoc retrieval.”
• Large Language Models for Information Retrieval: A Survey (2023, neutral): “Enc-only [140] Enc-dec [13], [141], [142], [143] Dec-only [131], [144], [145] Pointwise [146], [147], [148], [149], [150], [151] Listwise [152], [153], [154] Pairwise [155], [156] Data Augmentation - [157], [158], [159], [160], [161], [162] with lower inference time demands), the potential mismatch…”
• Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (2026, unknown): “The negatives combine hard negatives, which are non-relevant documents paired with Q in the training collection, and in-batch negatives, which are documents associated with the other queries in the same batch (Karpukhin et al., 2020; Ma et al., 2024).”
• Tevatron Meets Megatron: Expert-Parallel LLM Reranker Training on an Academic Budget (2026, unknown): “We use three first-stage retrievers chosen to span the lexical / dense / learned-sparse spectrum and a range of first-stage strength: BM25 (Robertson et al., 1995), E5-base-v2 (Wang et al., 2024), and SPLADE-v3 (Lassance et al., 2024), with BEIR-15 NDCG@10 of 0.415, 0.503, and 0.504 respectively.”
• A Systematic Multi-Domain Evaluation of Document Retrievers (2026, unknown): “Retrievers. docT5query [58] expands documents by generating multiple queries for each and retrieving with BM25. query2doc [80] instead expands the query, generating a pseudo-document and again retrieving with BM25, while HyDE [19] generates a hypothetical document and retrieves with the Contriever.”
• Improving Long-Context Retrieval with Multi-Prefix Embedding (2026, unknown): “DPR uses BERT encoders with [CLS] pooling (Karpukhin et al., 2020), while recent decoder-only embedding models such as RepLLaMA (Ma et al., 2024), E5-Mistral (Wang et al., 2024), and Qwen3-Embedding (Qwen Team, 2025) use last-token pooling.”
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs (2025, unknown): “5/4 data (Wang et al., 2023).”
• ReasonIR: Training Retrievers for Reasoning Tasks (2025, unknown): “Query decomposition is another test-time technique has been shown to be effective on multi-hop retrieval tasks (Wang et al., 2024; Jin et al., 2025).”
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “Previous pre-LLM-era retriever models predominantly utilized encoder-only architectures, such as BERT-base for English retrieval and XLM-RoBERTa-Large for multilingual retrieval.”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “Strong foundation models have proven crucial for IR performance (Neelakantan et al., 2022; Ma et al., 2024b).”
• LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “This enhanced capacity enables LLMs to generate more accurate and generalizable embeddings, which is expected to improve performance across a wide range of downstream tasks compared to traditional encoder-only models.”
• Can't Hide Behind the API: Stealing Black-Box Commercial Embedding Models (2024, unknown): “For efficiency, we prioritize using the shorter embeddings for retrieval whenever possible.”
• Bagging-Based Model Merging for Robust General Text Embeddings (2026, unknown): “Large language models (LLMs) have demonstrated strong performance across a variety of NLP tasks, and recent studies increasingly explore their potential as backbone encoders for text embedding [4, 26, 27, 29, 33, 37, 50, 65, 66, 69].”
• CoQuIR: A Comprehensive Benchmark for Code Quality-Aware Information Retrieval (2025, unknown): “5b-instruct, E5-Mistral-7b-instruct, and RepLLaMA [20] models at 3B and 8B scales.”