FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents

By Nandan Thakur, Jimmy Lin, Sam Havens, Michael Carbin, Omar Khattab, Andrew Drozdov. 2025

Subfields: Information Retrieval & Search, Evaluation & Benchmarks

Citations in the corpus

• MDKeyChunker: What Does One LLM Call per Chunk Buy for Markdown Retrieval? (2026, unknown): “FreshStack [Thakur et al., 2025] pairs real Stack Overflow questions with corpora of technical documentation and code.”
• Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings (2025, unknown): “…on assessing the effectiveness of embedding models (Thakur et al., 2021; Muennighoff et al., 2022; Saad-Falcon et al., 2024), but they rely on datasets where document chunks are by design self-contained answer to the queries, which is a largely idealized scenario in practice (Thakur et al., 2025).”