LongEmbed: Extending Embedding Models for Long Context Retrieval

By Dawei Zhu, Liang Wang, Nan Yang, Yifan Song, Wenhao Wu, Fu-Ru Wei, Su-Jian Li. 2024

Subfields: Information Retrieval & Search, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Retrieval & Search

1. Ziyan Jiang
2. Wenhu Chen
3. Xue-Guang Ma

In Representation & Self-Supervised Learning

1. Ziyan Jiang
2. Wenhu Chen
3. Xue-Guang Ma

Citations in the corpus

• HAKARI-Bench: A Lightweight Benchmark for Comparing Retrieval Architectures and Efficiency Settings under Unified Conditions (2026, unknown): “For multilingual monolingual retrieval, MIRACL (Zhang et al., 2023) is widely used; for short-query passage retrieval, MS MARCO (Bajaj et al., 2016); and for domain specialization, code retrieval CoIR (Li et al., 2024), long-document retrieval LongEmbed (Zhu et al., 2024), instruction-following retr”
• Granite Embedding Multilingual R2 Models (2026, unknown): “• Long Context Retrieval: To evaluate performance on retrieving long-context documents, we measure performance on the LongEmbed benchmark (Zhu et al., 2024).”
• Fantastic (small) Retrievers and How to Train Them: mxbai-edge-colbert-v0 Tech Report (2025, unknown): “In Table 11, we present the result of our model on a selected range of BEIR [28] datasets and their average, while Table 12, we present the result of our model on LongEmbed task [41].”
• Granite Embedding R2 Models (2025, unknown): “• Long Context Retrieval: To evaluate performance on retrieving long-context documents, we measure the performance on the MLDR task (Chen et al., 2024) and the LongEmbed benchmark (Zhu et al., 2024).”
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs (2025, unknown): “More recent retrieval benchmarks include AIR Bench (Chen et al., 2024), LongEmbed (Zhu et al., 2024) and BRIGHT (Su et al., 2024); we save the analysis of LLM scaling properties on these benchmarks for future work.”
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “When comparing our pruned model to existing E5-v2-base and its variants that apply RoPE positional embeddings (Zhu et al., 2024), D RAMA 0.1B (with the same parameter size) demonstrates a clear advantage across all input lengths.”
• LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “[196] 2025 Case/Legal Retrieval Symbolic, Legal Reasoning Legal”
• Length-Induced Embedding Collapse in PLM-based Models (2024, unknown): “…to develop long-context embedding models generally begin with acquiring a long-context backbone model, either by pre-training from scratch with long inputs (Gün-ther et al., 2023; Nussbaum et al., 2024; Chen et al., 2024a) or utilizing existing models (Wang et al., 2024; Zhu et al., 2024).”
• LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs (2024, methodology): “For HotpotQA, we compute AR only for questions with span answers, specifically the “bridge” type questions, while ignoring yes/no and comparison questions, following previous work (Khalifa et al., 2022 .69).”
• jina-embeddings-v4: Universal Embeddings for Multimodal Multilingual Retrieval (2025, unknown): “We have also evaluated the performance of our model on retrieval tasks that involve long text documents using the LongEmbed benchmark [Zhu et al., 2024].”