Generative Representational Instruction Tuning

By Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Fu-Ru Wei, Tao Yu, Amanpreet Singh, D. Kiela. 2024

Subfields: Large Language Models, Representation & Self-Supervised Learning, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

1. Hong-Liu Cao
2. Luke Merrick

In Representation & Self-Supervised Learning

1. Hong-Liu Cao

Citations in the corpus

• Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “Concurrent work GRITLM (Muennighoff et al., 2024) demonstrates that MoE models like Mixtral 8x7B can effectively handle both embedding and generation tasks through instruction tuning.”
• HypRAG: Hyperbolic Dense Retrieval for Retrieval Augmented Generation (2026, unknown): “More recently, decoder-only embedding models initialize from LLMs to exploit their pretrained linguistic knowledge (Muennighoff et al., 2024; Lee et al., 2024; Zhang et al., 2025).”
• GigaEmbeddings: Efficient Russian Language Embedding Model (2025, unknown): “• multilingual-E5-large-instruct (Wang et al., 2022); • E5-Mistral-instruct (Wang et al., 2024b); • SFR-Embedding-Mistral (Rui Meng, 2024); • GritLM-7B (Muennighoff et al., 2025)”
• Retrieval Capabilities of Large Language Models Scale with Pretraining FLOPs (2025, unknown): “More recent retrieval benchmarks include AIR Bench (Chen et al., 2024), LongEmbed (Zhu et al., 2024) and BRIGHT (Su et al., 2024); we save the analysis of LLM scaling properties on these benchmarks for future work.”
• ReasonIR: Training Retrievers for Reasoning Tasks (2025, unknown): “Following Muennighoff et al. (2024), we include a set of public training data collected from real-world applications, such as MS MARCO (Nguyen et al., 2016), Natural Questions (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018) (details in Appendix D).”
• Mieb: Massive Image Embedding Benchmark (2025, unknown): “While decoder-based generative models show inherent generalizability in embedding tasks [15, 26, 42, 49, 55, 60], it is likely still necessary to learn robust fine-grained nuances through contrasting multimodality finetuning paired with validated training recipes like large batch sizes and diverse…”
• DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers (2025, unknown): “Recent advancements in large language models (LLMs) have demonstrated their effectiveness and robustness in text retrieval tasks (Muennighoff et al., 2024; Sun et al., 2023; Li et al., 2025; BehnamGhader et al., 2024; Lee et al., 2025).”
• A Survey of Model Architectures in Information Retrieval (2025, unknown): “Future IR models should support end-to-end system optimization rather than focusing solely on ranking metrics.”
• mFollowIR: a Multilingual Benchmark for Instruction Following in Retrieval (2025, unknown): “In IR, steerability through instructions presents a unique opportunity to adapt retrieval models to unseen definitions of relevance at inference time [5, 34, 47, 59].”
• LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “For instance, methods such as corpus-aware pre-training [75], multi-task learning [108], hard negative construction [168], and distillation from cross-encoder models [69] are still natural and practical to LLM-based approaches.”