Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free

By Zi-Yue Li, Tianyi Zhou. 2024

Subfields: Large Language Models, Representation & Self-Supervised Learning, Model Architectures

Citations in the corpus

• Training Sparse Mixture Of Experts Text Embedding Models (2025, unknown): “While Hallee et al. (2024) explored domain-specific MoE embeddings and Li & Zhou (2024) investigated using MoE language model states as embeddings, our work is the first to develop a general-purpose MoE architecture specifically for text embeddings.”
• Mixture-of-Experts Language Models Can Be Strong and Efficient Retrievers (2026, unknown): “MoEE (Li and Zhou, 2025) instead uses the routing weights of a frozen MoE, alone or together with its hidden states, to construct training-free embeddings without fine-tuning the backbone for retrieval.”
• LLMs Are Also Effective Embedding Models: An In-depth Overview (2024, unknown): “Table 1 summarize summarizes the configurations of representative models in terms of these three aspects.”
• Unlocking Multimodal Document Intelligence: From Current Triumphs to Future Frontiers of Visual Document Retrieval (2026, unknown): “This includes establishing "firewalled" test sets derived from entirely held-out web crawls or proprietary document collections and developing standardized protocols for data contamination detection, thereby fostering a more reliable assessment of model generalization and pushing the field towards…”