Mixtral of Experts

By Albert Qiaochu Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, B. Savary, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna and 17 more. arXiv.org, 2024

2,015 citations recorded by the source corpus.

We introduce Mixtral 8x7B, a Sparse Mixture of Experts (SMoE) language model. Mixtral has the same architecture as Mistral 7B, with the difference that each layer is composed of 8 feedforward blocks (i.e. experts). For every token, at each layer, a router network selects two experts to process the current state and combine their outputs. Even though each token only sees two experts, the selected experts can be different at each timestep. As a result, each token has access to 47B parameters, but only uses 13B active parameters during inference. Mixtral was trained with a context size of 32k tokens and it outperforms or matches Llama 2 70B and GPT-3.5 across all evaluated benchmarks. In particular, Mixtral vastly outperforms Llama 2 70B on mathematics, code generation, and multilingual benchmarks. We also provide a model fine-tuned to follow instructions, Mixtral 8x7B - Instruct, that surpasses GPT-3.5 Turbo, Claude-2.1, Gemini Pro, and Llama 2 70B - chat model on human benchmarks. Both the base and instruct models are released under the Apache 2.0 license.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

View Paper: https://arxiv.org/abs/2401.04088

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Qwen2 Technical Report (2024, methodology): “In terms of the largest model of Qwen2, we compare Qwen2-72B with competitive baseline open-weight models, including Mixtral-8x22B (Jiang et al., 2024), Llama-3-70B (AI@Meta, 2024), as well as Qwen1.”
TravelPlanner: A Benchmark for Real-World Planning with Language Agents (2024, methodology): “We comprehensively evaluate five LLMs, such as GPT-4 (OpenAI, 2023), Gemini (G Team et al., 2023), and Mix-tral (Jiang et al., 2024), and four planning strategies, such as ReAct (Yao et al., 2022) and Reflexion (Shinn et al., 2023), on their capability of delivering complete plans and following…”
VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, methodology): “We do not extensively search the optimal LLMs for VideoLLaMA 2 but use Mistral-Instruct (Jiang et al., 2023) and Mixtral-Instruct (Jiang et al., 2024) across all of the experiments.”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, background): “Multi-modal Large Language Models (MLLMs), building upon the prevalence of Large Language Models (LLMs) [55, 56, 45, 28, 4] and large vision models [49, 29, 68, 67, 69], have become increasingly prominent in the field.”
RouteLLM: Learning to Route LLMs with Preference Data (2024, neutral): “We opt to use gpt-4-1106-preview [20] as a representative model in M strong , and Mixtral 8x7B [17] as a representative model in M weak , to concretely evaluate router performance.”
EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024, methodology): “In this paper, we only compare with speculative sampling based methods that do not need to finetune the backbone models, ensuring the output text distribution remains constant.”
A Survey on Mixture of Experts in Large Language Models (2024, neutral): “As illustrated in Fig.”
InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model (2024, background): “Other notable LLMs encompass a range of developments, such as the OPT [96], LLaMA series [78, 79], e.g ., Mistral [37, 38], InternLM [77], GLM series [25, 93], Qwen series [6, 65], Baichuan [7], Skywork [84] and Falcon [61] have made significant contributions to the field.”
OLMo: Accelerating the Science of Language Models (2024, background): “…model weights and a brief report (Jiang et al., 2024), while LLaMA came with in-depth adaptation training instructions (Touvron et al., 2023b), and Mosaic Pretrained Transformer came with many details, including the dataset distribution, though not the data itself (MosaicML NLP Team, 2023).”
VideoAgent: Long-form Video Understanding with Large Language Model as Agent (2024, neutral): “Current large language models (LLMs) excel in reasoning and handling long contexts [13, 48, 52, 62], yet they lack the capability to process visual information.”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, positive): “A. Appendix For SPHINX-MoE, the LLM backbone is based on Mixtral-8 × 7B (Jiang et al., 2024b), which is a mixture-of-experts-based large language model.”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “(Vavekanand & Sam, 2024), Gemma2 (9B, 27B) (Team et al., 2024), Mixtral (8x7B) (Jiang et al., 2024), and Qwen2 (7B, 72B) (Yang et al., 2024a), and the closed-source ones are GPT (3.5-Turbo, 4o, 4o-mini) (OpenAI, 2022; 2024b) and Claude-3.”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “5-34B [64] and MoE models: DeepSeek-MoE-16B [65] and Mixtral-8x7B [66].”
OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework (2024, methodology): “It supports experimental features like prefix caching and multi-LoRA support. vLLM seamlessly supports the most popular open-source models on HuggingFace, including Transformer-like LLMs (e.g., Llama), Mixture-of-Expert LLMs (e.g., Mixtral [13]).”
Are We on the Right Way for Evaluating Large Vision-Language Models? (2024, methodology): “Specifically, we employ two closed-source LLMs [41, 34] and six open-source LLMs [1, 42, 49, 3, 19, 43] sizing 20B or larger to serve as inspectors.”