Mistral 7B

By Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier and 8 more. arXiv.org, 2023

3,685 citations recorded by the source corpus.

We introduce Mistral 7B v0.1, a 7-billion-parameter language model engineered for superior performance and efficiency. Mistral 7B outperforms Llama 2 13B across all evaluated benchmarks, and Llama 1 34B in reasoning, mathematics, and code generation. Our model leverages grouped-query attention (GQA) for faster inference, coupled with sliding window attention (SWA) to effectively handle sequences of arbitrary length with a reduced inference cost. We also provide a model fine-tuned to follow instructions, Mistral 7B -- Instruct, that surpasses the Llama 2 13B -- Chat model both on human and automated benchmarks. Our models are released under the Apache 2.0 license.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

View Paper: https://arxiv.org/abs/2310.06825

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Mixtral of Experts (2024, methodology): “Mixtral is based on a transformer architecture [31] and uses the same modifications as described in [18], with the notable exceptions that Mix-tral supports a fully dense context length of 32k tokens, and the feed-forward blocks are replaced by Mixture-of-Expert layers (Section 2.1).”
MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, neutral): “We employ eight models: Llama-2-7B, Llama-2-7B-Chat, Llama-2-13B, Llama-2-13B-Chat [35], Mistral-7B [20], Gemma-7B [34], Yi-6B, and Yi-6B-Chat [43].”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “…7B leads in performance across all eight benchmarks among the open-source base models (including the widely-used general model Mistral 7B (Jiang et al., 2023) and the recently released Llemma 34B (Azerbayev et al., 2023) which underwent math training on Proof-Pile-2 (Azerbayev et al.,…”
LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders (2024, background): “These results are particularly encouraging for settings where it is hard to acquire high quality labeled data, a setting which we leave for future work.”
KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization (2024, methodology): “We used the LLaMA-7B/13B/30B/65B, LLaMA-2-7B/13B/70B, and Mistral-7B models to evaluate our methodology [15, 30, 31].”
AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration (2023, neutral): “…al., 2022); VQA T : TextVQA (Singh et al., 2019); POPE (Li et al., 2023d); MME (Fu et al., 2023); MMB: MMBench (Liu et al., 2023b); MMB CN : MMBench-Chinese (Liu et al., 2023b); SEED: SEED-Bench (Li et al., 2023a); LLaVA W : LLaVA-Bench (In-the-Wild) (Liu et al., 2023a); MM-Vet (Yu et al., 2023).”
Large Language Diffusion Models (2025, neutral): “To comprehensively evaluate the in-context learning and instruction-following capabilities of LLaDA 8B, we conducted detailed comparisons with existing LLMs [6, 21, 25, 26, 32, 33] of similar scale.”
KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache (2024, methodology): “We apply KIVI to Llama2-7B, Llama2-13B, Llama2-7B-Chat, Llama2-13B-Chat, Falcon-7B and Mistral-7B.”
SpinQuant: LLM quantization with learned rotations (2024, methodology): “To rigorously assess the effectiveness of SpinQuant , we executed comprehensive experiments across seven leading Large Language Models (LLMs), including LLaMA-2[41] models (7B/13B/70B), LLaMA-3[3] models (1B/3B/8B), and the Mistral [17] 7B model.”
CodeMind: A Framework to Challenge Large Language Models for Code Reasoning (2024, neutral): “5 (OpenAI, 2023a), Llama 2 (13B) (Touvron et al., 2023), Mistral (Jiang et al., 2023), CodeLlama (13B, instruction-tuned)(Roziere et al., 2023), StarCoder (15.5B)(Li et al., 2023), Wizard-Coder (15B, instruction-tuned)(Xu et al., 2023), Magic-Coder (7B)(Wei et al., 2023) (instruction-tuned),…”
AI and Memory Wall (2024, neutral): “Deploying recent SOTA models [4, 5, 16, 37] such as GPT-3 [3] or large recommendation systems [29] is quite challenging, as they require distributed-memory deployment for inference.”
InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, methodology): “…similar to other foundational models, InternVideo2 has the potential to embed biases present in its training data and the associated models used during training, such as neural teachers [Tong et al., 2022, Chen et al., 2023c] and language models (LLMs) [Jiang et al., 2023, Zheng et al., 2023].”
A Survey on Mixture of Experts in Large Language Models (2024, neutral): “Forexample,theMixtral8x7B[26],introducedbyMixtral AI, shares its foundational architecture with the earlier Mis-tral 7B [130], but with a notable difference: each layer comprises eight feed-forward networks (FFN) (i.e., experts).”
What matters when building vision-language models? (2024, neutral): “We start from SigLIP-SO400M and Mistral-7B-v0.”
Break the Sequential Dependency of LLM Inference Using Lookahead Decoding (2024, neutral): “However, current LLMs generate text based on (Touvron et al., 2023a;b; Jiang et al., 2023; OpenAI, 2023) autoregressive decoding , which falls short in efficiency, primarily for two reasons.”
SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics (2025, neutral): “…models (LLMs), which have demonstrated performance comparable to the average human in understanding and generating natural language, reasoning over complex topics, and anchoring in knowledge (Brown et al., 2020; Achiam et al., 2023; Dubey et al., 2024; Team et al., 2023; Jiang et al., 2023).”