The Llama 3 Herd of Models
By Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, A. Schelten, Amy Yang, Angela Fan and 517 more. 2024
17,085 citations recorded by the source corpus.
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Multimodal Learning
View Paper: https://arxiv.org/abs/2407.21783
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yoshua Bengio
- Percy Liang
- Yejin Choi
- Quoc V. Le
- Graham Neubig
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Cordelia Schmid
In Large Language Models
- Jason Aaron Edward Weston
- Graham Neubig
- Noah A. Smith
- Kai-Wei Chang
- Wenhu Chen
- Armand Joulin
- Huan Sun
- Oriol Vinyals
In Multimodal Learning
- Aniruddha Kembhavi
- Zhe Gan
- Jiasen Lu
- Jason Aaron Edward Weston
- Antoine Miech
- Devi Parikh
- Jean-Baptiste Alayrac
- Jingren Zhou
Citations in the corpus
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, neutral): “5-turbo’s performance using GPT-4-1106-Preview as judge.”
- The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity (2025, neutral): “Earlier iterations of LLMs [9, 10, 11] exhibited poor performance on reasoning benchmarks [12, 13, 14, 6].”
- Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, neutral): “Our multi-modal pretraining corpus is designed to provide high-quality data that enables models to process and understand information from multiple modalities, including text, images, and videos.”
- Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025, neutral): “3-70B-Instruct (Dubey et al., 2024), and DeepSeek-R1-Lite (Guo et al., 2025).”
- Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D Assets Generation (2025, positive): “As evidenced in other fields [115, 4, 3], the prosperity of a domain in the era of large models usually relies on a strong open-source foundational model, such as Stable Diffusion [74, 69, 24] for image generation, LLaMA [91, 92, 22] for language models, and HunyuanVideo [43] for video generation.”
- Muon is Scalable for LLM Training (2025, positive): “The rapid advancement of large language models (LLMs) (OpenAI et al. 2024; DeepSeek-AI et al. 2024; Grattafiori et al. 2024; Gemini Team et al. 2024) has significantly pushed forward the progress in artificial general intelligence.”
- Qwen2.5-Coder Technical Report (2024, background): “With the rapid development of large language models (LLMs) (Brown, 2020; Achiam et al., 2023; Touvron et al., 2023; Dubey et al., 2024; Jiang et al., 2023; Bai et al., 2023b; Yang et al., 2024; Anthropic, 2024; OpenAI, 2024), code-specific language models have garnered significant attention in the…”
- Gemma 3 Technical Report (2025, positive): “Several prior reports have released audits that quantify this risk by measuring the memorization rate (Anil et al., 2023; Chowdhery et al., 2022; Gemini Team, 2023, 2024; Gemma Team, 2024a,b; LLaMa Team, 2024).”
- MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark (2024, positive): “…the robustness of MMAU, we also perform a text-only evaluation using various open and closed-source Large Language Models (LLMs), including GPT-4o (OpenAI et al., 2024), a closed-source, state-of-the-art LLM, and Llama 3 8B Instruct (Dubey et al., 2024), an open-source, instruction-tuned model.”
- LLMs Get Lost In Multi-Turn Conversation (2025, neutral): “3-70B-Instruct, Llama 4 Scout) [23], AI2 OLMo-2-13B [56], Microsoft Phi-4 [1], Deepseek-R1 [24], and Cohere Command-A [15].”
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “The AI field has also seen a rise in open-source LLMs such as Mistral (Jiang et al., 2023), Llama Series (Touvron et al., 2023a;b; Dubey et al., 2024; Taylor et al., 2022), DeepSeek (Bi et al., 2024; Shao et al., 2024), Qwen (Bai et al., 2023; Yang et al., 2024) etc. Notably, Llama serves as a…”
- TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, positive): “Early work in language model post-training followed a standard recipe pioneered by models like Instruct-GPT [Ouyang et al., 2022], consisting of instruction-tuning followed by preference finetuning (PreFT) [Stiennon et al., 2020, Nakano et al., 2021, Askell et al., 2021, Ouyang et al., 2022].”
- PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, neutral): “In this section, we compare (Q)PiSSA and (Q)LoRA across 9 models, ranging from 7-70B parameters, including LLaMA 2-7/13B [51], LLaMA-3-8/70B [62], Mistral-7B [52], Gemma-7B [53], and Qwen1.”
- Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Models (2025, neutral): “2-V-Instruct [38] for SFT, resulting in another post-cold-start MLLM, Vision-R1-LlamaV-CI, which can also be used for subsequent RL training.”