Gemma 2: Improving Open Language Models at a Practical Size

By Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, Johan Ferret and 185 more. 2024

2,206 citations recorded by the source corpus.

In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 times bigger. We release all our models to the community.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, positive): “We perform experiments on a server with four NVIDIA A100-SXM4-80GB GPUs, an AMD EPYC 7763 64-Core Processor, and 512 GB of RAM, using the VLLM inference framework [28].”
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, neutral): “2 [DJP + 24], Ministral [Mis24] and Gemma2 [TRP + 24] series.”
Muon is Scalable for LLM Training (2025, neutral): “1-8B (Grattafiori et al. 2024), Gemma-9B (Gemma Team et al. 2024) and Qwen2.”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, neutral): “(Vavekanand & Sam, 2024), Gemma2 (9B, 27B) (Team et al., 2024), Mixtral (8x7B) (Jiang et al., 2024), and Qwen2 (7B, 72B) (Yang et al., 2024a), and the closed-source ones are GPT (3.5-Turbo, 4o, 4o-mini) (OpenAI, 2022; 2024b) and Claude-3.”
SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, positive): “We evaluate SimPO using Google’s recently released Gemma-2-9B-it model [72], which represents a strong open-source model.”
Kimi K2: Open Agentic Intelligence (2025, neutral): “For instance, logit soft-cap [70] directly clips the attention logits, but the dot products between queries and keys can still grow excessively before capping is applied.”
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference (2024, positive): “This choice is motivated by the proven performance of RoPE in short-and long-context language models (Black et al., 2022; Dubey et al., 2024; Gemma Team et al., 2024), efficient implementations in most frameworks, and ease of context extension.”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “At 8B, we compare to and surpass Llama 3.1 8B Instruct, Gemma 2 9B Instruct [Gemma Team et al., 2024], Nous Hermes 3 8B (trained on Llama 3.1 8B), Qwen 72.5B Instruct, and Ministral 8B Instruct 2410.”
GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models (2024, neutral): “17, we demonstrate that modern Language Learning Models (LLMs) such as Gemma2-9B and Phi3-Medium are capable of nearly perfect addition and subtraction up to a length of 4, as well as achieving very high accuracy in multiplication and division.”
SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformers (2024, positive): “The specific Gemma we use in Sana is the Gemma-2-2B-IT (Team, 2024) version.”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, positive): “GenRM outperforms LLM-as-a-Judge and DPO verifiers (Figure 1), while performing comparably or slightly better than discriminative verifiers (Figure C.1).”
Q-TIE: A Lightweight and Generalizable Re-ranking Framework for Temporal Information Retrieval (2026, unknown): “(Gemma Team, 2024)); reasoning-focused models (DeepSeekMath-7B (Shao et al., 2024), MetaMath-7B (Yu et al., 2023), DeepSeek-R1 (Guo et al., 2025)); and base models (Qwen2.”
Building Legal Reward Models for Grounding and Abstention (2026, unknown): “Table 2 shows that DPO refinement on CJB improves LRB performance for both Gemma 2-2B and Ministral-8B, despite using no legal-domain preference data during training.”
Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models (2026, unknown): “The system prompt collection parameterizes a single base model, google/gemma-3-4b-it [12], with 100 distinct system prompts, treating each (model, system prompt) pair as a behavioral variant. gemma-3-4b-it was chosen as the median-ASR model in the cross-model collection, enabling substantial…”