Gemma: Open Models Based on Gemini Research and Technology

By Gemma Team Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Kale, J Christopher Love, Pouya Dehghani Tafti and 97 more. arXiv.org, 2024

1,150 citations recorded by the source corpus.

This work introduces Gemma, a family of lightweight, state-of-the art open models built from the research and technology used to create Gemini models. Gemma models demonstrate strong performance across academic benchmarks for language understanding, reasoning, and safety. We release two sizes of models (2 billion and 7 billion parameters), and provide both pretrained and fine-tuned checkpoints. Gemma outperforms similarly sized open models on 11 out of 18 text-based tasks, and we present comprehensive evaluations of safety and responsibility aspects of the models, alongside a detailed description of model development. We believe the responsible release of LLMs is critical for improving the safety of frontier models, and for enabling the next wave of LLM innovations.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/b842b83a7ff5dff8e3b83915d8c15423b6085728

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei Xiong
2. Hanze Dong
3. Yaowei Zheng
4. Prateek Mittal
5. Liang Wang
6. Xanh Ho
7. Ashwinee Panda
8. Xiangyu Qi

Citations in the corpus

• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, positive): “Some results of the Gemma-7B and Qwen2-7B (Bai et al., 2023) models are not included in the table because the GaLore method may not be applicable to them.”
• Large Language Models for Information Retrieval: A Survey (2023, neutral): “This intriguing area remains open for further investigation and advancement.”
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, positive): “In our experiments, we adopt the Alpaca [49] implementation strategy, using the AdamW optimizer with a batch size of 128, a learning rate of 2e-5, cosine annealing schedules, and a warmup ratio of 0.03, without any weight decay.”
• What matters when building vision-language models? (2024, positive): “The development of powerful open large language models (Touvron et al., 2023; Jiang et al., 2023; Team et al., 2024) and image encoders (Zhai et al., 2023; Sun et al., 2023; Radford et al., 2021) enables researchers to build upon these unimodal pre-trained models to create advanced VLMs that solve…”
• The Linear Representation Hypothesis and the Geometry of Large Language Models (2023, methodology): “In Appendix D.2, we compare the Euclidean inner product to the causal inner product for both the LLaMA-2 model and a more recent Gemma large language model (Mesnard et al., 2024).”
• MoreHopQA: More Than Multi-hop Reasoning (2024, positive): “During analysis, we observed that the result of Gemma-7B often refuses to answer.”
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, positive): “A large body of work has examined improved methods for alignment (Rafailov et al., 2023; Ethayarajh et al., 2024; Zou et al., 2023a; Bai et al., 2022b; Ouyang et al., 2022; Touvron et al., 2023b; Team et al., 2024).”
• Domain Fine-Tuning vs. Retrieval-Augmented Generation for Medical Multiple-Choice Question Answering: A Controlled Comparison at the 4B-Parameter Scale (2026, unknown): “Fine-tuning / continued pretraining on medical corpora, which encodes domain knowledge into the model’s weights (e.g., MedGemma [10] on the Gemma [3] backbone).”
• Interpretability Framework for LLMs in Undergraduate Calculus (2025, unknown): “We evaluated three models, namely, Gemma 3 [44], LLaMA 2 [12], and LLaVA-7B [45], keeping in mind our computational capacity restrictions, which did not allow us to run larger models and literature surveys, as discussed in Section 2.”
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning (2025, unknown): “Multilingual LMs Multilingual models have been developed early on through the use of machine translation systems (Artetxe et al., 2017; Team et al., 2022; Fan et al., 2021), and recently with decoder-only LMs (Gemma et al., 2024; Dubey et al., 2024).”
• Malgta: large language model-based guided malware tactical analysis (2025, unknown): “This study includes several open-source models for a comprehensive comparison: Qwen2-7B [59], Gemma-7B [60], Llama-7B [61], and Phi3-14B [62].”
• Better Language Models Exhibit Higher Visual Alignment (2024, unknown): “Gemma-2 9B (Mesnard et al., 2024) 42 .”
• RLHF Workflow: From Reward Modeling to Online RLHF (2024, methodology): “• As a case study, consider using the behavior policy Gemma-7B-it to collect data for aligning Gemma-2B-it (Team et al., 2024) using 15k prompts from (Cui et al., 2023).”