Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
By Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser and 663 more. arXiv.org, 2024
3,818 citations recorded by the source corpus.
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.
Subfields: Artificial Intelligence, Large Language Models, Multimodal Learning
View Paper: https://arxiv.org/abs/2403.05530
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Graham Neubig
- Mohit Bansal
- Jie Tang
- Luke S. Zettlemoyer
- Jianfeng Gao
- Quoc V. Le
- Yoshua Bengio
- Percy Liang
In Large Language Models
In Multimodal Learning
Citations in the corpus
- Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “However, there is severe information loss in the captioning process; thus, using the captions (as opposed to vision features) may suffer from a lack of mutual synergy in the representation space of different modalities.”
- From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline (2024, methodology): “Additionally, Arena-Hard-Auto v0.”
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “These models, including state-of-the-art examples like GPT-4, Gemini, and Claude [2, 30, 6], are pushing the envelope both in general applicability across various tasks and specialized performance in specific areas.”
- MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention (2024, positive): “Large language models (LLMs) have entered the era of long-context processing, with some of them supporting context windows ranging from 128K to 10M tokens [Gra24, RST + 24, LYZA24, YCL + 24, AJA + 24, DA24].”
- Qwen3-Omni Technical Report (2025, positive): “Building on the rapid advances in the understanding and reasoning capabilities of unimodal large models (Brown et al., 2020; OpenAI, 2023; Gemini Team, 2024; Anthropic, 2023a;b; 2024; Bai et al., 2023a; Yang et al., 2024; 2025a; Touvron et al., 2023; Dubey et al., 2024; Li et al., 2023; Liu et al.,…”
- Qwen2.5-Omni Technical Report (2025, neutral): “…general artificial intelligence has become increasingly visible, largely due to advancements in Large Language Models (LLMs) (Brown et al., 2020; OpenAI, 2023; 2024; Gemini Team, 2024; Anthropic, 2023a;b; 2024; Bai et al., 2023a; Yang et al., 2024a; Touvron et al., 2023a;b; Dubey et al., 2024a).”
- Continual Learning of Large Language Models: A Comprehensive Survey (2024, neutral): “There are two popular pre-training paradigms for LLMs. (1) Decoder-only models typically employ auto-regressive language modeling ( LM ) tasks during pre-training, including the GPT family [1, 22, 173, 186], Gemini family [194, 225], and the open-source Llama family [230, 231].”
- LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, neutral): “5-pro [46], underscoring the effectiveness of our structured reasoning approach.”
- How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “In this approach, images are segmented into 448 × 448 pixel tiles, with the number of tiles ranging up to 12 based on the image’s aspect ratio and resolution during training.”
- DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, neutral): “• General language models that we compare include Llama3 70B (Meta, 2024), GPT-4 (OpenAI, 2023), Claude 3 Opus (Anthropic, 2024), and Gemini 1.5 Pro (Reid et al., 2024).”
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “5-Pro [200].”
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark (2024, methodology): “MMMU has quickly become a standard evaluation tool in the field, used to assess the capabilities of many prominent multimodal models upon their release (OpenAI, 2024a,b; Anthropic, 2024; Reid et al., 2024; Li et al., 2024b).”
- Training Language Models to Self-Correct via Reinforcement Learning (2024, neutral): “We implement this via reward shaping : by rewarding transitions that make “progress” towards learning the desired self-correction behavior.”
- SnapKV: LLM Knows What You are Looking for Before Generation (2024, neutral): “5 with 1M [4].”
- Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces (2024, neutral): “5 [78] and GPT-4o [34].”
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, background): “5-pro-0924 ( Team et al., to Preference 700K, the 378K dataset provides a competitive advantage in Chat Hard while maintaining balanced performance across all four categories.”