GAIA-1: A Generative World Model for Autonomous Driving

By Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, Gianluca Corrado. arXiv.org, 2023

613 citations recorded by the source corpus.

Autonomous driving promises transformative improvements to transportation, but building systems capable of safely navigating the unstructured complexity of real-world scenarios remains challenging. A critical problem lies in effectively predicting the various potential outcomes that may emerge in response to the vehicle's actions as the world evolves. To address this challenge, we introduce GAIA-1 ('Generative AI for Autonomy'), a generative world model that leverages video, text, and action inputs to generate realistic driving scenarios while offering fine-grained control over ego-vehicle behavior and scene features. Our approach casts world modeling as an unsupervised sequence modeling problem by mapping the inputs to discrete tokens, and predicting the next token in the sequence. Emerging properties from our model include learning high-level structures and scene dynamics, contextual awareness, generalization, and understanding of geometry. The power of GAIA-1's learned representation that captures expectations of future events, combined with its ability to generate realistic samples, provides new possibilities for innovation in the field of autonomy, enabling enhanced and accelerated training of autonomous driving technology.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/98478ac589e5b40a20630ff54bb4eec4ab4c5f6b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models (2024, background): “Recent works [19, 51, 52] argue that language captions are an important medium to connect human knowledge with the driving objective, helping to inform decisions and actions.”
Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, neutral): “Teaching the AI system to generate various video content plays a vital role in simulating the real-world dynamics (Hu et al., 2023; Brooks et al., 2024) and interacting with humans (Bruce et al., 2024; Valevski et al., 2024).”
Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion (2024, neutral): “For example, when attempting to auto-regressively generate a video (as opposed to text [6] or vector-quantized latents [33]) past the training horizon, slight errors in frame-to-frame predictions accumulate and the model diverges.”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “One example is the GAIA-1 model which generates predictions of driving video conditioned on arbitrary combinations of video, action, and text [191].”
Driving Into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving (2023, background): “GAIA-1 [28] and DriveDreamer [63] further extend to action-conditional diffusion models, enhancing the controllability and realism of generated videos.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Although a primary prospect of world models is to enable the generalization ability to novel environments, existing driving world models are still constrained by data scale [90, 125, 127, 143, 147] and geographical coverage [54, 61].”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, neutral): “Looking forward, we believe recent advances in video/image generation and world models [13, 20, 27, 62, 66] present promising opportunities to enhance generalization capabilities through improved visual reasoning and predictive modeling. dreamer: Learning to follow instructions via…”
VideoPoet: A Large Language Model for Zero-Shot Video Generation (2023, methodology): “Both families have been extended to text-to-video [36, 37, 65, 73] using paired data.”