Genie: Generative Interactive Environments
By Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holder, Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald, Chris Apps and 15 more. International Conference on Machine Learning, 2024
696 citations recorded by the source corpus.
We introduce Genie, the first generative interactive environment trained in an unsupervised manner from unlabelled Internet videos. The model can be prompted to generate an endless variety of action-controllable virtual worlds described through text, synthetic images, photographs, and even sketches. At 11B parameters, Genie can be considered a foundation world model. It is comprised of a spatiotemporal video tokenizer, an autoregressive dynamics model, and a simple and scalable latent action model. Genie enables users to act in the generated environments on a frame-by-frame basis despite training without any ground-truth action labels or other domain-specific requirements typically found in the world model literature. Further the resulting learned latent action space facilitates training agents to imitate behaviors from unseen videos, opening the path for training generalist agents of the future.
Subfields: Sociology and Political Science
View Paper: https://www.semanticscholar.org/paper/c064de2c71ebc5cf05493f49dc312b033c36b3b9
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Sociology and Political Science
Citations in the corpus
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Intelligent agents should be able to make efficacious decisions even under unseen situations [9, 57, 76, 113, 138, 149].”
- AgiBot World Colosseo: A Large-Scale Manipulation Platform for Scalable and Intelligent Embodied Systems (2025, positive): “To broaden the data pool by incorporating internet-scale human videos lacking action labels and cross-embodiment robot data, we employ latent actions [30] in Stage 1 to model the inverse dynamics of consecutive frames.”
- InternVideo2: Scaling Foundation Models for Multimodal Video Understanding (2024, background): “…is a critical research area in computer vision, holding diverse applications across domains such as video search [Gabeur et al., 2020], game control [Bruce et al., 2024], robotic learning [Driess et al., 2023], self-driving [Zablocki et al., 2022], and scientific studies [Team et al., 2023].”
- Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, neutral): “Teaching the AI system to generate various video content plays a vital role in simulating the real-world dynamics (Hu et al., 2023; Brooks et al., 2024) and interacting with humans (Bruce et al., 2024; Valevski et al., 2024).”
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, positive): “Autoregressive models, in contrast, are trained with next-token prediction objectives and generate spatiotemporal tokens sequentially at inference time [7, 38, 66, 86, 88, 94].”
- UniVLA: Learning to Act Anywhere with Task-centric Latent Actions (2025, positive): “In comparison to the latent action construction approach introduced in Genie [12], which captures all visual changes, our method demonstrates clear superiority.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, positive): “Genie [84] introduces a new class of generative models, termed Generative Interactive Environments.”
- V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “Some recent works leverage both internet-scale video and interaction data towards training general purpose (task-agnostic) action-conditioned video generation models for autonomous robots (Bruce et al., 2024; Agarwal et al., 2025; Russell et al., 2025).”