Learning Interactive Real-World Simulators
By Mengjiao Yang, Yilun Du, Kamyar Ghasemipour, Jonathan Tompson, Dale Schuurmans, Pieter Abbeel. International Conference on Learning Representations, 2023
467 citations recorded by the source corpus.
Generative models trained on internet data have revolutionized how text, image, and video content can be created. Perhaps the next milestone for generative models is to simulate realistic experience in response to actions taken by humans, robots, and other interactive agents. Applications of a real-world simulator range from controllable content creation in games and movies, to training embodied agents purely in simulation that can be directly deployed in the real world. We explore the possibility of learning a universal simulator (UniSim) of real-world interaction through generative modeling. We first make the important observation that natural datasets available for learning a real-world simulator are often rich along different dimensions (e.g., abundant objects in image data, densely sampled actions in robotics data, and diverse movements in navigation data). With careful orchestration of diverse datasets, each providing a different aspect of the overall experience, we can simulate the visual outcome of both high-level instructions such as"open the drawer"and low-level controls from otherwise static scenes and objects. We use the simulator to train both high-level vision-language policies and low-level reinforcement learning policies, each of which can be deployed in the real world in zero shot after training purely in simulation. We also show that other types of intelligence such as video captioning models can benefit from training with simulated experience, opening up even wider applications. Video demos can be found at https://universal-simulator.github.io.
Subfields: Artificial Intelligence, Control and Systems Engineering, Developmental and Educational Psychology
View Paper: https://www.semanticscholar.org/paper/c3d14e7a319ab764297a60112ce74af201762a73
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Tim Rocktaschel
- Nicolas Manfred Otto Heess
- Nando de Freitas
- Mohit Bansal
- Chelsea Finn
In Control and Systems Engineering
- Nando de Freitas
- Sherjil Ozair
- Simon Osindero
- Tim Rocktaschel
- Nicolas Manfred Otto Heess
- Yusuf Aytar
- Konrad Zolna
- Sergey Levine
In Developmental and Educational Psychology
Citations in the corpus
- Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion (2025, unknown): “This approach not only significantly reduces the viewing latency of generated videos but also unlocks numerous applications, including real-time interactive content creation [9, 46], game simulation [11, 61, 78, 102], and robotics learning [42, 96, 101].”
- Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, unknown): “(3) In collaboration with the reward function, it is also worth investigating if Vista could facilitate model-based reinforcement learning by boosting the sampling efficiency in real-world scenarios [137].”
- Genie: Generative Interactive Environments (2024, background): “GAIA-1 (Hu et al., 2023) and UniSim (Yang et al., 2023) learn world models for autonomous driving and robotic manipulation respectively.”
- A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “UniSim [86] builds a generative model based on real-world interaction videos.”