WorldSimBench: Towards Video Generation Models as World Simulators

By Yiran Qin, Zhelun Shi, Jiwen Yu, Xijun Wang, Enshen Zhou, Lijun Li, Zhen-fei Yin, Xihui Liu, Lu Sheng, Jing Shao and 3 more. arXiv.org, 2024

1,198 citations recorded by the source corpus.

Recent advancements in predictive models have demonstrated exceptional capabilities in predicting the future state of objects and scenes. However, the lack of categorization based on inherent characteristics continues to hinder the progress of predictive model development. Additionally, existing benchmarks are unable to effectively evaluate higher-capability, highly embodied predictive models from an embodied perspective. In this work, we classify the functionalities of predictive models into a hierarchy and take the first step in evaluating World Simulators by proposing a dual evaluation framework called WorldSimBench. WorldSimBench includes Explicit Perceptual Evaluation and Implicit Manipulative Evaluation, encompassing human preference assessments from the visual perspective and action-level evaluations in embodied tasks, covering three representative embodied scenarios: Open-Ended Embodied Environment, Autonomous, Driving, and Robot Manipulation. In the Explicit Perceptual Evaluation, we introduce the HF-Embodied Dataset, a video assessment dataset based on fine-grained human feedback, which we use to train a Human Preference Evaluator that aligns with human perception and explicitly assesses the visual fidelity of World Simulators. In the Implicit Manipulative Evaluation, we assess the video-action consistency of World Simulators by evaluating whether the generated situation-aware video can be accurately translated into the correct control signals in dynamic environments. Our comprehensive evaluation offers key insights that can drive further innovation in video generation models, positioning World Simulators as a pivotal advancement toward embodied artificial intelligence.

Subfields: Building and Construction, Control and Systems Engineering, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/bc2ddba47a9d6b378abd8a8584849f3e4c410f24

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Control and Systems Engineering

1. Jinwoo Shin
2. Zehuan Yuan

In Management Science and Operations Research

1. Jinwoo Shin
2. Ziyang Luo
3. Xin Li
4. Jongheon Jeong

Citations in the corpus

• VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs (2024, background): “Despite these successes, the domain of video understanding and generation (Kondratyuk et al., 2023; Menapace et al., 2024; Kondratyuk et al., 2023; Brooks et al., 2024) remains relatively nascent.”
• Emu3: Next-Token Prediction is All You Need (2024, neutral): “Unlike Sora [8] that employs the video diffusion model to generate a video from noise, Emu3 simply generates a video causally by predicting the next token in a video sequence.”
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “DiT[46] replaces the traditional U-Net with a transformer-based architecture [66, 17], and is used in recent image or video synthesis systems like Stable Diffusion 3.0 [18] and SORA [8].”
• A Survey on Vision--Language--Action Models for Embodied AI (2024, neutral): “This sets it apart from other AI models, such as conversational AI, which primarily handles textual conversations (ChatGPT [1]), or generative AI models that focus on tasks like text-to-video generation (Sora [33]).”
• Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, neutral): “As the size of foundation models increases, identifying and adapting PEFT strategies that remain effective is crucial.”
• Large Language Diffusion Models (2025, neutral): “The success of diffusion trans-formers [9, 10] on visual data [11] supports this claim.”
• Mini-Gemini: Mining the Potential of Multi-Modality Vision Language Models (2024, positive): “The convergence of computer vision and natural language processing has given rise to VLMs, which marry visual and linguistic models to achieve cross-modal comprehension and reasoning capabilities.”
• Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, neutral): “…tools have been metic-ulously crafted to produce diverse outputs across various modalities, such as Large Language Models (LLMs) including the GPT series [1]–[3] and the LLAMA series [4]–[6] for texts and codes, DALL-E [7]–[9] and Stable Diffusion [10] for images, and Sora [11] for videos.”
• Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, neutral): “This has inspired research into scaling in visual generation [7, 22, 59, 64, 82].”
• LaVie: High-Quality Video Generation with Cascaded Latent Diffusion Models (2023, neutral): “Several latest works (Brooks et al., 2024; Ma et al., 2024; Gupta et al., 2023; Lu et al., 2023; Yang et al., 2024) extend such idea on video generation and show promising results on large-scale text-to-video generation.”
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, neutral): “Other significant attempts include leveraging scalable transformer backbone (Ma et al., 2024), operating in space-temporal compressed latent space, e.g. , W.A.L.T. (Gupta et al., 2023) and Sora (Brooks et al., 2024), and using discrete token with language model for video generation (Kondratyuk et al”
• Pyramidal Flow Matching for Efficient Video Generative Modeling (2024, neutral): “Teaching the AI system to generate various video content plays a vital role in simulating the real-world dynamics (Hu et al., 2023; Brooks et al., 2024) and interacting with humans (Bruce et al., 2024; Valevski et al., 2024).”
• Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Think (2024, positive): “Based on these improvements, Pixart-α (Chen et al., 2024b), Pixart-Σ (Chen et al., 2024a), Stable diffusion 3 (Esser et al., 2024) show pure transformers can be scaled up for challenging text-to-image generation, and CMD (Yu et al., 2024), WALT (Gupta et al., 2024), and Sora (Brooks et al., 2024) de”
• Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “Furthermore, it provides standard metrics for evaluating agent performance across these tasks.”
• Flow-GRPO: Training Flow Matching Models via Online RL (2025, neutral): “It achieves competitive FID with far fewer denoising steps than diffusion, making it the dominant choice in recent image [4, 5] and video [24, 25, 26, 27] generation models.”
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation (2024, neutral): “It is worth noticing that our released models are still behind state-of-the-art visual generation models based on diffusion models [Alpha-VLLM 2024; Esser et al. 2024; Brooks et al. 2024].”
• Genie: Generative Interactive Environments (2024, neutral): “…to video models , which typically condition on initial frames (or text) and predict the remaining frames in a video (Blattmann et al., 2023b; Brooks et al., 2024; Clark et al., 2019; Finn et al., 2016; Ho et al., 2022a,b; Höppe et al., 2022; Kalchbrenner et al., 2017; Le Moing et al., 2021;…”