Physion: Evaluating Physical Prediction from Vision in Humans and Machines

By Daniel M. Bear, E. Wang, Damian Mrowca, Felix J. Binder, H. Tung, R. Pramod, Cameron Holdaway, Sirui Tao, Kevin A. Smith, Li Fei-Fei and 4 more. 2021

160 citations recorded by the source corpus.

While current vision algorithms excel at many challenging tasks, it is unclear how well they understand the physical dynamics of real-world environments. Here we introduce Physion, a dataset and benchmark for rigorously evaluating the ability to predict how physical scenarios will evolve over time. Our dataset features realistic simulations of a wide range of physical phenomena, including rigid and soft-body collisions, stable multi-object configurations, rolling, sliding, and projectile motion, thus providing a more comprehensive challenge than previous benchmarks. We used Physion to benchmark a suite of models varying in their architecture, learning objective, input-output structure, and training data. In parallel, we obtained precise measurements of human prediction behavior on the same set of scenarios, allowing us to directly evaluate how well any model could approximate human behavior. We found that vision algorithms that learn object-centric representations generally outperform those that do not, yet still fall far short of human performance. On the other hand, graph neural networks with direct access to physical state information both perform substantially better and make predictions that are more similar to those made by humans. These results suggest that extracting physical representations of scenes is the main bottleneck to achieving human-level and human-like physical understanding in vision algorithms. We have publicly released all data and code to facilitate the use of Physion to benchmark additional models in a fully reproducible manner, enabling systematic evaluation of progress towards vision algorithms that understand physical environments as robustly as people do.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “For physics, the furtherance of physics-based predictive models [113]–[115] have accentuate on the importance of embodied AI simulators with advanced physics features as they serve to provide an ideal testbed for training embodied AI agents to perform tasks with sophisticated physical interactions [”
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation (2020, methodology): “Advanced Physical Prediction Benchmark Using the TDW platform, we have created a comprehensive Pysion benchmark for training and evaluation of physically-realistic forward prediction algorithms [6].”
PIP: Physical Interaction Prediction via Mental Imagery with Span Selection (2021, background): “Physion [40] presents a dataset which comprises eight physical interaction scenarios in the ThreeDWorld simulator [8].”
Perception Test: A Diagnostic Benchmark for Multimodal Video Models (2023, background): “A few research works have highlighted the need for robust diagnostics benchmarks, e.g. CATER [25], CLEVRER [54], IntPhys [44], Physion [7].”