PHYRE: A New Benchmark for Physical Reasoning

By Anton Bakhtin, L. V. D. van der Maaten, Justin Johnson, Laura Gustafson, Ross B. Girshick. 2019

189 citations recorded by the source corpus.

Understanding and reasoning about physics is an important ability of intelligent agents. We develop the PHYRE benchmark for physical reasoning that contains a set of simple classical mechanics puzzles in a 2D physical environment. The benchmark is designed to encourage the development of learning algorithms that are sample-efficient and generalize well across puzzles. We test several modern learning algorithms on PHYRE and find that these algorithms fall short in solving the puzzles efficiently. We expect that PHYRE will encourage the development of novel sample-efficient agents that learn efficient but useful models of physics. For code and to play PHYRE for yourself, please visit this https URL.

Subfields: Artificial Intelligence, Prompting & Reasoning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Prompting & Reasoning

In Evaluation & Benchmarks

Citations in the corpus

Experience Grounds Language (2020, background): “Bakhtin et al. (2019) extends this physical reasoning to complex puzzles of cause and effect.”
CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, background): “…has been historically studied from the perspective of probabilistic and causal reasoning (Pearl, 2009), cognitive psychology (Grifﬁths & Tenenbaum, 2005), and more recently in the context of machine learning (Goyal et al., 2019b; Sch ¨ olkopf, 2019; Baradel et al., 2019; Bakhtin et al., 2019).”
Meta-Learning in Neural Networks: A Survey (2020, background): “Even Meta-World, ProcGen and PHYRE have dozens rather than hundreds of tasks like vision benchmarks such as tieredImageNet.”
COPHY: Counterfactual Learning of Physical Dynamics (2019, background): “Phyre (Bakhtin et al., 2019) is an environment for solving physics based puzzles, where achieving sample efﬁciency may implicitly require counterfactual reasoning, but this component is not explicitly evaluated, construction of parallel data with several alternative outcomes is not straightforward,…”
Discovering Symbolic Models from Deep Learning with Inductive Biases (2020, background): “Other related work which studies the physical reasoning abilities of deep models include [20, 21, 22].”
Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning (2021, background): “, 2020]), for Physical reasoning Bakhtin et al. [2019] and for video gaming domain (Arcade Learning Environment, CoinRun [Cobbe et al.”
Generalization to New Actions in Reinforcement Learning (2020, background): “Concurrently developed related environments (Allen et al., 2019; Bakhtin et al., 2019) focus on single-step physical reasoning with a few simple polygon tools.”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “, 2019) or visual perception (Bakhtin et al., 2019; Yi et al., 2018), for improved game playing (Lee et al.”
Physion: Evaluating Physical Prediction from Vision in Humans and Machines (2021, background): “Earlier benchmarks have tested on one or a few scenarios in reduced physical and physical environments [29, 49, 6], which severely limits the role of fine-scale object shape, motion, and”
Lagrangian Neural Networks (2020, background): “Whereas a human can quickly deduce that a ball thrown upward will return to their hand at roughly the same velocity, a neural network may never grasp this abstraction, even after seeing thousands of examples (Bakhtin et al., 2019).”
Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning (2021, background): “Recently, various benchmarks (Riochet et al., 2018; Bakhtin et al., 2019; Girdhar & Ramanan, 2020; Baradel et al., 2020; Gan et al., 2020) have been proposed to study dynamics and reasoning in physical scenes.”
PROST: Physical Reasoning about Objects through Space and Time (2021, background): “Bakhtin et al. (2019) provides a set of 2D puzzles that involve placing a new object in a scene to accomplish a goal.”
Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language (2021, background): “Benchmarks like PHYRE [7] study physical understanding and reasoning based on pure videos without concept learning and language inference.”
ESPRIT: Explaining Solutions to Physical Reasoning Tasks (2020, methodology): “We demonstrate ESPRIT on the PHYsical REasoning (PHYRE) benchmark (Bakhtin et al., 2019).”
CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning (2019, background): “Concurrent to us, CLEVRER (Yi et al., 2020), PHYRE (Bakhtin et al., 2019), COPHY (Baradel et al., 2020) and IntPhys (Riochet et al., 2018) benchmarks have been proposed with a focus on causal physical reasoning through QA, RL, prediction and ranking interfaces respectively.”