Learning to See Physics via Visual De-animation

By Jiajun Wu, Erika Lu, Pushmeet Kohli, Bill Freeman, Josh Tenenbaum. 2017

228 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

PIQA: Reasoning about Physical Commonsense in Natural Language (2019, background): “State-of-theart models can reliably answer questions given an encyclopedia article (Rajpurkar et al. 2016) or recognize named entities (Tjong Kim Sang and De Meulder 2003), but it is not clear whether they can robustly answer questions that require physical commonsense knowledge.”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, background): “Game Engines as World Models : The concept of using TextWorld as a “game engine” to represent the world is broadly related to inverse graphics (Kulkarni et al., 2015) and inverse dynamics (Wu et al., 2017) where abstract visual or physical models are used for reasoning and future predictions.”
Structured agents for physical construction (2019, background): “Modern advances in learning-based approaches have opened new avenues for using vector and convolutional representations for physical reasoning (Wu et al., 2015; 2016; 2017; Mottaghi et al., 2016; Fragkiadaki et al., 2016; Finn et al., 2016; Agrawal et al., 2016; Lerer et al., 2016; Li et al., 2016;…”
Stochastic Prediction of Multi-Agent Interactions from Partial Observations (2019, background): “( Vondrick et al., 2016a) train a conditional CNN to predict future feature vectors, and several papers (e.g., (Lerer et al., 2016; Mottaghi et al., 2016; Wu et al., 2017; van Steenkiste et al., 2018; Fragkiadaki et al., 2016; Watters et al., 2017)) predict future object states.”
Relational inductive biases, deep learning, and graph networks (2018, methodology): “…of principled hybrids of structure-based methods and deep learning (e.g., Reed and De Freitas, 2016; Garnelo et al., 2016; Ritchie et al., 2016; Wu et al., 2017; Denil et al., 2017; Hudson and Manning, 2018), we see great promise in synthesizing new techniques by drawing on the full AI toolkit…”
Meta-learning autoencoders for few-shot prediction (2018, background): “This class of problems is important in many areas, for example, learning and predicting physics [8, 26, 25, 3], reinforcement learning of games [15, 10], unsupervised learning of videos [20], and applications such as self-driving, where we cannot enumerate and train with all environments that the…”
Deep Learning: A Critical Appraisal (2018, background): “Recent work by Lerer et al (2016), Watters and colleagues (2017), Tenenbaum and colleagues(Wu, Lu, Kohli, Freeman, & Tenenbaum, 2017) and Davis and myself (Davis, Marcus, & Frazier-Logue, 2017) suggest some competing approaches to how to think about this, within the domain of everyday physical…”
Acquisition of Localization Confidence for Accurate Object Detection (2018, background): “Object detection serves as a prerequisite for a broad set of downstream vision applications, such as instance segmentation [19, 20], human skeleton [27], face recognition [26] and high-level object-based reasoning [30].”
Learning Plannable Representations with Causal InfoGAN (2018, background): “Previous work explored feedforward neural networks for predicting outcomes of physical experiments [22], neural networks for modelling relations between objects [47, 36], and prediction based on physics simulators [4, 48].”
Synthesizing Programs for Images using Reinforced Adversarial Learning (2018, background): “Structured object-attribute based ‘de-rendering’ models have been proposed for interpretation of images (Wu et al., 2017b) and videos (Wu et al., 2017a).”
COPHY: Counterfactual Learning of Physical Dynamics (2019, background): “On similar tasks, (Wu et al., 2017) propose to learn physics by interpreting and reconstructing the visual information stream leading to inverting physics or a graphical engine.”
3D-PhysNet: Learning the Intuitive Physics of Non-Rigid Object Deformations (2018, background): “Recently, [Wu et al., 2017] decouples the future prediction problem by learning an abstract physical representation of the world using a perception network, and using the physical representation as input to a physics engine and and a rendering engine in order to generate visual data, that can be…”
Forest R-CNN: Large-Vocabulary Long-Tailed Object Detection and Instance Segmentation (2020, background): “, person retrieval [42], human-object interaction [37], and visual reasoning [39].”
Visual Grounding of Learned Physical Models (2020, background): “People also have studied ways of reasoning about the physics and learning forward model directly from visual inputs (Finn & Levine, 2017; Babaeizadeh et al., 2018; Hafner et al., 2019; Ha & Schmidhuber, 2018; Wu et al., 2017).”
Analogues of mental simulation and imagination in deep learning (2019, unknown): “Mental simulation is also seen as general, flexible, and compositional, supporting behavior across a wide range of different tasks [[38,47 ,54]], capturing a large body of commonsense knowledge [[38,47 ,54],12], and operating over multiple levels of abstraction [86].”
RepPoints: Point Set Representation for Object Detection (2019, background): “s one of the most fundamental tasks in computer vision, it serves as a key component for many vision applications, including instance segmentation [34], human pose analysis [43], and visual reasoning [47]. The signiﬁcance of the object detection problem together with the rapid development of deep ne”
Unsupervised Discovery of Parts, Structure, and Dynamics (2019, background): “A few very recent papers have proposed to jointly learn a perception module and a dynamics model (Watters et al., 2017; Wu et al., 2017; van Steenkiste et al., 2018).”