Blocks World Revisited: Image Understanding Using Qualitative Geometry and Mechanics

By Abhinav Kumar Gupta, Alexei A. Efros, Martial Hebert. 2010

394 citations recorded by the source corpus.

Since most current scene understanding approaches operate either on the 2D image or using a surface-based representation, they do not allow reasoning about the physical constraints within the 3D scene. Inspired by the "Blocks World" work in the 1960's, we present a qualitative physical representation of an outdoor scene where objects have volume and mass, and relationships describe 3D structure and mechanical configurations. Our representation allows us to apply powerful global geometric constraints between 3D volumes as well as the laws of statics in a qualitative manner. We also present a novel iterative "interpretation-by-synthesis" approach where, starting from an empty ground plane, we progressively "build up" a physically-plausible 3D interpretation of the image. For surface layout estimation, our method demonstrates an improvement in performance over the state-of-the-art [9]. But more importantly, our approach automatically generates 3D parse graphs which describe qualitative geometric and mechanical properties of objects and relationships between objects within an image.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Category-specific object reconstruction from a single image (2014, neutral): “Polyhedral models [18, 40], which trace back to the early work of Roberts [29], and CAD models [25, 31] provide crude approximations of shape and given a set of point correspondences can be quite effective for determining instance viewpoints.”
Deep convolutional neural fields for depth estimation from a single image (2014, background): “, box models, to infer the spatial layout of a room [2,3] or outdoor scenes [4].”
"What Happens If..." Learning to Predict the Effect of Forces in Images (2016, background): “, [25,9,40]) are passive in that they are focused on predicting the scene structure, the objects, and their attributes and relations.”
Picture: A probabilistic programming language for scene perception (2015, methodology): “Procedural graphics techniques [2, 10] would support far more complex object and scene models [50, 14, 7, 15].”
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields (2015, background): “…computer vision systems to do so since no reliable cues can be exploited, such as temporal information in videos, stereo correspondences, etc. Previous work mainly focuses on enforcing geometric assumptions, e.g., box models, to infer the spatial layout of a room [4], [5] or outdoor scenes [6].”
To Fall Or Not To Fall: A Visual Approach to Physical Stability Prediction (2016, positive): “By including the additional clue from physical constraints into the inference mechanism, mostly from the support event, it has further improved results in segmentation of surfaces [12], scenes [26] from image data, and object segmentation in 3D point cloud data [31].”
Bringing Semantics into Focus Using Visual Abstraction (2013, background): “Depth ordering: The relative 3D location of objects can provide useful information for their detection [13, 15].”
Data-Driven 3D Primitives for Single Image Understanding (2013, neutral): “The most commonly used primitives include oriented 3D surfaces [14, 19, 23, 31] represented as segments in the image, or volumetric primitives such as blocks [11] and cuboids [18, 33].”
Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs (2015, methodology): “When specific scene dependant knowledge is available, depth estimation or 3D reconstruction from single images can be achieved by utilizing geometric assumptions such as the “Blocks World” model [5], the “Origami World” model [6], shape from shading [7] and repetition of structures [8].”
3DNN: 3D Nearest Neighbor (2014, background): “…systems rely implicitly on sifting through a collection of hypotheses (iterative reﬁne-ment(Hoiemetal.2008),sampling(Peroetal.2011),explicit search (Gupta et al. 2010), structured prediction (Hedau et al. 2009; Lee et al. 2010), matching and ranking mechanisms such as the ones we propose provide…”
Estimating the 3D Layout of Indoor Scenes and Its Clutter from Depth Sensors (2013, background): “Early approaches to semantic scene understanding in the outdoor setting focused on producing qualitative 3D parses [19, 13, 6], ground plane estimates [30] or parsing facades [27, 17].”
Robust real-time visual odometry for dense RGB-D mapping (2013, methodology): “able surfaces by reasoning on the location and shape of furniture, [20, 19] reason about the 3D geometry of the room and objects, while [9] focuses on interpreting the scene in a human-centric perspective.”
Discrete-Continuous Depth Estimation from a Single Image (2014, methodology): “The same simple geometric prior, blocks world, was exploited in outdoor scenes [7].”
Designing deep networks for surface normal estimation (2014, background): “However, decades of research have shown that the output space of this task is governed by powerful physical constraints and researchers have exploited these constraints from the very beginning of computer vision [27] through the line-labeling era [17, 2, 19] all the way to recent investigations [14,”
Discriminatively Trained Dense Surface Normal Estimation (2014, background): “In [14] Gupta et al. extracted a qualitative physical representation of an outdoor scene by reasoning about the pairwise depth relations (and thus also not via absolute depth); though their model is approximated to consist of blocks only.”
Understanding Indoor Scenes Using 3D Geometric Phrases (2013, background): “[11] incorporated the concept of physical gravity and reasoned about object supports.”