Discovering states and transformations in image collections

By Phillip Isola, Joseph J. Lim, Edward H. Adelson. 2015

329 citations recorded by the source corpus.

Objects in visual scenes come in a rich variety of transformed states. A few classes of transformation have been heavily studied in computer vision: mostly simple, parametric changes in color and geometry. However, transformations in the physical world occur in many more flavors, and they come with semantic meaning: e.g., bending, folding, aging, etc. The transformations an object can undergo tell us about its physical and functional properties. In this paper, we introduce a dataset of objects, scenes, and materials, each of which is found in a variety of transformed states. Given a novel collection of images, we show how to explain the collection in terms of the states and transformations it depicts. Our system works by generalizing across object classes: states and transformations learned on one set of objects are used to interpret the image collection for an entirely new object class.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations (2016, background): “objects (Isola et al. 2015) and understand their interactions within a scene (Sadeghi and Farhadi 2011).”
Joint Discovery of Object States and Manipulation Actions (2017, background): “[22] discover object states and transformations between them by analyzing large collections of still images downloaded from the Internet.”
Generating Videos with Scene Dynamics (2016, background): “Conceptually, our work is related to studies into fundamental roles of time in computer vision [30, 12, 2, 7, 24].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Relation to existing tasks Limited prior work considers object state change in photos [102,164] or video [8,68,242]; Ego4D is the first video benchmark dedicated to the task of understanding object state changes.”
TGIF-QA: Toward Spatio-Temporal Reasoning in Visual Question Answering (2017, background): “, repetition counting [21] and state transitions [15], intending to serve as a bridge between video understanding and video VQA.”
Learning Temporal Transformations from Time-Lapse Videos (2016, background): “Most relevant to our work, given a set of object transformation terms, such as ripe vs unripe, [7] learns visual classiﬁcation and regression models for object transformations from a collection of photos.”
Automatic Understanding of Image and Video Advertisements (2017, background): “There is initial work in understanding physical forces [53, 52, 86] and object transformations [26], but this work is still in its infancy and is not sufficient for full physical process understanding, as we need for many ads.”
Human Hands as Probes for Interactive Object Understanding (2021, methodology): “no further self-supervised pretraining), b) non-temporal self-supervision via SimCLR [6], c) an alternate temporal self-supervision method (Time Contrastive Networks, TCN [53]), and d) semantic supervision from action classification on EPIC-KITCHENS and state classification on MIT States dataset [24”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Here the robot may use passive and interactive perception to estimate object property values [119, 168, 242, 401].”
From Red Wine to Red Tomato: Composition with Context (2017, methodology): “We study compositionality in visual recognition in the context of two well known problems - objects and attributes [13, 28, 29, 46, 48], and subject-verb-object (SVO) phrases [24, 39, 53, 57].”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, background): “All prior works that learn attributes of objects from images [7, 20, 34, 36, 37] utilize fully annotated datasets, where the object the attributes relate to is the only object of interest in the image.”
Video Question Answering with Spatio-Temporal Reasoning (2019, methodology): “Our design of video VQA tasks is inspired by existing work in video understanding, e.g., repetition counting (Levy and Wolf 2015) and state transitions (Isola et al. 2015), intending to serve as a bridge between video understanding and video VQA.”
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models (2021, background): “MIT-States [18] contains images of entities in different states each labelled with one noun and one adjective.”
Deep Mixture of Experts via Shallow Embedding (2018, methodology): “generalized zero-shot learning setting proposed by Xian et al. [46]. We also provide results on the more challenging visual attributes composition task proposed by Misra et al. [32] on both MITStates [23] and StanfordVRD [29] datasets. Our model surpasses the state-of-the-art methods by a signiﬁcant”
Understanding Intra-Class Knowledge Inside CNN (2015, background): “[9] focuses on physical state transformations, and in addition to discovering states it also studies state pairs that define a transformation.”
TAFE-Net: Task-Aware Feature Embeddings for Low Shot Learning (2019, background): “Note that changing the task produces different embeddings for the same data. ments on two datasets: MITStates [21] (image samples in Figure 5) and the modiﬁed StanfordVRD [29] (image samples in Figure 4).”