Joint Discovery of Object States and Manipulation Actions

By Jean-Baptiste Alayrac, Josef Sivic, Ivan Laptev, Simon Lacoste-Julien. 2017

86 citations recorded by the source corpus.

Many human activities involve object manipulations aiming to modify the object state. Examples of common state changes include full/empty bottle, open/closed door, and attached/detached car wheel. In this work, we seek to automatically discover the states of objects and the associated manipulation actions. Given a set of videos for a particular task, we propose a joint model that learns to identify object states and to localize state-modifying actions. Our model is formulated as a discriminative clustering cost with constraints. We assume a consistent temporal order for the changes in object states and manipulation actions, and introduce new optimization techniques to learn model parameters without additional supervision. We demonstrate successful discovery of seven manipulation actions and corresponding object states on a new dataset of videos depicting real-life object manipulations. We show that our joint formulation results in an improvement of object state discovery by action recognition and vice versa.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips (2019, background): “Instructional videos are rising in popularity in the context of learning steps of complex tasks [2, 16, 41, 42, 46, 68], visual-linguistic reference resolution [17, 18], action segmentation in long untrimmed videos [66] and joint learning of object states and actions [3].”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “Prior works have used web videos+ASR to build weakly-supervised object detectors [75], action detectors/classifiers [108, 5, 51, 72], instruction aligners [65, 4, 15], video captioners [85, 38, 74, 89], and visual reference resolvers [41].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Relation to existing tasks Limited prior work considers object state change in photos [102,164] or video [8,68,242]; Ego4D is the first video benchmark dedicated to the task of understanding object state changes.”
Cross-Task Weakly Supervised Learning From Instructional Videos (2019, background): “As it is often the case in weakly supervised video learning [2, 18, 29], these assumptions constrain the search for when steps occur, helping tackle a combinatorial search space.”
Action Modifiers: Learning From Adverbs in Instructional Videos (2019, methodology): “Narrations are used to learn steps of complex tasks [1, 18, 30, 42, 45, 62], and more recently for video retrieval [33], visual grounding [17, 19], action segmentation [60] and learning actions through object state changes [2, 14].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “Affordances Affordances are often focused on objects, where the goal is to anticipate how an object could be used—for example learning to model object manipulation [1, 4] or how people would grasp an object [37, 51,”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “We exclude egocentric datasets that focus on inter-person interactions [29], [30], [31] as well as instructional videos [32], [33], [34], [35], as”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, background): “In computer vision, visual affordances are explored in various forms: predicting where to grasp an object from images and video [31, 32, 64, 38, 19, 62, 15, 5], inferring how people might use a space [48, 39] or tool [65], and priors for human body poses [26, 52, 58, 17].”
Beyond Instructional Videos: Probing for More Diverse Visual-Textual Grounding on YouTube (2020, unknown): “…tasks (Gupta et al., 2017; Huang et al., 2017; Huang* et al., 2018; Moriya et al., 2019), e.g., action detection/classiﬁcation (Yu et al., 2014; Alayrac et al., 2017; Chang et al., 2019; Kuehne et al., 2019), segmentation/captioning (Sener et al., 2015), and instruction alignment (Malmaud et…”
Finding "It": Weakly-Supervised Reference-Aware Visual Grounding in Instructional Videos (2018, methodology): “This use of transcription as supervision has been utilized in several contexts, such as action detection [55], object state discovery [2], entity reference [16], and procedural knowledge discovery [1, 29, 45].”
Videos as Space-Time Region Graphs (2018, background): “Our work is also related to video recognition with object cues [72,73,74] and object graph models [75,76,77,78].”
PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Event discovery tasks such as action recognition and temporal action segmentation [17]–[20], state understanding [21], video prediction [22] and video summarization / captioning [15], [23] study recognition of human actions in video sequences.”
Unsupervised Visual-Linguistic Reference Resolution in Instructional Videos (2017, background): “te temporal localization [35,62]. As signiﬁcant progress has been made on classifying temporally trimmed video clips, recent works aim to obtain the procedural knowledge from the instructional videos [2,3,52]. Our goal of reference resolution in instructional videos is a step further as it requires ”
Grounded Human-Object Interaction Hotspots From Video (2018, methodology): “For instructional video, methods explore ways to extract object states [1], modes of object interaction [8], interaction regions [12], or the anticipated trajectory of an object given a person’s skeleton pose [31].”