Robot Learning Manipulation Action Plans by "Watching" Unconstrained Videos from the World Wide Web

By Yezhou Yang, Yi Li, Cornelia Fermüller, Yiannis Aloimonos. 2015

230 citations recorded by the source corpus.

In order to advance action generation and creation in robots beyond simple learned schemas we need computational tools that allow us to automatically interpret and represent human actions. This paper presents a system that learns manipulation action plans by processing unconstrained videos from the World Wide Web. Its goal is to robustly generate the sequence of atomic actions of seen longer actions in video in order to acquire knowledge for robots. The lower level of the system consists of two convolutional neural network (CNN) based recognition modules, one for classifying the hand grasp type and the other for object recognition. The higher level is a probabilistic manipulation action grammar based parsing module that aims at generating visual sentences for robot manipulation. Experiments conducted on a publicly available unconstrained video dataset show that the system is able to learn manipulation actions by ``watching'' unconstrained videos with high accuracy.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

A self-organizing neural network architecture for learning human-object interactions (2017, methodology): “Yang et al. [31] proposed a method for learning actions comprising object manipulation from demonstrating videos.”
Combining self-supervised learning and imitation for vision-based rope manipulation (2017, background): “Many past works have proposed good solutions to learning from visual demonstrations [29], [30], [4].”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “problem by hand, for example, by manually specifying how human grasp poses correspond to robot grasps [20] or by manually defining how human activities or commands translate into robot actions [58, 23, 30, 37, 40].”
VirtualHome: Simulating Household Activities Via Programs (2018, methodology): “In [26], the authors detect objects and actions in cooking videos and generate an “action plan” using a probabilistic grammar.”
Zero-Shot Visual Imitation (2018, background): “A large body of work (Breazeal & Scassellati, 2002; Dillmann, 2004; Ikeuchi & Suehiro, 1994; Kuniyoshi et al., 1989; 1994; Yang et al., 2015) has sought to capture prior knowledge by manually pre-defining the state that must be ∗Denotes equal contribution. ar X iv :1 80 4.”
From Lifestyle Vlogs to Everyday Interactions (2017, background): “This has obvious applications in tasks like future prediction [41], intuitive physics [44], imitation learning from videos [45], and grasp analysis [18].”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, methodology): “Some works have studied this setting using explicit pose and object detection (Kuniyoshi et al., 1994; Lee et al., 2013; Miyamoto et al., 1996; Ramirez-Amaro et al., 2017; Yang et al., 2015), essentially resolving the correspondence problem (Nehaniv & Dautenhahn, 2002) by instrumenting paired data c”
Revisiting active perception (2016, background): “…of the scene and a subsequent semantic analysis (Wörgötter et al. 2012; Summers-Stay et al. 2013; Pastra and Aloimonos 2012; Manikonda et al. 1999; Dantam and Stilman 2013; Fainekos et al. 2005; Aksoy et al. 2011; Worch et al. 2015; Yang et al. 2015; Coates et al. 2008; Maitin-Shepard et al. 2010).”
Trends and challenges in robot manipulation (2019, background): “Alternatively, robotsmay learn from image data and videos available on the internet (84) or from demonstration by a live expert, usually a human.”
Deep learning in robotics: a review of recent research (2017, background): “Yang, Li, and Fermüller [79] trained a DNN to recognize 48 common kitchen objects and classify human grasps on them from 88 YouTube cooking videos.”
PERCEPTUAL GOAL SPECIFICATIONS FOR REINFORCEMENT LEARNING (2017, background): “In one approach, a robot is trained to cook by showing it cooking videos [65].”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “Other approaches, such as the Roboearth project [409], seek to acquire a library of skills from information available online, such as natural language and video data, which can then be shared with other robots [409, 430].”
Understanding Hand-Object Manipulation with Grasp Types and Object Attributes (2016, background): “It can also facilitate robotic task planning by studying the relationship between different components (grasps, objects and actions) in performing a manipulation task [11, 33].”