Learning the semantics of object–action relations by observation

By Eren Erdal Aksoy, Alexey Abramov, Johannes Dörr, KeJun Ning, Babette K. Dellen, Florentin Wörgötter. 2011

196 citations recorded by the source corpus.

Recognizing manipulations performed by a human and the transfer and execution of this by a robot is a difficult problem. We address this in the current study by introducing a novel representation of the relations between objects at decisive time points during a manipulation. Thereby, we encode the essential changes in a visual scenery in a condensed way such that a robot can recognize and learn a manipulation without prior object knowledge. To achieve this we continuously track image segments in the video and construct a dynamic graph sequence. Topological transitions of those graphs occur whenever a spatial relation between some segments has changed in a discontinuous way and these moments are stored in a transition matrix called the semantic event chain (SEC). We demonstrate that these time points are highly descriptive for distinguishing between different manipulations. Employing simple sub-string search algorithms, SECs can be compared and type-similar manipulations can be recognized with high confidence. As the approach is generic, statistical learning can be used to find the archetypal SEC of a given manipulation class. The performance of the algorithm is demonstrated on a set of real videos showing hands manipulating various objects and performing different actions. In experiments with a robotic arm, we show that the SEC can be learned by observing human manipulations, transferred to a new scenario, and then reproduced by the machine.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning human activities and object affordances from RGB-D videos (2012, background): “There is some recent work in interpreting human actions and interaction with objects (Lopes and Santos-Victor, 2005; Saxena et al., 2008; Aksoy et al., 2011; Konidaris et al., 2012; Li et al., 2012) in context of learning to perform actions from demonstrations.”
Revisiting active perception (2016, methodology): “Such knowledge allows a linguistic treatment of action, since onewould be able using suchmodels to create a parsing of the scene and a subsequent semantic analysis (Wörgötter et al. 2012; Summers-Stay et al. 2013; Pastra and Aloimonos 2012; Manikonda et al. 1999; Dantam and Stilman 2013; Fainekos et”
Autonomously constructing hierarchical task networks for planning and human-robot collaboration (2016, methodology): “We use Semantic Event Chains [31] to define the observation sets for each action within the HTN, with a naı̈ve Bayes assumption of observation independence.”
A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms (2019, background): “One common approach is to segment by salient sensory events defined by haptic, tactile, audio, or visual cues [182, 365, 113, 7].”
Robot Learning Manipulation Action Plans by "Watching" Unconstrained Videos from the World Wide Web (2015, background): “Most work on learning from demonstrations in robotics has been conducted in fully controlled lab environments (Aksoy et al. 2011).”
Manipulation action tree bank: A knowledge resource for humanoids (2014, background): “A visual semantic graph (VSG), proposed in the work of [24], provides a plausible formalism for semantic activity representation.”
Bootstrapping humanoid robot skills by extracting semantic representations of human-like activities from virtual reality (2014, background): “The definition and some examples of the motions and object properties are further explained in [5].”