You-Do, I-Learn: Egocentric unsupervised discovery of objects and their modes of interaction towards video-based guidance

By Dima Damen, Teesid Leelasawassuk, Walterio W. Mayol-Cuevas. 2015

57 citations recorded by the source corpus.

Discovering task-relevant objects from egocentric video sequences of multiple users, using appearance, position, motion and attention features.Distinguishing different ways in which a task-relevant object has been used.Automatically extracting usage snippets, to be used for video-based guidance.Tested on a variety of daily tasks such as initialising a printer, preparing a coffee and setting up a gym machine. Display Omitted This paper presents an unsupervised approach towards automatically extracting video-based guidance on object usage, from egocentric video and wearable gaze tracking, collected from multiple users while performing tasks. The approach (i)źdiscovers task relevant objects, (ii) builds a model for each, (iii)źdistinguishes different ways in which each discovered object has been used and (iv)źdiscovers the dependencies between object interactions. The work investigates using appearance, position, motion and attention, and presents results using each and a combination of relevant features. Moreover, an online scalable approach is presented and is compared to offline results. The paper proposes a method for selecting a suitable video guide to be displayed to a novice user indicating how to use an object, purely triggered by the user's gaze. The potential assistive mode can also recommend an object to be used next based on the learnt sequence of object interactions. The approach was tested on a variety of daily tasks such as initialising a printer, preparing a coffee and setting up a gym machine.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Human-Computer Interaction

Citations in the corpus

Is First Person Vision Challenging for Object Tracking? (2020, unknown): “Since many FPV tasks such as object interaction [20] and early action recognition [34], or action anticipation [19], require real-time computation, we evaluated the ability of trackers to provide their object localization in such a setting.”
Human Hands as Probes for Interactive Object Understanding (2021, background): “…of learning a) where and how to grasp [9, 20, 25, 26, 28, 33, 34, 37, 41, 49], b) state classifiers [24], c) interaction hotspots [14, 40, 42, 59], d) spatial priors for action sites [44], e) object articulation modes [11,36], f) reward functions [27,29,48,50], g) functional correspondences [32].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “, learning to model object manipulation [2, 5], how people would grasp an object [38, 52, 11, 7], or how body pose benefits object recognition [8, 19].”
Seeing with Humans: Gaze-Assisted Neural Image Captioning (2016, methodology): “There have been several methods proposed for robust gaze-guided object segmentation [12], [26], [27], and based on a similar assumption ﬁxation information is also used for localizing important objects in videos [5], [28], [29].”
Computer Vision and Image Understanding Computer Vision for Assistive Technologies (unknown): “…has been also used to superimpose digital content on physical objects creating augmented reality that can be exploited to guide on object usage ( Damen et al., 2016 ), even through a robotic manipulator ( Jiang et al., 2016 ), or to increase the selective and sustained attention of children…”