Understanding Hand-Object Manipulation with Grasp Types and Object Attributes

By Minjie Cai, Kris M. Kitani, Yoichi Sato. 2016

94 citations recorded by the source corpus.

Our goal is to automate the understanding of natural hand-object manipulation by developing computer visionbased techniques. Our hypothesis is that it is necessary to model the grasp types of hands and the attributes of manipulated objects in order to accurately recognize manipulation actions. Specifically, we focus on recognizing hand grasp types, object attributes and actions from a single image within an unified model. First, we explore the contextual relationship between grasp types and object attributes, and show how that context can be used to boost the recognition of both grasp types and object attributes. Second, we propose to model actions with grasp types and object attributes based on the hypothesis that grasp types and object attributes contain complementary information for characterizing different actions. Our proposed action model outperforms traditional appearance-based models which are not designed to take into account semantic constraints such as grasp types or object attributes. Experiment results on public egocentric activities datasets strongly support our hypothesis.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

In Robotics & Control

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[26] Minjie Cai, Kris M Kitani, and Yoichi Sato.”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “Other works tackled tasks related to EHOI detection proposing hand-centric methods [5, 4, 53].”
Interactive Perception: Leveraging Action in Perception and Perception in Action (2016, background): “An interaction may only be locally applied to the scene (e.g. through pushing or pulling a specific object) or it may affect the scene globally (e.g. shaking a tray with objects standing on it).”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “Understanding the interactions between a camera wearer and the surrounding objects is a fundamental problem in First Person Vision (FPV) [19, 81, 51, 30, 20, 68, 12, 5, 6].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “[34] propose a structured approach where grasp types, object attributes and their contextual relationships are analyzed together.”
Human Hands as Probes for Interactive Object Understanding (2021, background): “This understanding can take different forms: grasp type classification [3,51,63] from a hand-defined taxonomy [15], hand keypoint and pose estimation [17], understanding gestures [18], detecting hands, their states and objects of interaction [55, 56], 3D reconstruction of the hand and the object of…”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, methodology): “Grasps: We identified 34 different grasps following the same taxonomy as in [50], including the most frequently studied ones [8] (i.”
First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, background): “Therefore, understanding hands with computer vision systems has been deeply explored through hand tracking [41, 54], hand pose estimation [18, 21, 22, 27, 29], grasp detection [10, 47], hand gesture recognition [58], multi-view prediction [29], and hand-action classiﬁcation [50].”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, background): “The concept of hand-object interactions is further explored in [20] with the detection of grasps in relation to the shape of objects for modeling actions.”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “Other studies tackled tasks related to EHOI recognition proposing hand-centric methods [4, 1, 82, 52].”
Hands Holding Clues for Object Recognition in Teachable Machines (2019, background): “Prior work on hand-object manipulation [14, 23, 39] argues that people tend to use same or similar hand shapes for a given object shape and function.”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “, learning to model object manipulation [2, 5], how people would grasp an object [38, 52, 11, 7], or how body pose benefits object recognition [8, 19].”