You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions

By Evonne Ng, Donglai Xiang, Hanbyul Joo, Kristen Grauman. 2019

113 citations recorded by the source corpus.

The body pose of a person wearing a camera is of great interest for applications in augmented reality, healthcare, and robotics, yet much of the person's body is out of view for a typical wearable camera. We propose a learning-based approach to estimate the camera wearer's 3D body pose from egocentric video sequences. Our key insight is to leverage interactions with another person---whose body pose we can directly observe---as a signal inherently linked to the body pose of the first-person subject. We show that since interactions between individuals often induce a well-ordered series of back-and-forth responses, it is possible to learn a temporal model of the interlinked poses even though one party is largely out of view. We demonstrate our idea on a variety of domains with dyadic interaction and show the substantial impact on egocentric body pose estimation, which improves the state of the art.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Furthermore, while existing datasets rely largely on graduate students as camera wearers [43,44,66,129,129,138, 168, 179, 194, 210], Ego4D camera wearers are of a much wider demographic, as detailed below.”
Human Hands as Probes for Interactive Object Understanding (2021, background): “[45] use body pose of another person to predict the self-pose in egocentric videos.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Egocentric video understanding The unique viewpoint in egocentric video presents interesting research challenges including action recognition and anticipation [77, 21, 59, 1], daily life summary generation [36, 74], inferring body pose [32, 51], and estimating gaze [37, 31].”