Seeing Invisible Poses: Estimating 3D Body Pose from Egocentric Video

By Hao Jiang, Kristen Grauman. 2016

108 citations recorded by the source corpus.

Understanding the camera wearers activity is central to egocentric vision, yet one key facet of that activity is inherently invisible to the camera—the wearers body pose. Prior work focuses on estimating the pose of hands and arms when they come into view, but this 1) gives an incomplete view of the full body posture, and 2) prevents any pose estimate at all in many frames, since the hands are only visible in a fraction of daily life activities. We propose to infer the invisible pose of a person behind the egocentric camera. Given a single video, our efficient learning-based approach returns the full body 3D joint positions for each frame. Our method exploits cues from the dynamic motion signatures of the surrounding scene—which change predictably as a function of body pose—as well as static scene structures that reveal the viewpoint (e.g., sitting vs. standing). We further introduce a novel energy minimization scheme to infer the pose sequence. It uses soft predictions of the poses per time instant together with a non-parametric model of human pose dynamics over longer windows. Our method outperforms an array of possible alternatives, including typical deep learning approaches for direct pose regression from images.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “…challenges, such as human-object interactions [26,46,163], activity recognition [110,139,243], anticipation [4,75,86,144,205], video summarization [48, 129,131,147,148,232], detecting hands [16,134], parsing social interactions [66,168,231], and inferring the camera wearer’s body pose [107].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, methodology): “gaze estimation [10], [12], [13], hand analysis [72], [73], [74], [75], [76], pose estimation [14], [15], physiological parameter estimation [77], [78], user identification [79], [80], [81] and video summarization [16], [17], [18].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Egocentric video understanding The unique viewpoint in egocentric video presents interesting research challenges including action recognition and anticipation [77, 21, 59, 1], daily life summary generation [36, 74], inferring body pose [32, 51], and estimating gaze [37, 31].”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, methodology): “• PathPose: an adaptation of a CRF-based method [17].”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, methodology): “While this makes state-of-the-art third-person pose methods poorly suited [63, 45, 58, 34, 24, 10], recent work suggests that an ego-video stream nonetheless offers implicit cues for first-person body pose [26, 69, 70].”
Who Let the Dogs Out? Modeling Dog Behavior from Visual Data (2018, background): “[13] estimates the 3D joint position of unseen body joints using ego-centric videos.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “…to encourage efﬁcient RL. Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [69, 61, 8, 19], including analyzing hand-object interactions [55, 2, 60, 7, 24] and egocentric video understanding [34, 21, 37, 31, 13].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “And, in total, it offers ∼ 14M frames of 3D GT and pseudo-GT combined.”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “This special viewpoint offers an array of interesting challenges, such as detecting gaze [40, 29], monitoring human-object interactions [4, 6, 51], creating daily life activity summaries [44, 39, 70, 43], or inferring the camera wearer’s identity or body pose [28, 33].”