Ego-Pose Estimation and Forecasting As Real-Time PD Control

By Ye Yuan, Kris M. Kitani. 2019

151 citations recorded by the source corpus.

We propose the use of a proportional-derivative (PD) control based policy learned via reinforcement learning (RL) to estimate and forecast 3D human pose from egocentric videos. The method learns directly from unsegmented egocentric videos and motion capture data consisting of various complex human motions (e.g., crouching, hopping, bending, and motion transitions). We propose a video-conditioned recurrent control technique to forecast physically-valid and stable future motions of arbitrary length. We also introduce a value function based fail-safe mechanism which enables our method to run as a single pass algorithm over the video data. Experiments with both controlled and in-the-wild data show that our approach outperforms previous art in both quantitative metrics and visual quality of the motions, and is also robust enough to transfer directly to real-world scenarios. Additionally, our time analysis shows that the combined use of our pose estimation and forecasting can run at 30 FPS, making it suitable for real-time applications.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “For 3D human pose estimation, recent work has adopted physics-based trajectory optimization [92, 65, 73, 72] and motion imitation [87, 88, 28, 91, 86, 47, 48] to model human dynamics.”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, background): “However, UHC uses residual force control [50], which applies a non-physical force at the root of the humanoid to help balance.”
AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting (2021, methodology): “Prior work has also used RNNs to model the temporal dynamics of 3D human pose [11, 58, 60].”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, unknown): “And, in total, it offers ∼ 14M frames of 3D GT and pseudo-GT combined.”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “While this makes state-of-the-art third-person pose methods poorly suited [63, 45, 58, 34, 24, 10], recent work suggests that an ego-video stream nonetheless offers implicit cues for first-person body pose [26, 69, 70].”
DLow: Diversifying Latent Flows for Diverse Human Motion Prediction (2020, background): “Most previous work takes a deterministic approach to modeling human motion and regress a single future motion from past 3D poses [17, 34, 9, 45, 18, 52, 57, 13, 22, 1, 69, 51] or video frames [10, 77, 75].”