Future Person Localization in First-Person Videos

By Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, Yoichi Sato. 2017

183 citations recorded by the source corpus.

We present a new task that predicts future locations of people observed in first-person videos. Consider a first-person video stream continuously recorded by a wearable camera. Given a short clip of a person that is extracted from the complete stream, we aim to predict that person's location in future frames. To facilitate this future person localization ability, we make the following three key observations: a) First-person videos typically involve significant ego-motion which greatly affects the location of the target person in future frames; b) Scales of the target person act as a salient cue to estimate a perspective effect in first-person videos; c) First-person videos often capture people up-close, making it easier to leverage target poses (e.g., where they look) for predicting their future locations. We incorporate these three observations into a prediction framework with a multi-stream convolution-deconvolution architecture. Experimental results reveal our method to be effective on our new dataset as well as on a public social interaction dataset.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[230] Takuma Yagi, Karttikeya Mangalam, Ryo Yonetani, and Yoichi Sato.”
Peeking Into the Future: Predicting Future Person Activities and Locations in Videos (2019, methodology): “in [33] used person keypoint features with a convolutional neural network to predict future path in first-person videos.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, background): “Given that the first-person vision could better capture people’s intention and interactions, as well as its applicability to AR and robotics [39,43,68], estimating human motions in egocentric videos [49,64,88] worth more attention.”
Ego-Pose Estimation and Forecasting As Real-Time PD Control (2019, background): “Plenty of work has investigated third-person [61, 31, 3, 21, 1, 43, 65] and ﬁrst-person [51] trajectory forecasting, but this line of work only forecasts a person’s future positions instead of poses.”
Core Challenges of Social Robot Navigation: A Survey (2021, background): “Although beyond the scope of our survey, we mention that similar trends can be found in the autonomous driving literature, where behavior prediction is often conditioned on agents’ goal estimates [42, 134, 155, 156, 206, 280, 286] or multiagent interaction modes [210, 239].”
EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, background): “Ego-motion, as the signature of firstperson videos, has also been especially considered in person identification [20, 58] and motion estimation [58, 61].”
Multimodal Motion Prediction with Stacked Transformers (2021, methodology): “CNN Encoder-Decoder is proposed in [33] to extract features from vehicles’ past positions and directions and directly regress the future positions.”
From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting (2020, background): “[46] tackles motion forecasting in ego-centric views and develops a system that exploits subtle cues like body pose and gaze along with camera wearer’s ego-motion for other agent’s future location prediction.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, background): “For instance, the authors of [76] evaluated their approach on their own private dataset of only about 4.”