Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos

By Shao-Wei Liu, Subarna Tripathi, Somdeb Majumdar, Xiaolong Wang. 2022

144 citations recorded by the source corpus.

We propose to forecast future hand-object interactions given an egocentric video. Instead of predicting action labels or pixels, we directly predict the hand motion trajectory and the future contact points on the next active object (i.e., interaction hotspots). This relatively low-dimensional representation provides a con-crete description of future interactions. To tackle this task, we first provide an automatic way to collect trajectory and hotspots labels on large-scale data. We then use this data to train an Object-Centric Transformer (OCT) model for prediction. Our model performs hand and object interaction reasoning via the self-attention mechanism in Transformers. OCT also provides a probabilistic framework to sample the future trajectory and hotspots to handle uncertainty in prediction. We perform experi-ments on the Epic-Kitchens-55, Epic-Kitchens-100 and EGTEA Gaze+ datasets, and show that OCT significantly outperforms state-of the-art approaches by a large margin. Project page is available at https://stevenlsw.github.io/hoi-forecast.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Deepak Pathak
3. Amlan Kar
4. Dandan Shan
5. Unnat Jain
6. Dima Damen
7. Jian Ma
8. David F. Fouhey

In Computer Vision and Pattern Recognition

1. Jitendra Malik
2. Luc Van Gool
3. Trevor Darrell
4. Sanja Fidler
5. Kristen Grauman
6. Abhinav Kumar Gupta
7. Dacheng Tao
8. Andrea Vedaldi

In Human-Computer Interaction

1. Dandan Shan
2. Dima Damen
3. David F. Fouhey
4. Amlan Kar
5. Sanja Fidler
6. Jitendra Malik

Citations in the corpus

• EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations (2022, background): “VISOR is also related to efforts that capture hand-object relations, including for identifying active objects [35, 4], next active objects [33, 25], object articulation [44] and grasping [18, 39, 38, 40].”
• Affordances from Human Videos as a Versatile Representation for Robotics (2023, methodology): “• stevenlsw/hoi-forecast for Skin segmentation code and HOI baseline [66].”