Shaping embodied agent behavior with activity-context priors from egocentric video
By Tushar Nagarajan, Kristen Grauman. 2021
23 citations recorded by the source corpus.
Complex physical tasks entail a sequence of object interactions, each with its own preconditions -- which can be difficult for robotic agents to learn efficiently solely through their own experience. We introduce an approach to discover activity-context priors from in-the-wild egocentric video captured with human worn cameras. For a given object, an activity-context prior represents the set of other compatible objects that are required for activities to succeed (e.g., a knife and cutting board brought together with a tomato are conducive to cutting). We encode our video-based prior as an auxiliary reward function that encourages an agent to bring compatible objects together before attempting an interaction. In this way, our model translates everyday human experience into embodied agent skills. We demonstrate our idea using egocentric EPIC-Kitchens video of people performing unscripted kitchen activities to benefit virtual household robotic agents performing various complex tasks in AI2-iTHOR, significantly accelerating agent learning. Project page: http://vision.cs.utexas.edu/projects/ego-rewards/
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Saurabh Gupta
- Lingzhi Zhang
- Sahil Modi
- Shenghao Zhou
- Mohit Goyal
- Simon Stent
- Rishabh Goyal
- Jianbo Shi
In Computer Vision and Pattern Recognition
- Jitendra Malik
- Luc Van Gool
- Saurabh Gupta
- Trevor Darrell
- Dacheng Tao
- Abhinav Kumar Gupta
- Xiaogang Wang
- Andrew Zisserman
In Robotics & Control
Citations in the corpus
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “Other directions of hand-object interaction involve using hands as probes for object understanding [15], affordance hotspot reasoning [47,49], or even leveraging visual hand-object understanding for robotic learning [41,42,48].”
- Human Hands as Probes for Interactive Object Understanding (2021, background): “[43] works in a similar setting but focuses on learning activity-context priors.”