Who Let the Dogs Out? Modeling Dog Behavior from Visual Data

By Kiana Ehsani, Hessam Bagherinezhad, Joseph Redmon, Roozbeh Mottaghi, Ali Farhadi. 2018

63 citations recorded by the source corpus.

We study the task of directly modelling a visually intelligent agent. Computer vision typically focuses on solving various subtasks related to visual intelligence. We depart from this standard approach to computer vision; instead we directly model a visually intelligent agent. Our model takes visual information as input and directly predicts the actions of the agent. Toward this end we introduce DECADE, a dataset of ego-centric videos from a dog's perspective as well as her corresponding movements. Using this data we model how the dog acts and how the dog plans her movements. We show under a variety of metrics that given just visual input we can successfully model this intelligent agent in many situations. Moreover, the representation learned by our model encodes distinct information compared to representations trained on image classification, and our learned representation can generalize to other domains. In particular, we show strong results on the task of walkable surface estimation and scene classification by using this dog modelling task as representation learning.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yejin Choi
2. Rowan Zellers
3. Luc Van Gool
4. Yonatan Bisk
5. Li Fei-Fei
6. Ali Farhadi
7. Dacheng Tao
8. Jianfeng Gao

Citations in the corpus

• SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, background): “The question of ‘what will happen next?’ has also been studied in photo albums (Huang et al., 2016), videos of team sports, (Felsen et al., 2017) and egocentric dog videos (Ehsani et al., 2018).”
• From Recognition to Cognition: Visual Commonsense Reasoning (2018, neutral): “Commonsense Reasoning Our task unifies work involving reasoning about commonsense phenomena, such as physics [54, 84], social interactions [2, 77, 12, 27], procedure understanding [91, 3] and predicting what might happen next in a video [74, 17, 92, 78, 18, 64, 85].”
• PlaTe: Visually-Grounded Planning With Transformers in Procedural Tasks (2021, background): “Recent works have shown that deep networks can learn to plan directly from pixel observations in domains such as tabletop manipulation [11], [12], navigation in VizDoom [13], and locomotion in joint space [14].”