Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Active Tasks

By Alexander Sax, Bradley Emi, Amir Zamir, Leonidas J. Guibas, Silvio Savarese, Jitendra Malik. 2018

16 citations recorded by the source corpus.

One of the ultimate promises of computer vision is to help robotic agents perform active tasks, like delivering packages or doing household chores. However, the conventional approach to solving "vision" is to define a set of offline recognition problems (e.g. object detection) and solve those first. This approach faces a challenge from the recent rise of Deep Reinforcement Learning frameworks that learn active tasks from scratch using images as input. This poses a set of fundamental questions: what is the role of computer vision if everything can be learned from scratch? Could intermediate vision tasks actually be useful for performing arbitrary downstream active tasks? We show that proper use of mid-level perception confers significant advantages over training from scratch. We implement a perception module as a set of mid-level visual representations and demonstrate that learning active tasks with mid-level features is significantly more sample-efficient than scratch and able to generalize in situations where the from-scratch approach fails. However, we show that realizing these gains requires careful selection of the particular mid-level features for each downstream task. Finally, we put forth a simple and efficient perception module based on the results of our study, which can be adopted as a rather generic perception module for active frameworks.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, methodology): “We use the training hyperparameters from [59], who use a rollout of size 512 and optimize using Adam [32].”
Adaptive Auxiliary Task Weighting for Reinforcement Learning (2019, methodology): “For example, visual features can be taken from pre-trained features of other tasks, such as image classification or depth estimation [8].”
ShapeMask: Learning to Segment Novel Objects by Refining Shape Priors (2019, methodology): “We propose to improve generalization in instance segmentation (Figure 1) by introducing intermediate representations [29, 43, 11], and instance-specific grouping-based learning [40, 23].”
SplitNet: Sim2Sim and Task2Task Transfer for Embodied Visual Navigation (2019, background): “These techniques often leverage the additional supervision and auxiliary tasks given by the simulators to bootstrap their learning [20, 29].”