Does computer vision matter for action?

By Brady Zhou, Philipp Krahenbuhl, Vladlen Koltun. 2019

119 citations recorded by the source corpus.

Controlled experiments indicate that explicit intermediate representations help action. Controlled experiments indicate that explicit intermediate representations help action.

Subfields: Cognitive Neuroscience, Social Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Social Psychology

Citations in the corpus

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “DEPTH is among the most useful intermediate representations for action in physical environments [1].”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, result): “As shown in Figure 7, our final scores on these environments significantly exceed those reported in prior work (Dosovitskiy & Koltun, 2017; Zhou et al., 2019).”
Masked Visual Pre-training for Motor Control (2022, background): “Zhou et al. (2019) show the effectiveness on visual representations in driving settings.”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “original RGB image contains all the information present in the predicted images, explicitly extracting such pre-defined representations and using them as (additional) inputs has been shown to improve model robustness [45], [47].”
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding (2021, background): “While more recent large-scale sets [2,30] that cover adverse conditions, such as Waymo Open [42] and nuScenes [3], include bounding boxes, they still lack dense pixel-level semantic annotations, which are vital for real-world autonomous agents [63].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, background): “Zhang et al. [39] and Zhou et al. [59] utilize a motion planner to generate a series of waypoints that describe the future trajectory.”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “[65] demonstrate the effectiveness of semantics and depth as explicit intermediate representations for driving.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Robotics is a particularly inspiring domain for sim2real research, and here it has been again found that more abstract representations than RGB, such as segmentations [50, 94], can improve performance.”
Learning by Cheating (2019, background): “First, the privileged agent operates on a compact intermediate representation of the environment, and can thus learn faster and generalize better [24, 25].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Behl et al. [43] and Zhou et al. [44] demonstrate the effectiveness of semantics and depth as explicit intermediate representations for driving.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “Early fusion combines sensory inputs before feeding them into shared feature extractors, where concatenation is a common way for fusion [32], [135], [136], [137], [138].”
UniDepth: Universal Monocular Metric Depth Estimation (2024, background): “The precise pixel-wise depth estimation is crucial to understanding the geometric scene structure, with applications in 3D modeling [9], robotics [10, 52], and autonomous vehicles [31, 41].”
Learning high-speed flight in the wild (2021, result): “This is inline with previous ﬁndings [28, 50] showing that intermediate image representations have a strong positive effect on performance when sufﬁciently informative for the task.”