Vision Transformers for Dense Prediction

By René Ranftl, Alexey Bochkovskiy, Vladlen Koltun. 2021

2,887 citations recorded by the source corpus.

We introduce dense prediction transformers, an architecture that leverages vision transformers in place of convolutional networks as a backbone for dense prediction tasks. We assemble tokens from various stages of the vision transformer into image-like representations at various resolutions and progressively combine them into full-resolution predictions using a convolutional decoder. The transformer backbone processes representations at a constant and relatively high resolution and has a global receptive field at every stage. These properties allow the dense prediction transformer to provide finer-grained and more globally coherent predictions when compared to fully-convolutional networks. Our experiments show that this architecture yields substantial improvements on dense prediction tasks, especially when a large amount of training data is available. For monocular depth estimation, we observe an improvement of up to 28% in relative performance when compared to a state-of-the-art fully-convolutional network. When applied to semantic segmentation, dense prediction transformers set a new state of the art on ADE20K with 49.02% mIoU. We further show that the architecture can be fine-tuned on smaller datasets such as NYUv2, KITTI, and Pascal Context where it also sets the new state of the art. Our models are available at https://github.com/intel-isl/DPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, neutral): “Remove Objects from Image [10, 35] Replace Objects from Image [10, 35] Change Image by the Text [35] Image Question Answering [23] Image-to-Text [23] Text-to-Image [35] Image-to-Edge [45] Edge-to-Image [53] Image-to-Line [16] Line-to-Image [53] Image-to-Hed [44] Hed-to-Image [53] Image-to-Seg [24] S”
Language-driven Semantic Segmentation (2022, methodology): “We find that LSeg performs competitively when using the RN50 × 16 text encoder and incurs only a negligible loss in performance when compared to the closest fixed-label segmentation method (DPT).”
Multiview Compressive Coding for 3D Reconstruction (2023, methodology): “[49], which differs from CO3D’s COLMAP output.”
SOCRATES: Introducing Depth in Visual Wildlife Monitoring Using Stereo Vision (2022, methodology): “3. it allows to generate training data for monocular depth estimation methods such as Godard et al. (2019); Ranftl et al. (2020, 2021).”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[185] René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun.”
XCiT: Cross-Covariance Image Transformers (2021, background): “In addition, Zheng et al. [83] and Ranftl et al. [54] study problems like semantic segmentation and monocular depth estimation with the quadratic self-attention operation.”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, methodology): “On the other hand, we use pre-trained depth DPT [41] to estimate the depth of I and crop a local patch d dpt with the same spatial location as d r .”
DNGaussian: Optimizing Sparse-View 3D Gaussian Radiance Fields with Global-Local Depth Normalization (2024, methodology): “The first group [11, 31] is to extract accurate but sparse depth values from reliable point clouds, and the second [14, 36, 40, 42, 55] distills depth knowledge from current powerful monocular depth estimators [29, 30].”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, methodology): “Sometimes, MTL papers take mix datasets for a “single” task and consider each dataset as a different task [37, 47, 35, 46].”
Depth Anything V2 (2024, positive): “For example, MiDaS [56, 55, 7] and Metric3D [95] collect 2M and 8M labeled images respectively.”
Multi-class Token Transformer for Weakly Supervised Semantic Segmentation (2022, background): “…to model long-range dependencies of long sequences in the ﬁeld of NLP. Recently, transformer models have been adapted to accommodate a wide variety of vision tasks [18], such as image classiﬁcation [10], saliency detection [27] and semantic segmentation [30], achieving promising performances.”
MultiMAE: Multi-modal Multi-task Masked Autoencoders (2022, methodology): “To pseudo label depth, we use a DPT-Hybrid [50] that was trained on Omnidata [23].”
Grounding Image Matching in 3D with MASt3R (2024, background): “Larger images would require significantly more compute power to train, and ViTs do not generalize yet to larger test-time resolutions [62,65].”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, positive): “Existing depth estimation methods are categorized into learning metric depth [17], [18], [19], [20], relative depth [21], [22], [23], [24], and afﬁne-invariant depth [25], [26], [27], [28], [29].”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “Recent work [16, 52, 53], in particular Omnidata [16], has made significant headway in terms of prediction quality and generalization to new scenes using very large datasets for training.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, methodology): “Recently, transformer-based methods [96, 80, 88, 65, 63, 44, 11, 10] have shown great potentials and outperform CNN-based methods.”
Zero-1-to-3: Zero-shot One Image to 3D Object (2023, methodology): “Since MCC requires depth input, we use MiDaS [42, 41] off-the-shelf for depth estimation.”
DINOv2: Learning Robust Visual Features without Supervision (2023, methodology): “5, the segmentation training in this experiment took 28 hours on 16 V100 GPUs.”
Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles (2023, background): “…(Dosovitskiy et al., 2021), video classification (Fan et al., 2021; Arnab et al., 2021; Bertasius et al., 2021), semantic segmentation (Ranftl et al., 2021), object detection (Carion et al., 2020; Li et al., 2022b), video object segmentation (Duke et al., 2021), 3D object detection…”