Vision Transformers Need Registers

By Timothée Darcet, Maxime Oquab, Maxime Oquab, Julien Mairal, Piotr Bojanowski. International Conference on Learning Representations, 2023

917 citations recorded by the source corpus.

Transformers have recently emerged as a powerful tool for learning visual representations. In this paper, we identify and characterize artifacts in feature maps of both supervised and self-supervised ViT networks. The artifacts correspond to high-norm tokens appearing during inference primarily in low-informative background areas of images, that are repurposed for internal computations. We propose a simple yet effective solution based on providing additional tokens to the input sequence of the Vision Transformer to fill that role. We show that this solution fixes that problem entirely for both supervised and self-supervised models, sets a new state of the art for self-supervised visual models on dense visual prediction tasks, enables object discovery methods with larger models, and most importantly leads to smoother feature maps and attention maps for downstream visual processing.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/10bd38673951f5d7729568284093cbd80482ab16

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

VGGT: Visual Geometry Grounded Transformer (2025, neutral): “…each input image I i , we augment the corresponding image to-kens t Ii with an additional camera token t g i ∈ R 1 × C ′ and four register tokens [10] Here, the camera token and register tokens of the ﬁrst frame ( t g 1 := ¯t g , t R 1 := ¯t R ) are set to a different set of learnable tokens t g…”
CoTracker: It is Better to Track Together (2023, background): “However, this approximation is crude as points are often strongly correlated ( e.g ., when they belong to the same object).”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, neutral): “We compare the performance of V-JEPA 2 on motion and appearance tasks with several other visual encoders: DINOv2 with registers (Darcet et al., 2024) is the current state-of-the-art model for self-supervised learning with images, while SigLIP2 (Tschannen et al., 2025) and the Perception Encoder PE…”
Depth Anything V2 (2024, neutral): “1 [7] Depth Anything V1 [”
ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation (2024, methodology): “Given the captured image, we first use DINOv2 with registers (ViT-S14) [5, 137] to extract the patch-wise features F patch ∈ R h ′ × w ′ × d .”
Think before you speak: Training Language Models With Pause Tokens (2023, background): “…only as an input has found its use in various forms, most commonly as <cls> (Chang et al., 2023; Liu, 2019; Devlin et al., 2019), <sep> or <mask> in BERT (Devlin et al., 2019) and in a line of work on adding memory to transformers (Burtsev et al., 2020; Bulatov et al., 2022; Darcet et al., 2023).”
What matters when building vision-language models? (2024, background): “Other variations over the Perceiver architecture (Mañas et al., 2023; Darcet et al., 2024; Vallaeys et al., 2024) resulted in decreased performance.”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, neutral): “For transformers, we apply DINO v2-reg [86], [104] vision transformers [87] (ViT) as backbones, DPT [28] as decoders.”
Nomic Embed Vision: Expanding the Latent Space (2024, methodology): “Recently released ViTs from (Wightman, 2019) using techniques like global registers (Darcet et al., 2023) and rotary positional embeddings (Fang et al., 2023b) show promise even though they are trained on a small dataset like Imagenet.”