Fine-tuning Image Transformers using Learnable Memory

By Mark Sandler, Andrey Zhmoginov, Max Vladymyrov, Andrew Jackson. 2022

57 citations recorded by the source corpus.

In this paper we propose augmenting Vision Transformer models with learnable memory tokens. Our approach allows the model to adapt to new tasks, using few parameters, while optionally preserving its capabilities on previously learned tasks. At each layer we introduce a set of learnable embedding vectors that provide contextual information useful for specific datasets. We call these “memory tokens”. We show that augmenting a model with just a handful of such tokens per layer significantly improves accuracy when compared to conventional head-only fine-tuning, and performs only slightly below the significantly more expensive full fine-tuning. We then propose an attention-masking approach that enables extension to new downstream tasks, with a computation reuse. In this setup in addition to being parameters efficient, models can execute both old and new tasks as a part of single inference at a small incremental cost.

Subfields: Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Visual Prompt Tuning (2022, methodology): “Due to the disparity between vision and language modalities, in this paper we ask: can the same method can be applied successfully to image encoders? We are the first work (see related concurrent works [52,62,11,2]) to tackle this question and investigate the generality and feasibility of visual pro”
Doubly Right Object Recognition: A Why Prompt for Visual Rationales (2022, background): “Recently, the computer vision field has adapted the language and proposed the visual prompts to adapt vision models [4, 5, 21, 43].”
Vision Transformers Need Registers (2023, background): “Sandler et al. (2022) extend this line to the vision domain for fine-tuning but observe that such tokens do not transfer well across tasks.”