A Simple Cache Model for Image Recognition

By Emin Orhan. 2018

36 citations recorded by the source corpus.

Training large-scale image recognition models is computationally expensive. This raises the question of whether there might be simple ways to improve the test performance of an already trained model without having to re-train or fine-tune it with new data. Here, we show that, surprisingly, this is indeed possible. The key observation we make is that the layers of a deep network close to the output layer contain independent, easily extractable class-relevant information that is not contained in the output layer itself. We propose to extract this extra class-relevant information using a simple key-value cache memory to improve the classification performance of the model at test time. Our cache memory is directly inspired by a similar cache model previously proposed for language modeling (Grave et al., 2017). This cache component does not require any training or fine-tuning; it can be applied to any pre-trained model and, by properly setting only two hyper-parameters, leads to significant improvements in its classification performance. Improvements are observed across several architectures and datasets. In the cache component, using features extracted from layers close to the output (but not from the output layer itself) as keys leads to the largest improvements. Concatenating features from multiple layers to form keys can further improve performance over using single-layer features as keys. The cache component also has a regularizing effect, a simple consequence of which is that it substantially increases the robustness of models against adversarial attacks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “Similar k NN models to ours have been proposed for computer vision tasks (Papernot & McDaniel, 2018; Orhan, 2018; Zhao & Cho, 2018), primarily motivated by improving interpretability and robustness to adversarial attacks.”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, methodology): “Different from CLIP-Adapter, Tip-Adapter does not require SGD to train the adapter but constructs a query-key cache model [20, 32, 48] from few-shot supervisions to obtain the weights of adapter.”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Different from CLIP-Adapter, the weights of adapter is obtained in a training-free manner from a query-key cache model [140], [57] constructed from fewshot supervisions in a non-parametric manner.”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, methodology): “Different from existing methods, ours does not require extra training, but designs the adapter as a query-key cache model [30,45,18] from the few-shot dataset.”