Differentiable Patch Selection for Image Recognition

By Jean-Baptiste Cordonnier, Aravindh Mahendran, Alexey Dosovitskiy, Dirk Weissenborn, Jakob Uszkoreit, Thomas Unterthiner. 2021

123 citations recorded by the source corpus.

Neural Networks require large amounts of memory and compute to process high resolution images, even when only a small part of the image is actually informative for the task at hand. We propose a method based on a differentiable Top-K operator to select the most relevant parts of the input to efficiently process high resolution images. Our method may be interfaced with any downstream neural network, is able to aggregate information from different patches in a flexible way, and allows the whole model to be trained end-to-end using backpropagation. We show results for traffic sign recognition, inter-patch relationship reasoning, and fine-grained recognition without using object/part bounding box annotations during training.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Yadong Mu
2. Zhicheng Sun

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Yizeng Han
5. Le Yang
6. Yadong Mu
7. Ricardo Henao
8. Honghui Wang

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Le Yang
5. Yizeng Han
6. Honghui Wang
7. Fanjie Kong
8. Ricardo Henao

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, methodology): “an iterative manner, the recent differentiable patch selection [163] adopts a differentiable top-K module to select a fixed number of patches in one step.”
• Towards mental time travel: a hierarchical memory for reinforcement learning agents (2021, methodology): “However, we are not aware of prior transformers in which the coarse computations are used to select chunks for more detailed computation, as in HCAM (although recent work has demonstrated top-k patch selection in CNNs [8]); nor are we aware of prior works that have demonstrated their approaches in R”
• Patch-based Knowledge Distillation for Lifelong Person Re-Identification (2022, methodology): “[10] formulate patch sampling as a top-K problem and solve it with the Gumbel-max trick[24, 65] and perturbed optimizers [5] respectively.”
• Efficient Classification of Very Large Images with Tiny Objects (2021, methodology): “, attention sampling models (ATS) [20], Differentiable Patch Selection (Top-K) [7], BagNet [37], EfficientNet [44] and streaming CNNs [38].”