Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification

By Yulin Wang, Kangchen Lv, Rui Huang, Shiji Song, Le Yang, Gao Huang. 2020

172 citations recorded by the source corpus.

The accuracy of deep convolutional neural networks (CNNs) generally improves when fueled with high resolution images. However, this often comes at a high computational cost and high memory footprint. Inspired by the fact that not all regions in an image are task-relevant, we propose a novel framework that performs efficient image classification by processing a sequence of relatively small inputs, which are strategically selected from the original image with reinforcement learning. Such a dynamic decision process naturally facilitates adaptive inference at test time, i.e., it can be terminated once the model is sufficiently confident about its prediction and thus avoids further redundant computation. Notably, our framework is general and flexible as it is compatible with most of the state-of-the-art light-weighted CNNs (such as MobileNets, EfficientNets and RegNets), which can be conveniently deployed as the backbone feature extractor. Experiments on ImageNet show that our method consistently improves the computational efficiency of a wide variety of deep models. For example, it further reduces the average latency of the highly efficient MobileNet-V3 on an iPhone XS Max by 20% without sacrificing accuracy. Code and pre-trained models are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “By allowing early exiting, both spatially and temporally adaptive inference can be realized [37], [151].”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “Unlike lightweight convolution neural networks [37, 16, 44, 33], the quadratic computation complexity O(n(2)) with respect to sequence length n leads to high computation costs when employing self-attention with”
Dynamic Region-Aware Convolution (2020, methodology): “[31, 34, 8] use dynamic inference to reduce computation time.”
QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection (2021, background): “[55] selects small regions from the original input image by reinforcement learning and computes on these regions with a dynamic decision process.”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, background): “Besides, the spatial adaptive paradigm [15, 4, 47, 42, 46] has been proposed for efficient image and video recognition.”
Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length (2021, background): “, maxj pij (defined as confidence [22, 49, 44]), is compared with a threshold ηi.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “Besides the model-level adaptation, others further extend this idea to data-level adaptation, by either reducing the spatial redundancy [63] or focusing on key area [59].”
Adaptive Focus for Efficient Video Recognition (2021, background): “For example, in 2D-image classification, a number of recent works successfully improve the efficiency of CNNs by attending to some task-relevant or more informative parts of images [47, 41, 13].”
Vision Transformer with Deformable Attention (2022, methodology): “Our method can also be viewed as a spatial adaptive mechanism, which has been proved effective in various works [16,38].”