Resolution Adaptive Networks for Efficient Inference

By Le Yang, Yizeng Han, Xi Chen, Shiji Song, Jifeng Dai, Gao Huang. 2020

247 citations recorded by the source corpus.

Adaptive inference is an effective mechanism to achieve a dynamic tradeoff between accuracy and computational cost in deep networks. Existing works mainly exploit architecture redundancy in network depth or width. In this paper, we focus on spatial redundancy of input samples and propose a novel Resolution Adaptive Network (RANet), which is inspired by the intuition that low-resolution representations are sufficient for classifying “easy” inputs containing large objects with prototypical features, while only some “hard” samples need spatially detailed information. In RANet, the input images are first routed to a lightweight sub-network that efficiently extracts low-resolution representations, and those samples with high prediction confidence will exit early from the network without being further processed. Meanwhile, high-resolution paths in the network maintain the capability to recognize the “hard” samples. Therefore, RANet can effectively reduce the spatial redundancy involved in inferring high-resolution inputs. Empirically, we demonstrate the effectiveness of the proposed RANet on the CIFAR-10, CIFAR-100 and ImageNet datasets in both the anytime prediction setting and the budgeted batch classification setting.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges (2021, background): “[158] leverage EE strategies for multi-scale inputs and propose an approach to classify “easy” samples with smaller neural models.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, background): “Multi-scale dense networks [24] and adaptive resolution networks [54] focus on spatial redundancy of input samples and use a multi-scale dense connection archi-”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “In the context of recognition tasks, MSDNet and its variants [23, 52, 27] develop a multi-classifier CNN architecture to perform early exiting for easy samples.”
HMS: Hierarchical Modality Selection for Efficient Video Recognition (2021, methodology): “Conditional computation methods have been proposed for both image and video recognition to reduce computation while maintaining accuracy by dynamically performing early exiting in deep networks [2, 16], varying input resolutions [39, 49], skipping network layers or blocks [43] and selecting smaller ”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “Additionally, we believe our method is also complementary to other clip selection methods leveraging multi-modal inputs such as audio [20, 12], as well as adaptive spatial resolution modulating methods [26, 36, 46].”
Dynamic Network Quantization for Efficient Video Inference (2021, unknown): “Representative methods for image classification, dynamically adjust network depth [13, 33, 58, 21, 62], width [65, 7, 20], perform routing [26, 33] or switch resolutions [61].”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “…the research of dynamic networks can be categorized into three main directions, which are dynamic depth for network early exiting [5] or layer skipping [35, 61], dynamic width for skipping neurons [4] or channels [37], and dynamic routing for multi-branch or tree structure networks [21, 34, 63].”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “Dyn-Perceiver [13] achieves efficient visual recognition through dynamic early exiting [15, 14, 51].”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “One way is to change the architecture of the model to support adaptive computations [4,14,15, 18, 25, 27, 30, 37, 42, 51, 54].”
Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, methodology): “, maxj ptj , (treated as confidence following earlier works [20, 64]) is compared to a pre-defined threshold ηt.”
AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, unknown): “3.3.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “Besides the model-level adaptation, others further extend this idea to data-level adaptation, by either reducing the spatial redundancy [63] or focusing on key area [59].”
Adaptive Focus for Efficient Video Recognition (2021, background): “It has been observed that considerable spatial redundancy exists in the process of extracting deep features from image-based data [18, 48].”