Dynamic Neural Networks: A Survey

By Yizeng Han, Gao Huang, Shiji Song, Le Yang, Honghui Wang, Yulin Wang. 2021

940 citations recorded by the source corpus.

Dynamic neural network is an emerging research topic in deep learning. Compared to static models which have fixed computational graphs and parameters at the inference stage, dynamic networks can adapt their structures or parameters to different inputs, leading to notable advantages in terms of accuracy, computational efficiency, adaptiveness, etc. In this survey, we comprehensively review this rapidly developing area by dividing dynamic networks into three main categories: 1) sample-wise dynamic models that process each sample with data-dependent architectures or parameters; 2) spatial-wise dynamic networks that conduct adaptive computation with respect to different spatial locations of image data; and 3) temporal-wise dynamic models that perform adaptive inference along the temporal dimension for sequential data such as videos and texts. The important research problems of dynamic networks, e.g., architecture design, decision making scheme, optimization technique and applications, are reviewed systematically. Finally, we discuss the open problems in this field together with interesting future research directions.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Fast Inference from Transformers via Speculative Decoding (2022, methodology): “These adaptive computation methods (e.g. Han et al., 2021; Sukhbaatar et al., 2019; Schuster et al., 2021; Scardapane et al., 2020; Bapna et al., 2020; Elbayad et al., 2019; Schwartz et al., 2020) aim to use less compute resources for easier inference steps.”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “Dyn-Perceiver [13] achieves efficient visual recognition through dynamic early exiting [15, 14, 51].”
Nonlinear Transform Source-Channel Coding for Semantic Communications (2021, methodology): “Therefore, in this paper, we introduce the dynamic neural network structure [36] into Transformers to realize the deep JSCC encoder fe.”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “Many adaptive computation methods are explored (Bengio et al., 2015; Wang et al., 2018; Graves, 2016; Hu et al., 2020; Wang et al., 2020b; Han et al., 2021b) to alleviate the computation burden.”
IA-RED2: Interpretability-Aware Redundancy Reduction for Vision Transformers (2021, methodology): “To overcame this issue, many adaptive computation methods are explored during the inference stage [1, 2, 57, 15, 26, 58, 20].”
SimMIM: a Simple Framework for Masked Image Modeling (2021, background): “Such claim is also partly supported by recent works of sparse inference [19] that the recognition accuracy has very little drop after throwing a large portion of image features [23,41,51].”
Domain Generalization: A Survey (2021, background): “One potential solution is to develop dynamic architectures [230], e.”
Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length (2021, background): “Designing dynamic architectures is an effective approach for efficient deep learning [17].”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “Differing from the traditional static network structures, dynamic networks can adapt their structures or parameters to the given example during inference, yielding appealing properties like better representation power, adaptiveness, compatibility and generality [15].”
Dynamic Slimmable Network (2021, neutral): “Please refer to [19] for a more comprehensive review of dynamic neural networks.”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, background): “Designing dynamic architectures is an effective approach for efficient deep learning [18].”
Adaptive Focus for Efficient Video Recognition (2021, background): “It has been observed that considerable spatial redundancy exists in the process of extracting deep features from image-based data [18, 48].”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “A more detailed analysis of the literature is present in [19].”