Convolutional Networks with Adaptive Inference Graphs

By Andreas Veit, Serge J. Belongie. 2017

416 citations recorded by the source corpus.

Do convolutional networks really need a fixed feed-forward structure? What if, after identifying the high-level concept of an image, a network could move directly to a layer that can distinguish fine-grained differences? Currently, a network would first need to execute sometimes hundreds of intermediate layers that specialize in unrelated aspects. Ideally, the more a network already knows about an image, the better it should be at deciding which layer to compute next. In this work, we propose convolutional networks with adaptive inference graphs (ConvNet-AIG) that adaptively define their network topology conditioned on the input image. Following a high-level structure similar to residual networks (ResNets), ConvNet-AIG decides for each input image on the fly which layers are needed. In experiments on ImageNet we show that ConvNet-AIG learns distinct inference graphs for different categories. Both ConvNet-AIG with 50 and 101 layers outperform their ResNet counterpart, while using 20%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$20\%$$\end{document} and 38%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$38\%$$\end{document} less computations respectively. By grouping parameters into layers for related classes and only executing relevant layers, ConvNet-AIG improves both efficiency and overall classification quality. Lastly, we also study the effect of adaptive inference graphs on the susceptibility towards adversarial examples. We observe that ConvNet-AIG shows a higher robustness than ResNets, complementing other known defense mechanisms.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, methodology): “SkipNet [47] and convolutional network with adaptive inference graph (Conv-AIG) [48] are two typical approaches to enabling dynamic layer skipping.”
Routing Networks with Co-training for Continual Learning (2020, background): “Sparsely-Gated Mixture-of-Experts (Shazeer et al., 2017) and other routing network variants (Rosenbaum et al., 2017; Veit & Belongie, 2018; Maziarz et al., 2019; Guo et al., 2019) can be trained by gradient descent.”
Conditional Channel Gated Networks for Task-Aware Continual Learning (2020, background): “Such gates may either drop entire residual blocks [38, 39] or speciﬁc units within a layer [6, 3].”
Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation (2019, background): “Another extension of the layer-wise dropout solution is to design early-exiting prediction branches to reduce the average execution depth in inference [17, 1, 35, 23].”
SpotTune: Transfer Learning Through Adaptive Fine-Tuning (2018, methodology): “Veit and Belongie [46] investigated the same idea using Gumbel Softmax [18] for on-the-fly selection of residual blocks.”
AdaFrame: Adaptive Frame Selection for Fast Video Recognition (2018, background): “Related are also [30, 26, 32, 15, 5] that learn to drop layers in residual networks or learn where to look in images conditioned on inputs.”
Dynamic Modality Interaction Modeling for Image-Text Retrieval (2021, methodology): “In other words, on the basis of [31], we introduced the gumbel-softmax [13] trick to discretize path values (i.”
AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition (2021, methodology): “Many conditional computation methods have been recently proposed with the goal of improving computational efﬁciency (Bengio et al., 2015; 2013; Veit & Belongie, 2018; Wang et al., 2018b; Graves, 2016; Meng et al., 2020; Pan et al., 2021).”
Mitigating Forgetting in Online Continual Learning via Instance-Aware Parameterization (2020, methodology): “Instance-aware has been used in image classiﬁcation [7–9] for solving the diverse and various samples in the dataset.”
Fast and Practical Neural Architecture Search (2019, background): “Recent works [29, 30, 24] also showed that it may be more efﬁcient to have different blocks at various positions in a network.”
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, background): “Another line of works that resemble conditional computation focus on dynamic network configuration [29, 1, 7, 2, 4].”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, methodology): “Conv-AIG [45] learns conditional gating of residual blocks via Gumbel-softmax trick.”
Not All Images are Worth 16x16 Words: Dynamic Vision Transformers with Adaptive Sequence Length (2021, background): “Another type of dynamic CNNs skips redundant layers [38, 42, 48] or channels [28] conditioned on the inputs.”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, background): “Another type of dynamic CNNs skips redundant layers [41, 45, 51] or channels [29] conditioned on the inputs.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “a, conditional computation) methods have been developed in the image domain, achieving reduced computation by dynamically selecting channels [2, 25], skipping layers [42, 11, 40, 37], performing early exiting with auxiliary structures [22, 18, 3, 46], adaptively switching input resolutions [27, 36, ”