Fully Dynamic Inference With Deep Neural Networks

By Wenhan Xia, Hongxu Yin, Xiaoliang Dai, Niraj Kumar Jha. 2020

55 citations recorded by the source corpus.

Modern deep neural networks are powerful and widely applicable models that extract task-relevant information through multi-level abstraction. Their cross-domain success, however, is often achieved at the expense of computational cost, high memory bandwidth, and long inference latency, which prevents their deployment in resource-constrained and time-sensitive scenarios, such as edge-side inference and self-driving cars. While recently developed methods for creating efficient deep neural networks are making their real-world deployment more feasible by reducing model size, they do not fully exploit input properties on a per-instance basis to maximize computational efficiency and task accuracy. In particular, most existing methods typically use a one-size-fits-all approach that identically processes all inputs. Motivated by the fact that different images require different feature embeddings to be accurately classified, we propose a fully dynamic paradigm that imparts deep convolutional neural networks with hierarchical inference dynamics at the level of layers and individual convolutional filters/channels. Two compact networks, called Layer-Net (L-Net) and Channel-Net (C-Net), predict on a per-instance basis which layers or filters/channels are redundant and therefore should be skipped. L-Net and C-Net also learn how to scale retained computation outputs to maximize task accuracy. By integrating L-Net and C-Net into a joint design framework, called LC-Net, we consistently outperform state-of-the-art dynamic frameworks with respect to both efficiency and classification accuracy. On the CIFAR-10 dataset, LC-Net results in up to <inline-formula><tex-math notation="LaTeX">$11.9\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>11</mml:mn><mml:mo>.</mml:mo><mml:mn>9</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="xia-ieq1-3056031.gif"/></alternatives></inline-formula> fewer floating-point operations (FLOPs) and up to 3.3 percent higher accuracy compared to other dynamic inference methods. On the ImageNet dataset, LC-Net achieves up to <inline-formula><tex-math notation="LaTeX">$1.4\times$</tex-math><alternatives><mml:math><mml:mrow><mml:mn>1</mml:mn><mml:mo>.</mml:mo><mml:mn>4</mml:mn><mml:mo>×</mml:mo></mml:mrow></mml:math><inline-graphic xlink:href="xia-ieq2-3056031.gif"/></alternatives></inline-formula> fewer FLOPs and up to 4.6 percent higher Top-1 accuracy than the other methods.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Efficiency & Compression

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “to decide: 1) which channels need to be activated [15], [85], [86], [87], [88] width, 2) which layers need to be skipped [47], [48], [92], [93], 3) which paths should be selected in a SuperNet [107], or 4) what locations of the input should be allocated computations [146], [147], [148], [182].”
Continual Object Detection via Prototypical Task Correlation Guided Gating Mechanism (2022, unknown): “Such strategy has also been used in other conditional inference methods including dynamic depth [47–49] and dynamic routing [50].”
A-ViT: Adaptive Tokens for Efficient Vision Transformer (2021, background): “One way is to change the architecture of the model to support adaptive computations [4,14,15, 18, 25, 27, 30, 37, 42, 51, 54].”