Learning Time/Memory-Efficient Deep Architectures with Budgeted Super Networks

By Tom Véniat, Ludovic Denoyer. 2017

84 citations recorded by the source corpus.

We propose to focus on the problem of discovering neural network architectures efficient in terms of both prediction quality and cost. For instance, our approach is able to solve the following tasks: learn a neural network able to predict well in less than 100 milliseconds or learn an efficient model that fits in a 50 Mb memory. Our contribution is a novel family of models called Budgeted Super Networks (BSN). They are learned using gradient descent techniques applied on a budgeted learning objective function which integrates a maximum authorized cost, while making no assumption on the nature of this cost. We present a set of experiments on computer vision problems and analyze the ability of our technique to deal with three different costs: the computation cost, the memory consumption cost and a distributed computation cost. We particularly show that our model can discover neural network architectures that have a better accuracy than the ResNet and Convolutional Neural Fabrics architectures on CIFAR-10 and CIFAR-100, at a lower cost.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

A Comprehensive Survey of Neural Architecture Search (2020, background): “In addition, these works [70–72] are also exploring the search for network architecture on continuous domains.”
Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search (2018, methodology): “The idea of super net and stochastic super net is also used in Saxena & Verbeek (2016); Veniat & Denoyer (2017) to explore macro architectures of neural nets.”
Efficient Continual Learning with Modular Networks and Task-Driven Priors (2020, methodology): “This algorithm alternates between one step of gradient descent over the paths via REINFORCE (Williams, 1992) as in (Veniat & Denoyer, 2018), and one step of gradient descent over the parameters for a given path.”
Single Path One-Shot Neural Architecture Search with Uniform Sampling (2019, background): “BSN[22] Stochastic super networks + policy gradient Weight of cost penalty Single path Soft constraint in training.”
Learning Dynamic Routing for Semantic Segmentation (2020, methodology): “Different from traditional RLbased methods [36, 38, 34], which adopt policy gradient to update the agent for discrete path selection, we propose the soft conditional gate for differentiable routing.”
SGAS: Sequential Greedy Architecture Search (2019, background): “In order to search in a continuous domain [41, 1, 43, 50], DARTS [33] proposes a continuous relaxation of the architecture representation, making architecture search differentiable and hence much more efficient.”
Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, methodology): “Budgeted Super Networks [32] is a method that finds an architecture satisfying a resource budget by sparsifying a super network at the module level.”
Green AI (2019, methodology): “FPO has been used in the past to quantify the energy footprint of a model [26, 42, 12, 41], but is not widely adopted in AI.”
InstaNAS: Instance-aware Neural Architecture Search (2018, methodology): “Several conditional computation methods have been proposed to dynamically execute different modules of a model on a per-example basis [2, 12, 14, 27, 32, 30].”