Learning Time-Efficient Deep Architectures with Budgeted Super Networks
By Tom Véniat, Ludovic Denoyer. 2017
22 citations recorded by the source corpus.
Learning neural network architectures is a way to discover new highly predictive models. We propose to focus on this problem from a different perspective where the goal is to discover architectures efficient in terms of both prediction quality and computation cost, e.g time in milliseconds, number of operations... For instance, our approach is able to solve the following task: find the best neural network architecture (in a very large set of possible architectures) able to predict well in less than 100 milliseconds on my mobile phone. Our contribution is based on a new family of models called Budgeted Super Networks that are learned using reinforcement-learning inspired techniques applied to a budgeted learning objective function which includes the computation cost during disk/memory operations at inference. We present a set of experiments on computer vision problems and show the ability of our method to discover efficient architectures in terms of both predictive quality and computation time.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Liang-Chieh Chen
- Trevor Darrell
- Sergey Levine
- Song Han
- Alan L. Yuille
- Jiashi Feng
- Dacheng Tao
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Trevor Darrell
- Dacheng Tao
- Quoc V. Le
- Alan L. Yuille
- Jiashi Feng
- Yi Yang
- Ming-Hsuan Yang
Citations in the corpus
- MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks (2017, methodology): “These methods use RL [35, 30] or genetic algorithms [25, 17] with the purpose of finding a network architecture which maximizes performance.”
- Neural Architecture Construction using EnvelopeNets (2018, background): “Convolutional Neural fabrics (29), PathNet (9), Budgeted SuperNets (31) adopt this strategy.”
- Structured Pruning of Neural Networks With Budget-Aware Regularization (2018, methodology): “Contrary to our method, this work does not leverage dropout-based SL. Budgeted Super Networks [32] is a method that finds an architecture satisfying a resource budget by sparsifying a super network at the module level.”
- Fast Neural Architecture Construction using EnvelopeNets (2018, background): “Convolutional Neural fabrics (37), PathNet (12), Budgeted SuperNets (39) adopt this strategy.”
- Inverted Residuals and Linear Bottlenecks: Mobile Networks for Classification, Detection and Segmentation (2018, methodology): “Recently there has been lots of progress in algorithmic architecture exploration included hyper-parameter optimization [8, 9, 10] as well as various methods of network pruning [11, 12, 13, 14, 15, 16] and connectivity learning [17, 18].”
- Gradient-free Policy Architecture Search and Adaptation (2017, methodology): “Architecture search has been investigated through different frameworks including reinforcement learning [24, 2, 22] and evolutionary techniques [12].”
- MobileNetV2: Inverted Residuals and Linear Bottlenecks (2018, methodology): “methods of network pruning [12, 13, 14, 15, 16, 17] and connectivity learning [18, 19].”