MaskConnect: Connectivity Learning by Gradient Descent

By Karim Ahmed, Lorenzo Torresani. 2018

52 citations recorded by the source corpus.

Although deep networks have recently emerged as the model of choice for many computer vision problems, in order to yield good results they often require time-consuming architecture search. To combat the complexity of design choices, prior work has adopted the principle of modularized design which consists in defining the network in terms of a composition of topologically identical or similar building blocks (a.k.a. modules). This reduces architecture search to the problem of determining the number of modules to compose and how to connect such modules. Again, for reasons of design complexity and training cost, previous approaches have relied on simple rules of connectivity, e.g., connecting each module to only the immediately preceding module or perhaps to all of the previous ones. Such simple connectivity rules are unlikely to yield the optimal architecture for the given problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Jiashi Feng
3. Quoc V. Le
4. Trevor Darrell
5. Dacheng Tao
6. Yi Yang
7. Li Fei-Fei
8. Luc Van Gool

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Alan L. Yuille
4. Ming-Hsuan Yang
5. Jiashi Feng
6. Trevor Darrell
7. Xiaogang Wang
8. Yi Yang

In Model Architectures

1. Alan L. Yuille
2. Liang-Chieh Chen
3. Jiashi Feng
4. Shuicheng Yan
5. Frank Hutter
6. Zhangyang Wang
7. Yunpeng Chen
8. Iasonas Kokkinos

In Optimization & Training

1. Shuicheng Yan
2. Jiashi Feng
3. Alan L. Yuille
4. Li Fei-Fei
5. Zhangyang Wang
6. Arber Zela
7. Sherif Sakr
8. Mohamed Maher

Citations in the corpus

• AutoML: A Survey of the State-of-the-Art (2019, methodology): “The widely used methods contains reinforcement learning (RL) [5], [6], [8], [9], [16], evolutionary algorithm (EA) [17]–[23], and gradient descent (GD) [10], [24], [25], Bayesian optimization [26]– [32] and so on.”
• A Comprehensive Survey of Neural Architecture Search (2020, background): “MaskConnect [69] found that the existing cell-based network architecture generally adopts a predefined fixed connection method between modules, for example, each module only connects its first two modules [29], or connects all the previous modules [35].”
• AutoGAN: Neural Architecture Search for Generative Adversarial Networks (2019, methodology): “For the optimization al-gorithm, popular options include reinforcement learning [4, 71, 69, 72], evolutionary algorithm [60], Bayesian optimization [24], random search [7], and gradient-based optimization methods [38, 1].”
• Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Ahmed and Torresani [1] used gradient based approach to learn network architecture.”
• Evaluating the Search Phase of Neural Architecture Search (2019, background): “2 Related Work Since its introduction in [36], neural architecture search demonstrated great potential to surpass the human design of deep networks for both visual recognition [17, 1, 7, 26, 15] and natural language processing [36, 27, 20, 37, 17, 5].”
• Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, background): “Additionally, several works [59, 49, 1] utilized architectural sharing among sampled models instead of training each of them individually, thereby further reduced the search cost.”
• Attentive Single-Tasking of Multiple Tasks (2019, background): “A deep single-task implementation of this would be the gating mechanism of [1].”
• Multi-Fiber Networks for Video Recognition (2018, background): “The idea of using spare connections to reduce the computational cost is similar to low-power networks built for mobile devices [25–27] as well as other recent approaches that try to sparsify parts of the network either through group convolutions [28] or through learning connectivity [29].”
• Task-Driven Modular Networks for Zero-Shot Compositional Learning (2019, methodology): “These networks have been used in the past to speed up computation at test time [1] and to improve generalization for multi-task learning [19, 26], reinforcement learning [7], continual learning [30], visual question answering [2, 25],”
• NAS-DIP: Learning Deep Image Prior with Neural Architecture Search (2020, methodology): “The primary methods are evolution [5,52,56, 63, 64, 70, 71, 80], reinforcement learning [7, 13, 74, 87, 89, 90], and differentiable search [2, 53].”
• AutoDispNet: Improving Disparity Estimation With AutoML (2019, background): “Orthogonally, relaxations make architectural decisions like branching patterns [1] and number of channels per layer [61] learnable via gradient descent.”