Multiscale Deep Equilibrium Models

By Shaojie Bai, Vladlen Koltun, J. Kolter. 2020

259 citations recorded by the source corpus.

We propose a new class of implicit networks, the multiscale deep equilibrium model (MDEQ), suited to large-scale and highly hierarchical pattern recognition domains. An MDEQ directly solves for and backpropagates through the equilibrium points of multiple feature resolutions simultaneously, using implicit differentiation to avoid storing intermediate states (and thus requiring only O(1) memory consumption). These simultaneously-learned multi-resolution features allow us to train a single model on a diverse set of tasks and loss functions, such as using a single MDEQ to perform both image classification and semantic segmentation. We illustrate the effectiveness of this approach on two large-scale vision tasks: ImageNet classification and semantic segmentation on high-resolution images from the Cityscapes dataset. In both settings, MDEQs are able to match or exceed the performance of recent competitive computer vision models: the first time such performance and scale have been achieved by an implicit deep learning approach. The code and pre-trained models are at this https URL .

Subfields: Artificial Intelligence, Ocean Engineering, Statistics and Probability, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Trevor Darrell
2. Vladlen Koltun
3. Bernard Ghanem
4. Luc Van Gool
5. Sergey Levine
6. Hongpeng Liao
7. Chien-Yao Wang
8. Dequan Wang

In Ocean Engineering

1. I-Hau Yeh
2. Hongpeng Liao
3. Chien-Yao Wang
4. Simon Lacoste-Julien
5. Michael W. Mahoney
6. Vladlen Koltun
7. Trevor Darrell
8. Stanley J. Osher

In Statistics and Probability

1. I-Hau Yeh
2. Hongpeng Liao
3. Chien-Yao Wang
4. Simon Lacoste-Julien
5. Matthias Müller
6. Samy Wu Fung
7. Michael W. Mahoney
8. Vladlen Koltun

In Model Architectures

1. Trevor Darrell
2. Bernard Ghanem
3. Dequan Wang
4. Vladlen Koltun
5. Hongpeng Liao
6. Chien-Yao Wang
7. Guohao Li
8. Matthias Müller

Citations in the corpus

• Multiset-Equivariant Set Prediction with Approximate Implicit Differentiation (2021, background): “Recent works in deep learning have successfully applied implicit differentiation to large-scale language modelling (Bai et al., 2019), semantic segmentation (Bai et al., 2020), meta-learning (Rajeswaran et al., 2019), neural architecture search (Zhang et al., 2021b), and more (Agrawal et al., 2019).”
• Training Graph Neural Networks with 1000 Layers (2021, methodology): “To construct a stable or contractive GNN block, we mimic the design of MDEQ (Bai et al., 2020).”
• JFB: Jacobian-Free Backpropagation for Implicit Networks (2021, methodology): “Moreover, JFB is competitive with Multiscale DEQs (Bai, Koltun, and Kolter 2020) despite having less than a tenth as many parameters.”
• Long Expressive Memory for Sequence Modeling (2021, background): “More recently, one can view convolutional neural networks as multiscale architectures for processing multiple spatial scales in data (Bai et al., 2020).”
• Tent: Fully Test-Time Adaptation by Entropy Minimization (2021, methodology): “To gauge its generality in practice, we evaluate new architectures based on self-attention (SAN) (Zhao et al., 2020) and equilibrium solving (MDEQ) (Bai et al., 2020) for corruption robustness on CIFAR-100-C.”
• You Only Learn One Representation: Unified Network for Multiple Tasks (2021, methodology): “The methods that belong to the category of implicit deep learning are mainly implicit neural representations [11] and deep equilibrium models [2, 3, 19].”