Dynamic Filter Networks

By Xu Jia, Bert De Brabandere, Tinne Tuytelaars, Luc Van Gool. 2016

1,102 citations recorded by the source corpus.

In a traditional convolutional layer, the learned filters stay fixed after training. In contrast, we introduce a new framework, the Dynamic Filter Network, where filters are generated dynamically conditioned on an input. We show that this architecture is a powerful one, with increased flexibility thanks to its adaptive nature, yet without an excessive increase in the number of model parameters. A wide variety of filtering operation can be learned this way, including local spatial transformations, but also others like selective (de)blurring or adaptive feature extraction. Moreover, multiple such layers can be combined, e.g. in a recurrent architecture. We demonstrate the effectiveness of the dynamic filter network on the tasks of video and stereo prediction, and reach state-of-the-art performance on the moving MNIST dataset with a much smaller model. By visualizing the learned filters, we illustrate that the network has picked up flow information by only looking at unlabelled training data. This suggests that the network can be used to pretrain networks for various supervised tasks in an unsupervised way, like optical flow and depth estimation.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring Self-Attention for Image Recognition (2020, background): “[17] explored dynamic filter networks, which generalized convolutions, but the construction incurred significant memory and computational costs and was not scaled up to high-resolution images and larger datasets.”
Scaling Autoregressive Video Models (2019, background): “Earlier work on video generation mostly focused on deterministic approaches [33, 4, 44, 24, 17], which fail to capture the high degree of stochasticity inherent in video.”
Local Relation Networks for Image Recognition (2019, methodology): “There are also some methods that learn a meta-network {θ} on input features to generate adaptive aggregation weights [15] or an adaptive aggregation scope across spatial positions [6], or learn a fixed prior about spatial aggregation scope (Ω) [14].”
HyperNetworks (2016, background): “…with another network is central to the work of (Jaderberg et al., 2016; Andrychowicz et al., 2016), and especially (Denil et al., 2013; Yang et al., 2015; Bertinetto et al., 2016; De Brabandere et al., 2016), where certain parameters in a convolutional network are predicted by another network.”
K-Net: Towards Unified Image Segmentation (2021, background): “Previous works [16, 18, 25, 26, 71] explore different kinds of dynamic kernels to improve the flexibility and performance of models.”
Dynamic Neural Networks: A Survey (2021, methodology): “Specifically, a filter generation network is built in DFN [116] to produce the filters for a convolutional layer.”
Deformable Convolutional Networks (2017, background): “Dynamic Filter [2] Similar to deformable convolution, the dynamic filters are also conditioned on the input features and change over samples.”
Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, methodology): “Many methods have been proposed to alleviate this problem using dynamic filters [29] or self-attention operations [63].”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “dynamic convolution [85, 92, 43, 99]).”
BAM: Bottleneck Attention Module (2018, background): “Dynamic Filter Network [27] proposes to generate convolutional features based on the input features for flexibility.”
DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, methodology): “Following [47], all these recent works [46, 48, 49] utilize the same concept to learn a steerable filter [46], weather prediction filter [48], an image enhancement filter [49], and a dynamic motion motion representation filter [50] using source-target image pairs.”
SMASH: One-Shot Model Architecture Search through HyperNetworks (2017, methodology): “, 2017), which are one of a class of techniques for dynamically adapting weights through use of an auxiliary model (Denil et al., 2013; Jia et al., 2016; Rebuffi et al., 2017; Schmidhuber, 1992).”
SchNet: A continuous-filter convolutional neural network for modeling quantum interactions (2017, methodology): “As in dynamic filter networks [32], this filter-generating function is modeled with a neural network.”
Bayesian Hypernetworks (2017, background): “A hypernetwork (Ha et al., 2017; Brabandere et al., 2016; Bertinetto et al., 2016) is a neural net that outputs parameters of another neural net (the "primary network") 2 .”
Continual learning with hypernetworks (2019, background): “Hypernetworks can therefore be thought of as weight generators, which were originally introduced to dynamically parameterize models in a compressed form (Ha et al., 2017; Schmidhuber, 1992; Bertinetto et al., 2016; Jia et al., 2016).”
Dynamic Edge-Conditioned Filters in Convolutional Neural Networks on Graphs (2017, positive): “To this end, we borrow the idea from Dynamic filter networks [5] and define a filter-generating network F l : R 7→ Rll−1 which given edge label L(j, i) outputs edge-specific weight matrix Θji ∈ R dl×dl−1 , see Figure 1.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, methodology): “In this work, we extend video prediction methods that are based on predicting a transformation from the previous image [31], [40].”