Spatial Transformer Networks
By Max Jaderberg, Karen Simonyan, Andrew Zisserman, Koray Kavukcuoglu. 2015
8,156 citations recorded by the source corpus.
Convolutional Neural Networks define an exceptionally powerful class of models, but are still limited by the lack of ability to be spatially invariant to the input data in a computationally and parameter efficient manner. In this work we introduce a new learnable module, the Spatial Transformer, which explicitly allows the spatial manipulation of data within the network. This differentiable module can be inserted into existing convolutional architectures, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself, without any extra training supervision or modification to the optimisation process. We show that the use of spatial transformers results in models which learn invariance to translation, scale, rotation and more generic warping, resulting in state-of-the-art performance on several benchmarks, and for a number of classes of transformations.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Sergey Levine
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
- Dacheng Tao
- Jianfeng Gao
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Quoc V. Le
- Yoshua Bengio
- Alan L. Yuille
- Kaiming He
- Xiaogang Wang
- Ruslan Salakhutdinov
- Dacheng Tao
- Jiashi Feng
Citations in the corpus
- Mask R-CNN (2017, methodology): “We use bilinear interpolation [22] to compute the exact values of the input features at four regularly sampled locations in each RoI bin, and aggregate the result (using max or average).”
- Fine-Grained Image Classification via Combining Vision and Language (2017, methodology): “Compared with the methods [37, 38, 19, 2, 5] which do not use object and part annotations, our CVL approach obtains a 1.”
- Learning Visual Features from Large Weakly Supervised Data (2015, background): “…studies have shown that using visual features extracted from convolutional networks trained on large object recognition datasets [26, 43, 46] can lead to state-of-the-art results on many vision problems including fine-grained classification [21, 39], object detection [15], and segmentation [36].”
- Speed/Accuracy Trade-Offs for Modern Convolutional Object Detectors (2016, methodology): “This is similar to the differentiable cropping mechanism of [5], the attention model of [11] as well as the Spatial Transformer Network [15].”
- Instance-Aware Semantic Segmentation via Multi-task Network Cascades (2015, methodology): “The RoI warping layer has a pre-defined output size and arbitrary input sizes, in contrast to [18].”
- Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering (2016, background): “invariance has been sought: (i) many data augmentation schemes have used rotated versions of images and (ii) models have been developed to learn this invariance, like the Spatial Transformer Networks [14]. Other explanations are the lack of experience on architecture design and the need to investiga”
- PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation (2016, background): “[9] introduces the idea of spatial transformer to align 2D images through sampling and interpolation, achieved by a specifically tailored layer implemented on GPU.”
- Invariant Information Clustering for Unsupervised Image Classification and Segmentation (2018, methodology): “The core differentiable operator in spatial transformer networks [32].”
- K-Net: Towards Unified Image Segmentation (2021, background): “Previous works [16, 18, 25, 26, 73] explore different kinds of dynamic kernels to improve the flexibility and performance of models.”
- Dynamic Neural Networks: A Survey (2021, background): “affine/projective/thin plate spline transformation) can be performed on images to undo certain variations [157] for better generalization ability, or to”
- Value Iteration Networks (2016, background): “First, the mapping from the state input to the shaped-reward map can be generalized by adding a spatial transformation layer (Jaderberg et al., 2015) between them.”
- Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering (2017, methodology): “In this work, rather than using hand-crafted or differentiable region proposals [41, 50, 17], we leverage Faster R-CNN [32], establishing a closer link between vision and language tasks and recent progress in object detection.”
- Squeeze-and-Excitation Networks (2017, background): “More recent work has sought to better model spatial dependence [1, 31] and incorporate spatial attention [19].”
- Deformable Convolutional Networks (2017, methodology): “Our approach shares similar high level spirit with spatial transform networks [26] and deformable part models [11].”
- Bilinear Attention Networks (2018, background): “attention heads) are used [13, 6, 15], then ŷ = fG g=1 P i ↵g,iyi, the concatenation of attended outputs.”
- Learning What and Where to Draw (2016, background): “Spatial Transformer Networks (STN) [Jaderberg et al., 2015] have proven to be an effective visual attention mechanism, and have already been incorporated into the latest deep generative models.”
- DenseCap: Fully Convolutional Localization Networks for Dense Captioning (2015, methodology): “Additionally, we replace their RoI pooling mechanism with a differentiable, spatial soft attention mechanism [19, 16].”
- Interaction Networks for Learning about Objects, Relations and Physics (2016, background): “The “graph neural network” [22] is a framework that shares learning 50 across nodes and edges, the “recursive autoencoder” [24] adapts its processing architecture to exploit 51 an input parse tree, the “neural programmer-interpreter” [21] is a composable neural network that 52 mimics the execution t”