Learning to Zoom: a Saliency-Based Sampling Layer for Neural Networks

By Adrià Recasens, Petr Kellnhofer, Simon Stent, Wojciech Matusik, Antonio Torralba. 2018

153 citations recorded by the source corpus.

We introduce a saliency-based distortion layer for convolutional neural networks that helps to improve the spatial sampling of input data for a given task. Our differentiable layer can be added as a preprocessing block to existing task networks and trained altogether in an end-to-end fashion. The effect of the layer is to efficiently estimate how to sample from the original data in order to boost task performance. For example, for an image classification task in which the original data might range in size up to several megapixels, but where the desired input images to the task network are much smaller, our layer learns how best to sample from the underlying high resolution data in a manner which preserves task-relevant information better than uniform downsampling. This has the effect of creating distorted, caricature-like intermediate images, in which idiosyncratic elements of the image that improve task performance are zoomed and exaggerated. Unlike alternative approaches such as spatial transformer networks, our proposed layer is inspired by image saliency, computed efficiently from uniformly downsampled data, and degrades gracefully to a uniform sampling strategy under uncertainty. We apply our layer to improve existing networks for the tasks of human gaze estimation and fine-grained object classification. Code for our method is available in: http://github.com/recasens/Saliency-Sampler.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Deep Plastic Surgery: Robust and Controllable Image Editing with Human-Drawn Sketches (2020, methodology): “We use the sampling layer [9] to deform lines.”
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-Grained Image Recognition (2019, methodology): “ our TASN with three sampling-based methods: 1) uniformed sampling with high resolution (i.e., zoom in), 2) uniformed sampling with attention (i.e., crop) and 3) nonuniformed sampling proposed in SSN [23]. As shown in Table5, higher resolution can signiﬁcantly improve ﬁnegrained recognition performa”
Efficient Segmentation: Learning Downsampling Near Semantic Boundaries (2019, background): “They explore different parameterizations for spatial transformation including affine, projective, splines [28] or specially designed saliency-based layers [42].”