Flat2Sphere: Learning Spherical Convolution for Fast Features from 360° Imagery

By Yu-Chuan Su, Kristen Grauman. 2017

100 citations recorded by the source corpus.

While 360° cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial. Convolutional neural networks (CNNs) trained on images from perspective cameras yield “flat" filters, yet 360° images cannot be projected to a single plane without significant distortion. A naive solution that repeatedly projects the viewing sphere to all tangent planes is accurate, but much too computationally intensive for real problems. We propose to learn a spherical convolutional network that translates a planar CNN to process 360° imagery directly in its equirectangular projection. Our approach learns to reproduce the flat filter outputs on 360° data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient feature extraction for 360° images and video, and 2) the ability to leverage powerful pre-trained networks researchers have carefully honed (together with massive labeled image training sets) for perspective images. We validate our approach compared to several alternative methods in terms of both raw CNN output accuracy as well as applying a state-of-the-art “flat" object detector to 360° data. Our method yields the most accurate results while saving orders of magnitude in computation versus the existing exact reprojection solution.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Gauge Equivariant Convolutional Networks and the Icosahedral CNN (2019, methodology): “…we present in this paper a speciﬁc model (the Icosahedral CNN), which can be viewed as a fast and simple alternative to Spherical CNNs (Cohen et al., 2018b; Esteves et al., 2018; Boomsma & Frellsen, 2017; Su & Grauman, 2017; Perraudin et al., 2018; Jiang et al., 2018; Kondor et al., 2018a).”
Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, neutral): “Learning images on the sphere in a rotation invariant way has applications in a wide range of domains from 360 degree video through drone navigation to molecular chemistry [9, 10, 11, 12, 13, 14, 15, 16].”
Deep Learning 3D Shapes Using Alt-az Anisotropic 2-Sphere Convolution (2018, background): “Recently, researchers have started to explore deep spherical convolutional neural networks for tasks such as molecular modeling (Boomsma & Frellsen (2017)), ominidirectional vision (Su & Grauman (2017)) and 3D shape recognition (Cohen et al. (2018); Esteves et al. (2018)).”
SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images (2018, methodology): “However, unlike [30] we keep weight tying in place for image rows with filters of the same dimension, thus reducing the number of model parameters.”
DeepSphere: Efficient spherical Convolutional Neural Network with HEALPix sampling for cosmological applications (2018, methodology): “One approach is to apply a standard 2D CNN to a grid discretisation of the sphere [20–22].”