Making 360° Video Watchable in 2D: Learning Videography for Click Free Viewing

By Yu-Chuan Su, Kristen Grauman. 2017

115 citations recorded by the source corpus.

360° Video requires human viewers to actively control where to look while watching the video. Although it provides a more immersive experience of the visual content, it also introduces additional burden for viewers, awkward interfaces to navigate the video lead to suboptimal viewing experiences. Virtual cinematography is an appealing direction to remedy these problems, but conventional methods are limited to virtual environments or rely on hand-crafted heuristics. We propose a new algorithm for virtual cinematography that automatically controls a virtual camera within a 360° video. Compared to the state of the art, our algorithm allows more general camera control, avoids redundant outputs, and extracts its output videos substantially more efficiently. Experimental results on over 7 hours of real in the wild video show that our generalized camera control is crucial for viewing 360° video, while the proposed efficient algorithm is essential for making the generalized control computationally tractable.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, methodology): “A number of authors have addressed this problem in different ways [14, 15].”
Saliency in VR: How Do People Explore Virtual Environments? (2016, background): “[51, 52] introduced the problem of automatically extracting paths of a camera with a smaller field-of-view through 360◦ panorama videos, dubbed pano2vid.”
Kernel Transformer Networks for Compact Spherical Convolution (2018, methodology): “Existing methods often take an off-the-shelf model trained on perspective images and either 1) apply it repeatedly to multiple perspective projections of the 360 ◦ image [10,38,40,43] or 2) apply it once to a single equirectan-gular projection [19,30].”
SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images (2018, background): “Over the last years, omnidirectional imaging devices have gained in popularity due to their wide field of view and their widespread applications ranging from virtual reality to robotics [10,16,21,27,28].”
Learning Spherical Convolution for Fast Features from 360° Imagery (2017, methodology): “Motivated by the limitations of existing interfaces for viewing 360° video, several methods study how to automate field-of-view (FOV) control for display [19, 26, 34, 35], adopting one of the two existing strategies for convolutions (Fig.”
Flat2Sphere: Learning Spherical Convolution for Fast Features from 360° Imagery (2017, methodology): “In the second existing strategy, the 360 ° image is repeatedly projected to tangent planes around the sphere, each of which is then fed to the CNN/ﬁlters [27,28,31,34] (Fig.”
Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (2019, methodology): “These methods mainly leverage visual information such as saliency [47], [48], [49], [50], [51].”