Pano2Vid: Automatic Cinematography for Watching 360° Videos

By Yu-Chuan Su, Dinesh Jayaraman, Kristen Grauman. 2016

137 citations recorded by the source corpus.

We introduce the novel task of Pano2Vid — automatic cinematography in panoramic 360\(^{\circ }\) videos. Given a 360\(^{\circ }\) video, the goal is to direct an imaginary camera to virtually capture natural-looking normal field-of-view (NFOV) video. By selecting “where to look” within the panorama at each time step, Pano2Vid aims to free both the videographer and the end viewer from the task of determining what to watch. Towards this goal, we first compile a dataset of 360\(^{\circ }\) videos downloaded from the web, together with human-edited NFOV camera trajectories to facilitate evaluation. Next, we propose AutoCam, a data-driven approach to solve the Pano2Vid task. AutoCam leverages NFOV web video to discriminatively identify space-time “glimpses” of interest at each time instant, and then uses dynamic programming to select optimal human-like camera trajectories. Through experimental evaluation on multiple newly defined Pano2Vid performance measures against several baselines, we show that our method successfully produces informative videos that could conceivably have been captured by human videographers.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Clebsch-Gordan Nets: a Fully Fourier Space Spherical Convolutional Neural Network (2018, methodology): “Learning images on the sphere in a rotation invariant way has applications in a wide range of domains from 360 degree video through drone navigation to molecular chemistry [9, 10, 11, 12, 13, 14, 15].”
Learning to Localize Sound Sources in Visual Scenes: Analysis and Applications (2019, methodology): “We use the AUTOCAM [48] method to generate a path of the sound source in 360◦ videos.”
Making 360° Video Watchable in 2D: Learning Videography for Click Free Viewing (2017, methodology): “Following [37], we train the classifier using leave-one-360°-video-out strategy for each keyword.”
Saliency in VR: How Do People Explore Virtual Environments? (2016, background): “[51, 52] introduced the problem of automatically extracting paths of a camera with a smaller field-of-view through 360◦ panorama videos, dubbed pano2vid.”
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks (2017, background): “Saliency and attention: Previous work studies the question of “where to look” to prioritize portions of already captured image/video data, so as to reserve computation for the most salient regions or block out distractors [1, 5, 8, 23, 41, 47, 52, 62, 71], or to predict the gaze or preference of a h”
Learning Spherical Convolution for Fast Features from 360° Imagery (2017, methodology): “5a shows the output error of layers conv3_3 and conv5_3 on the Pano2Vid [35] dataset (see Supp.”