Predicting Important Objects for Egocentric Video Summarization
By Yong Jae Lee, Kristen Grauman. 2015
166 citations recorded by the source corpus.
We present a video summarization approach for egocentric or “wearable” camera data. Given hours of video, the proposed method produces a compact storyboard summary of the camera wearer’s day. In contrast to traditional keyframe selection techniques, the resulting summary focuses on the most important objects and people with which the camera wearer interacts. To accomplish this, we develop region cues indicative of high-level saliency in egocentric video—such as the nearness to hands, gaze, and frequency of occurrence—and learn a regressor to predict the relative importance of any new region based on these cues. Using these predictions and a simple form of temporal event detection, our method selects frames for the storyboard that reflect the key object-driven happenings. We adjust the compactness of the final summary given either an importance selection criterion or a length budget; for the latter, we design an efficient dynamic programming solution that accounts for importance, visual uniqueness, and temporal displacement. Critically, the approach is neither camera-wearer-specific nor object-specific; that means the learned importance metric need not be trained for a given user or context, and it can predict the importance of objects and people that have never been seen previously. Our results on two egocentric video datasets show the method’s promise relative to existing techniques for saliency and summarization.
Subfields: Computer Vision and Pattern Recognition, Summarization
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Kristen Grauman
- Luc Van Gool
- Jitendra Malik
- Trevor Darrell
- Dacheng Tao
- Andrew Zisserman
- Cordelia Schmid
- Ming-Hsuan Yang
In Summarization
- Ah-Hwee Tan
- Cheston Tan
- Ana Garcia Del Molino
- Joo-Hwee Lim
- Tushar Nagarajan
- James M. Rehg
- Miao Liu
- Yin Li
Citations in the corpus
- Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “2011; Ryoo and Matthies 2013) as well as features from human gaze, hands, and objects (Spriggs, De La Torre, and Hebert 2009; Lee, Ghosh, and Grauman 2012; Fathi, Farhadi, and Rehg 2011; Pirsiavash and Ramanan 2012; Li and Kitani 2013; Lee and Grauman 2015).”
- Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “…challenges, such as human-object interactions [26,46,163], activity recognition [110,139,243], anticipation [4,75,86,144,205], video summarization [48, 129,131,147,148,232], detecting hands [16,134], parsing social interactions [66,168,231], and inferring the camera wearer’s body pose [107].”
- The Open World of Micro-Videos (2016, background): “Wearable cameras such as Google Glass and GoPro have spurred interest in analyzing egocentric views [18,19,20,21].”
- Privacy-Preserving Visual Learning Using Doubly Permuted Homomorphic Encryption (2017, background): “By allowing the cloud administrator to curate these videos, he will be able to learn activity classifiers [19, 20, 37, 45, 48, 52] from realistic and diverse data.”
- Computer Vision and Image Understanding Computer Vision for Assistive Technologies (unknown): “…and temporal segmentation of daily actions and contexts in video ( Damen et al., 2014; Furnari et al., 2015; 2016; Gurrin et al., 2014; Lee et al., 2012; Lee and Grauman, 2015; Lu and Grauman, 2013; Ortis et al., 2016; Poleg et al., 2014 ) can greatly help to support users in their daily life.”
- Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “This special viewpoint offers an array of interesting challenges, such as detecting gaze [40, 29], monitoring human-object interactions [4, 6, 51], creating daily life activity summaries [44, 39, 70, 43], or inferring the camera wearer’s identity or body pose [28, 33].”
- Next-active-object prediction from egocentric videos (2017, background): “Despite the fact First Person Vision (FPV) systems are exposed to a huge amount of user-related ∗Corresponding author Email addresses: furnari@dmi.unict.it (Antonino Furnari), battiato@dmi.unict.it (Sebastiano Battiato), grauman@cs.utexas.edu (Kristen Grauman), gfarinella@dmi.unict.it (Giovanni Mari”
- In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, methodology): “gaze estimation [10], [12], [13], hand analysis [72], [73], [74], [75], [76], pose estimation [14], [15], physiological parameter estimation [77], [78], user identification [79], [80], [81] and video summarization [16], [17], [18].”
- Summarization of Egocentric Videos: A Comprehensive Survey (2017, methodology): “Finally, importance may be estimated with supervised learning [23], [25], [26]; inferred from impersonal cues such as saliency [17], people and object interaction [10], [12], [16], [17], [19], and location [13]; or predicted from the wearer attention patterns, using sensors [14], [15], [22] or motio”
- Deepdiary: Lifelogging image captioning and summarization (2018, unknown): “An alternative is to use unsupervised techniques that do not try to recognize speci fi c content, but instead look for recurring visual patterns that may correspond to important content [57,81], or choose key images to summarize video or photo streams [62,73,90].”